Computer Vision
GHR-VLM: Making Zero-Shot Transit Video Analytics Realizable with Grounded Hybrid Reasoning
Kaicong Huang, Weiheng Oh, Jack M. Reilly et al. (+2 more)
Transit
video understanding can provide valuable fine-grained data that conventional passenger counters and fare systems cannot capture. However, supervised
video models require task-specific annotations, while applying
vision-
language models (VLMs) directly to long onboard
videos is unreliable and costly. To leverage the complementary strengths of both approaches, we propose GHR-VLM, a visual grounded hybrid
reasoning framework for zero-shot transit-bus
video analytics. It is motivated by the observation that explicit visual grounding can improve VLM
reasoning by converting long surveillance streams into compact, passenger-centered spatiotemporal evidence. Specifically, we propose an edge-cloud design in which a lightweight edge-based monitor continuously tracks door status and segments passenger clips. A backend VLM then identifies boarding passengers and classifies payment behavior through a two-stage coarse-to-fine refinement of spatiotemporal evidence. By invoking the VLM only on grounded passenger clips and contact sheets, GHR-VLM reduces cloud inference, avoids payment-specific training data, and supplies the localized evidence that VLMs otherwise struggle to identify. Evaluation on 486 minutes of real-world bus surveillance
video demonstrates the potential of grounded edge-cloud
reasoning for passenger-level payment analytics while highlighting the challenges posed by degraded
video conditions.
Read the full abstract ▾
All authors Kaicong Huang, Weiheng Oh, Jack M. Reilly, Thomas Guggisberg, Ruimin Ke
Computer Vision
Nexus: Native Mesh Generation with Diffusion
Hanxiao Wang, Ying-Tian Liu, Yuan-Chen Guo et al. (+5 more)
Generating high-quality triangle meshes is essential for film, gaming, and interactive 3D applications. Mainstream methods rely on mesh serialization and autoregressive processes, which stuggles in effective inference and is sensitive to error accumulation. In this paper, we present Nexus, a
diffusion method that achieves holistic mesh generation via decoupled vertex and
topology generation. First, we view mesh vertices as sparse voxels organized as an octree and adopt a
diffusion model to generate the vertices in a coarse-to-fine manner. Second, for
topology modeling, we propose Spacetime Interval, as an extension of Spacetime Distance to encode arbitrary edge and face
topology into continuous per-vertex
embeddings. It allows for a global and efficient recovery of complex
topology. We then employ a
diffusion model to generate the continuous
embeddings on the generated vertices. Extensive experiments on the Objaverse and Toys4K
datasets and in-the-wild images demonstrate that our method outperforms state-of-the-art autoregressive and two-stage baselines, effectively circumventing the inherent limitations of sequential mesh modeling. A blind user study from 3D practitioners confirms strong perceptual preference for our results.
Read the full abstract ▾
All authors Hanxiao Wang, Ying-Tian Liu, Yuan-Chen Guo, Qi-Yuan Feng, Zi-Xin Zou, Ding Liang, Biao Zhang, Yan-Pei Cao
Computer Vision
ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning
Yuan Xiao, Can Wang, Xiangyu Kong et al. (+1 more)
While traditional graphics methods often synthesize 3D indoor scenes autoregressively or hierarchically, recent
vision-
language model (VLM)-based generators predominantly adopt a one-shot paradigm where the full layout is planned at once. This one-shot approach often requires global re-optimization or complete reconstruction during interactive editing (e.g., inserting or moving objects) and can lead to physically or semantically poorly organized arrangements. To address these challenges, we propose ThinkBLOX, a VLM-based progressive
reasoning framework that iteratively designs and refines 3D scenes. ThinkBLOX treats layout generation as a state-conditioned, step-by-step reasoningand-action process. To power this, we construct the ThinkBLOX-Data-200K
dataset, containing 224,757 procedural placement pairs annotated with multi-view scene context, explicit Chain-of-Thought (CoT) rationales, and structured JSON layouts. Through supervised fine-tuning (SFT) on this
dataset, the VLM learns to bridge the
reasoning-action gap under incremental updates. Furthermore, recognizing that scene synthesis is inherently a multisolution task where SFT suffers from reward conflict, we introduce Tier-Decoupled GDPO. This reinforcement learning scheme organizes heterogeneous rewards into distinct tiers, stabilizing policy optimization across physical validity, semantic plausibility, and
reasoning-action consistency. Extensive experiments show that ThinkBLOX significantly outperforms recent one-shot and iterative baselines in physical plausibility, semantic
alignment, and interactive editability. Additionally, we show that it supports diverse applications, including both global and local generation and rearrangement of 3D scenes.
Read the full abstract ▾
All authors Yuan Xiao, Can Wang, Xiangyu Kong, Jing Liao
Computer Vision
VGIF-Score: Interpretable and Diagnostic Evaluation of Spatio-Temporal Instruction Following in Video Generation
Songyu Xu, Xin Wang, Qiang Chen et al. (+6 more)
Recent
video generation models (VGMs) have made substantial progress in visual fidelity, yet their ability to follow long, compositional instructions remains insufficiently evaluated. Existing evaluation protocols often rely on
prompts that are short and semantically shallow, with limited atomic constraints and weak spatio-temporal dependencies. They also frequently depend on costly human evaluation or handcrafted
vision pipelines, while providing little diagnostic insight into which instruction constraints succeed or fail. To address this gap, we propose VGIF-Score, a highly automated and interpretable framework for evaluating instruction following in
video generation. VGIF-Score consists of two complementary components: an objective completion branch that parses
prompts into a Spatio-Temporal Directed Acyclic Graph (ST-DAG) and performs dependency-aware QA with short-
circuit diagnostics, and a subjective satisfaction branch that uses instruction-conditioned AutoRubric to assess cinematography, visual purity, motion smoothness, and physics adherence. Together, these components produce a unified score that captures both objective completion and perceptual satisfaction. We instantiate this framework on VGIF-Bench, a
benchmark of 223 long, structurally entangled
prompts paired with approximately 4.3K fine-grained evaluation items. Experiments on 14 proprietary and open-source VGMs across more than 3K generated
videos show that VGIF-Score provides reliable, interpretable, and diagnostically useful evaluation of
video generation instruction following. The code will be available at https://github.com/PRIS-CV/VGIF-SCORE.
Read the full abstract ▾
All authors Songyu Xu, Xin Wang, Qiang Chen, Xinran Wang, Muxi Diao, Yuxuan Zhang, Kongming Liang, Rui Lin, Zhanyu Ma
Computer Vision · Cross-listed from cs.RO
Ishneet Sukhvinder Singh, Dhanoosh Pooranakumaran, Alex Nguyen et al. (+1 more)
A
robot must understand the state of its own body, but a camera sees only part of it. Force and contact leave almost no trace in a single frame, and raw
vision features read force at $R^2$ at or below $0.10$ on every
robot we test. We present Kepler-Encoder-v0.1, a
robot-first
multimodal encoder that treats
robot state as a modality and fuses
vision, proprioception, and force/torque into a single shared latent with a learned-query cross-
attention layer, trained self-supervised by masked cross-modal prediction under the LeJEPA/SIGReg objective. At evaluation only
vision enters, which poses a sharp question. Does fusing state into training make the
vision-only latent carry anything the pixels do not already contain? On the RH20T corpus the answer is yes, precisely where the camera is weakest. On held-out scenes, the
vision-only latent recovers end-effector state, and force in particular, significantly above both raw frozen-ViT features and a compute-matched
vision-only control on every sensored
robot, though absolute force recovery at a single timestep is modest; on motor state, which the camera largely sees, it is statistically tied with the strongest
vision baselines, and it is the only feature whose latent geometry tracks state. A single embodiment-agnostic encoder covers four
robots, and a data-matched control shows this breadth reflects embodiment diversity rather than data volume. The frozen latent is directly useful. Its own cross-modal prediction error is a training-free invalid-state monitor (AUROC $0.90$ on out-of-range states, $0.69$ on scene-swapped states), and a
diffusion decoder (PixNerd) reconstructs the camera frame from the latent, confirming the spatial compression preserves world-state. This report validates the single-timestep case; native-rate temporal fusion is the next step.
Read the full abstract ▾
All authors Ishneet Sukhvinder Singh, Dhanoosh Pooranakumaran, Alex Nguyen, Jia Qi Yip
Computer Vision
DriveFace: A Cross-Spectral Through-Glass Face Dataset for On-the-Move Vehicular Border Control
Anjith George, Luis S. Luevano, Alain Komaty et al. (+3 more)
The continuous growth in cross-border mobility places increasing pressure on existing border control infrastructures, motivating on-the-move biometric authentication, in which travellers are identified directly inside their vehicles at checkpoints. Face recognition is well-suited to this setting, as it can be acquired passively and at a distance. Its development, however, is hindered by the lack of representative
datasets: existing
benchmarks are collected in controlled environments and do not capture the challenges inherent to vehicular acquisition, including motion blur, variable illumination, occlusions, and cross-spectral enrollment. To address this gap, we introduce a
dataset for on-the-move face recognition in border-control scenarios, comprising NIR vehicle-crossing
videos paired with smartphone-based pre-enrollment data. Baseline evaluations with state-of-the-art models show clear performance limitations under these realistic conditions, highlighting the need for dedicated methods to advance the field.
Read the full abstract ▾
All authors Anjith George, Luis S. Luevano, Alain Komaty, Zeina Al Amine, Vidit Vidit, Sebastien Marcel
Computer Vision
TRACE-PCa: Predicting Prostate Cancer Progression from Longitudinal MRI During Active Surveillance
Hongye Zeng, Shreeram Athreya, Dingyuan Dai et al. (+4 more)
Active surveillance (AS) is the preferred strategy for favorable-risk prostate cancer, yet current protocols rely on scheduled repeat biopsies, most of which reveal no progression and are unnecessary. Existing risk-stratification tools operate on single time-point imaging or depend on explicit lesion
segmentation, limiting their ability to capture longitudinal change and excluding patients without an MRI-visible lesion. In this study, we propose an end-to-end temporal and
multimodal model for predicting pathological progression during AS without lesion
segmentation. We encode each serial scan with a pretrained 3D MRI foundation model and introduce a temporal
attention gate that recalibrates the multi-visit features to amplify focal imaging changes associated with progression. The gated imaging representation is then fused with clinical variables in a
multimodal framework to estimate the probability of progression. Validated on a longitudinal AS cohort, our approach consistently outperforms competing baselines and performs comparably to the radiologist assessment representing current clinical practice. It maintains high negative predictive value while achieving higher positive predictive value, demonstrating its potential to safely reduce unnecessary biopsies during surveillance.
Read the full abstract ▾
All authors Hongye Zeng, Shreeram Athreya, Dingyuan Dai, Steve Raman, Leonard Marks, William Speier, Corey Arnold
Computer Vision
DP-BOA: Dirichlet-Process Birth-or-Assign for On-the-Fly Category Discovery
Peiyan Gu, Zixin Teng and Xuming He
On-the-fly category discovery requires deciding for each incoming test sample whether to assign it to an existing category or spawn a new one. Existing methods typically implement this decision through matching-based heuristics, such as radius- or hash-based rules. While effective in practice, these methods usually treat category birth implicitly as a fallback when no existing category matches confidently, rather than as an explicit alternative supported by its own statistical evidence. To address this, we propose DP-BOA, a posterior-predictive decision framework based on an online Dirichlet-process Gaussian mixture model with a Normal-Inverse-Wishart prior. During training, we use labeled data to calibrate a shared NIW prior over category Gaussians and warm-start the known-category posteriors. At test time, for each incoming sample, DP-BOA compares the posterior predictive evidence for assignment to existing categories against the evidence for spawning a new category induced by the DP prior, and then updates category statistics online after the decision. The method captures anisotropic category geometry and naturally adapts decision confidence as evidence accumulates. Across standard OCD
benchmarks, DP-BOA consistently outperforms strong baselines and delivers particularly strong novel-class discovery performance while maintaining competitive known-class accuracy.
Read the full abstract ▾
Computer Vision
Xuanyi Hao, Zuoyuan Zhang, Zhibo Wang et al. (+4 more)
Vision-
Language Models (VLMs) have achieved strong performance in
multimodal understanding, yet remain challenging to deploy on resource-constrained edge devices due to the substantial computational overhead of processing numerous visual
tokens.
Token reduction is a promising direction for accelerating VLMs inference, but existing approaches either rely on
attention maps that are incompatible with modern acceleration frameworks or depend on computationally intensive pairwise similarity comparisons, which undermine scalability and negate their practical benefits in deployment. In this paper, we propose an
attention-free and lightweight
token reduction framework as a plug-and-play module for VLMs, which preserves both important and diverse
tokens to produce a compact visual representation. First, to enable
attention-free importance estimation, we adopt an information-theoretic perspective and quantify
token information using a novel
entropy-based criterion, retaining those with more expressive and less degenerate feature representations. Second, to ensure diverse visual coverage in a lightweight manner, we introduce a transformation-induced consistency signal where similar
tokens yield similar signals, such that sorting by this signal places similar
tokens close to each other and enables stride-based selection to produce a diverse
token set. Extensive experiments across multiple VLMs
benchmarks demonstrate that our framework achieves a favorable accuracy-efficiency trade-off, maintaining competitive performance under aggressive compression.
Read the full abstract ▾
All authors Xuanyi Hao, Zuoyuan Zhang, Zhibo Wang, Xiaoyi Pang, Jiahui Hu, Jiacheng Du, Shuguo Zhuo
Computer Vision
M2P-AD: Memory-to-Prototype Learning with Boundary-aware Score Refinement for 3D Anomaly Detection
Seyoung Jeong, Jong Pil Yun and Sang Jun Lee
3D anomaly detection has recently emerged as an important research topic in computer
vision. Although existing methods have achieved high performance, excessive anomaly responses in normal regions and false positives near object boundaries remain unresolved challenges. To address these challenges, we propose a novel 3D anomaly detection model,
Memory-to-Prototype Anomaly Detection (M2P-AD), which effectively models the distribution of normal features while suppressing excessive anomaly scores in normal regions and false positives near object boundaries. Specifically, we introduce a
Memory-to-Prototype (M2P) module that learns representative prototypes from normal feature
embeddings to preserve important structural information of objects. In addition, a Boundary extraction (BE) module is integrated to identify object boundaries, and a Boundary-aware score refinement (BSR) strategy is applied to recalibrate anomaly scores by incorporating boundary characteristics. The proposed method is evaluated on Real3D-AD, Anomaly-ShapeNet, and MulSen-AD, achieving state-of-the-art performance. Qualitative results demonstrate that excessive anomaly scores in normal regions are reduced and false positives near object boundaries are suppressed, resulting in more accurate and stable anomaly localization. The results indicate that the proposed approach enables more reliable 3D anomaly detection and provides a robust solution applicable to real-world industrial environments.
Read the full abstract ▾
Computer Vision
CASA-SDF: Curriculum-Aware Spatial Adaptation with Curvature-Guided Density for Neural Implicit Surface Reconstruction
Lei Yang, Weiqing Li, Zhiyong Su et al. (+1 more)
Neural implicit representations have emerged as a powerful paradigm for 3D reconstruction. However, high-fidelity indoor surface reconstruction remains a significant challenge, primarily due to the pronounced
geometric heterogeneity of indoor scenes. Large texture-less planar regions typically require stronger regularization to suppress high-frequency artifacts, while thin structures demand sharper, more adaptive representations to mitigate the spectral bias of multi-layer perceptrons (MLPs) and prevent over-smoothing. Existing approaches often rely on spatially indiscriminate prior supervision and a scene-global SDF-to-density transformation, which constrains their ability to balance planar smoothness and detail preservation. In this paper, we propose CASA-SDF (Curriculum-Aware Spatial Adaptation for SDF), a unified framework that addresses this challenge via complementary adaptations of supervision and representation capacity. Specifically, Hybrid Spatially-Adaptive Uncertainty Annealing (SAUA) fuses semantic and photometric uncertainties to construct a pixel-wise curriculum for
monocular prior supervision. This strategy maintains regularization in reliable regions while attenuating unreliable supervision early in training to enable data-driven photometric refinement. Meanwhile,
Curvature-Aware Locally Adaptive Density Transformation (CALADT) progressively modulates the sharpness of the SDF-to-density mapping via a
curvature proxy to enhance the representation of thin structures. Extensive experiments on
benchmark indoor
datasets demonstrate that CASA-SDF improves surface completeness and detail recovery on high-frequency structures, without compromising the stability of planar surfaces.
Read the full abstract ▾
All authors Lei Yang, Weiqing Li, Zhiyong Su, Liang Xiao
Computer Vision
GPOcc++: Unified Sparse Gaussian Occupancy Prediction with Visual Geometry Priors
Changqing Zhou, Yueru Luo, Yulan Guo et al. (+3 more)
Accurate 3D scene understanding is fundamental to
embodied intelligence and autonomous driving, where 3D occupancy provides a unified representation of objects, structures, and free space. However, recovering such a complete volumetric representation from visual observations remains challenging, particularly in occluded and unobserved regions. Visual geometry priors offer strong and generalizable
geometric cues for addressing this challenge, but their outputs are inherently surface-centric, whereas occupancy prediction requires
reasoning about volumetric interiors and free space. To bridge this gap, we introduce GPOcc, which transforms visual geometry priors into occupancy-aware sparse Gaussian representations for efficient and expressive volumetric scene modeling. Building on GPOcc, GPOcc++ models multi-view observations and temporal sequences within a unified framework, allowing spatial and temporal evidence to be handled through the same representation. We further extend GPOcc++ from indoor scenes to outdoor occupancy prediction. Extensive experiments on both indoor and outdoor
benchmarks demonstrate consistently strong performance across both multi-view and temporal settings, together with favorable efficiency and generalization. Code will be released at https://github.com/JuIvyy/GPOcc.
Read the full abstract ▾
All authors Changqing Zhou, Yueru Luo, Yulan Guo, Bing Wang, Jie Qin, Changhao Chen
Computer Vision · Cross-listed from cs.RO
EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal
Alex Brandes, Haig Conti Georges Sajelian, Manthan Patel et al. (+11 more)
Deploying
humanoid robots in unstructured terrain remains an open problem. While classic reinforcement learning struggles with the sheer complexity of real-world interactions, more promising methods leveraging human priors remain limited to models lacking contextual awareness. The restricted motion synthesis is a direct consequence of existing
dataset pipelines failing to capture human-scene sequences in challenging environments. To bridge this gap between
humanoid learning and scene reconstruction, we introduce the
Egocentric Human-Terrain Reconstruction (EgoHTR)
dataset. We develop and open-source a reconstruction pipeline capturing 55 scene-aligned 4D human motion sequences in diverse, complex environments using a multi-sensor setup of
egocentric wearables and a portable 3D scanner. The resulting
dataset comprises over 150k frames, which we evaluate against motion-capture ground truth, demonstrating state-of-the-art accuracy and establishing a rigorous
benchmark for human motion analysis and synthesis. Further, we leverage this data to train perceptive locomotion policies, demonstrating hardware deployment on a Unitree G1 for reconstructed reference motions. Our pipeline enables community-driven
dataset extensions and factors the problem to help researchers build foundational, context-aware
robots that reliably traverse uneven terrain.
Read the full abstract ▾
All authors Alex Brandes, Haig Conti Georges Sajelian, Manthan Patel, Dominik Hollidt, Chenhao Li, Matthias Heyrman, Oliver Hausdoerfer, Manuel Kaufmann, Xi Wang, Jonas Frey, Angela P. Schoellig, Christian Holz, Marc Pollefeys, Marco Hutter
Computer Vision
Bring Music The Horizon: Music-Driven 360$^\circ$ Video Generation
Kai Hsu Tsai, Yong Wei Fu, Hung I Yang et al. (+1 more)
Music visualization offers a powerful way to enhance listeners' understanding and experience of music by translating auditory signals into visual forms. However, most existing approaches either rely heavily on lyrics or generate flat, non-immersive
videos similar to conventional music
videos, which limits their ability to convey the emotional dynamics of music and provide an immersive listening experience. We propose Bring Music The Horizon, an
emotion-aware pipeline for music-driven 360$^\circ$
video generation. Given an input song, our work first estimates its emotional trajectory by predicting valence-arousal values at the level of every four bars. These values are then converted into
emotion-aware visual guidance using EmotiCrafter, and these guidance vectors can be manipulated by the SEGA framework, which provides fine-grained semantic control for keyframe generation. Finally, image-to-
video models are applied to the generated keyframes to synthesize temporally continuous 360$^\circ$
videos for immersive music visualization. Our pipeline generates 360$^\circ$ music visualization
videos that reflect the emotional progression and temporal structure of the input song. We demonstrate its capability using songs from different genres and provide qualitative comparisons with From-Sound-To-Sight, a representative audio-to-visual generation baseline, on our project page at https://etoile-et-toi-mp3.github.io/BMTH_Project_Page/.
Read the full abstract ▾
All authors Kai Hsu Tsai, Yong Wei Fu, Hung I Yang, Yu-Chih Chen
Computer Vision
HIVE-3D: Hierarchical Voxel Enhancement for High-Quality 3D Scene Generation
Bin Zang, Wenting Zheng, Xiaoliang Luo et al. (+8 more)
Recently, a line of works can generate impressive 3D objects from a single image, but they are limited by restricted representation resolution, making them unsuitable for 3D scene generation. In this work, we introduce HIVE-3D, a novel method for high-quality 3D scene generation based on hierarchical voxel enhancement framework. Specifically, given a single scene image as input, we first produce a coarse initial scene, then introduce image
segmentation and
attention-based
retrieval to align 2D image components with 3D scene components. Subsequently, we organize these scene relations into a hierarchical component tree, where nodes closer to the leaves denote finer-grained components. Finally, we propose a voxel super-resolution model that generates refined voxels for the target instance while maintaining strong consistency with the coarse voxels. Equipped with this model, we perform coarse-to-fine hierarchical super-resolution on images and voxels for each component, producing a high-resolution and high-quality 3D scene. Extensive experiments demonstrate that our method significantly outperforms previous approaches, achieving state-of-the-art performance.
Read the full abstract ▾
All authors Bin Zang, Wenting Zheng, Xiaoliang Luo, Zhiyuan Fang, Shi Li, Lvchun Wang, Wei Yu, Yi Zhao, Tian Xie, Yuchi Huo, Rengan Xie
Computer Vision
LPM: Industrial-Scale Generative Video Restoration
Bichuan Zhu, Fulin Li, Jiachao Gong et al. (+14 more)
We present the Large Processing Model (LPM), a
diffusion-based generative framework for photorealistic
video restoration under complex, in-the-wild degradations. To our knowledge, LPM is the first generative
video restoration model deployed at industrial scale. LPM addresses the diverse degradations in user-generated content (UGC) through a unified system encompassing large-scale data engineering, foundation-model training, and efficient inference. Its enhanced architecture, progressive training strategy, and temporal-pyramid inference mechanism jointly enable high-fidelity, temporally consistent restoration of arbitrarily long
videos across the broad content distribution encountered on UGC platforms. LPM has been deployed in production at Kuaishou, where
videos processed by the model account for approximately 45% of total viewing time, delivering consistent improvements across key quality-of-experience metrics. Beyond perceptual enhancement, LPM delivers substantial system-level benefits: at comparable perceptual quality, it reduces bitrate by 20% relative to Kuaishou's in-house codec, yielding annual bandwidth cost savings on the order of hundreds of millions. Its low serving cost also enables integration into products such as Kling, demonstrating that generative restoration can be practical, scalable, and cost-effective for large-scale
video processing.
Read the full abstract ▾
All authors Bichuan Zhu, Fulin Li, Jiachao Gong, Jinhua Hao, Kai Zhao, Kun Yuan, Pengcheng Xu, Qiang Wang, Qiao Mo, Yanlong Yuan, Yizhen Shao, Yuxiao Hu, Zixi Tuo, Ming Sun, Chao Zhou, Bin Chen, Bin Yu
Computer Vision
2D Rotary Position Embedding for Scene Text Recognition with Transformers
Zobeir Raisi
Scene Text Recognition (STR) remains challenging due to the diversity of text appearances, including
curvature, rotation, and perspective distortion. Recent
Transformer-based approaches perform well but usually rely on one-dimensional positional encodings that ignore the 2D spatial structure of text images. Axial 2D extensions of Rotary Position
Embedding (RoPE) exist for
vision Transformers, but they assume roughly square, isotropic image content and apply the rotation only within encoder self-
attention. Scene text violates both assumptions: crops are markedly anisotropic, and STR models are encoder-decoder, so the decoder must relate its queries to the encoder's 2D layout through cross-
attention. We introduce 2D-RoPE-STR, which adapts axial 2D-RoPE to this setting through (1) an anisotropic row/column dimension allocation matched to the aspect ratio of text, and (2) an extension of the rotary coupling into encoder-decoder cross-
attention, letting autoregressive decoding steps attend to encoder
tokens by their 2D layout, a setting not addressed by prior encoder-only formulations. Both changes are essentially parameter-free and require no architectural redesign beyond the positional-encoding module. We further introduce a diagnostic protocol (a controlled ablation pair isolating only the positional encoding, an image-level net-win disagreement analysis, and encoder
attention visualization) that identifies where and why relative 2D position helps: curved, rotated, and perspective-distorted layouts where reading order departs from a straight horizontal line. On six standard
benchmarks (IIIT5K, SVT, ICDAR 2013, ICDAR 2015, CUTE80, SVTP), gains concentrate on exactly these irregular layouts, with ablations isolating each design choice against 1D RoPE and 2D sinusoidal and learnable alternatives.
Read the full abstract ▾
Computer Vision
TreeSRNF: Square-Root Normal Fields for Generative Modelling of the Geometric and Structural Variability in Tree-like 3D Objects
Tahmina Khanam, Hamid Laga, Mohammed Bennamoun et al. (+4 more)
We introduce a novel mathematical framework for analyzing and generating complex tree-shaped 3D objects, such as botanical trees and plants, which deform both in their 3D geometry and branching structure. Unlike previous works, which either consider only the skeletal structure of tree-like objects or approximate their 3D geometry using branch thickness, the proposed framework accurately models both the 3D geometry of the tree branches and the way they are interconnected. In this paper, we first generalize the Square Root Normal Fields (SRNF) representation, originally proposed for the statistical analysis of genus-0 surfaces, to tree-shaped 3D objects. We then treat tree-shaped 3D objects as points on a novel Riemannian tree-shape space equipped with a novel Riemannian metric that measures the amount of surface bending and stretching, and structural changes one needs to apply to one 3D tree-shape to align it with another. This way, deformations become trajectories in this novel tree-shape space. We analyze the theoretical properties of this novel tree-shape space and the corresponding metric and develop
algorithms for computing point-wise and branch-wise correspondences and geodesic paths between complex 3D trees. We finally show how to use these building blocks for (1) computing statistical summaries, \ie means and modes of variation, of collections of tree-shaped 3D objects, and (2) synthesizing novel tree-shaped 3D objects by sampling from probability distributions fitted to a population of tree-shaped 3D objects. We demonstrate the performance and utility of the proposed framework on real and synthetic plants and botanical trees and show that it significantly outperforms the state-of-the-art.
Read the full abstract ▾
All authors Tahmina Khanam, Hamid Laga, Mohammed Bennamoun, Guanjin Wang, Ferdous Sohel, Farid Boussaid, Anuj Srivastava
Computer Vision
GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understanding
Hao Li, Han Fang, Zixin Pan et al. (+8 more)
Although
multimodal large
language models (MLLMs) have achieved remarkable progress, understanding 3D spatial relationships from 2D images remains a critical challenge. Existing methods primarily rely on symbolic text
tokens, which inherently lack the fidelity to represent continuous
geometric information. While recent methods use latent representations to enhance
reasoning, relying on a single latent type cannot adapt to the diversity of spatial tasks, leading to misalignment in complex
geometric scenarios. To address these limitations, we propose GeoAnchor, an interleaved text-latent
reasoning framework. GeoAnchor decomposes 3D spatial information into three complementary components: position latents for object grounding, direction latents for relational orientation, and geometry latents for scene structure. These components are recombined in a structured space to construct local evidence while capturing global context, enabling dynamic and interpretable
reasoning. Furthermore, we introduce a collaborative training strategy that guides the model from local spatial perception to comprehensive 3D understanding. Extensive experiments on diverse and complex 3D
reasoning tasks demonstrate that GeoAnchor outperforms the state of the art, validating its effectiveness and generalization capabilities.
Read the full abstract ▾
All authors Hao Li, Han Fang, Zixin Pan, Xin Wei, Hongbo Sun, Jinglin Xu, Zhiyu Lin, Ye Yuan, Zhongjiang He, Yu Yu, Hao Sun
Computer Vision
Symbiosis-Inspired Knowledge Distillation for Incremental Object Detection
Mingyue Zeng, De Cheng, Zhipeng Xu et al. (+3 more)
Incremental object detection (IOD) aims to extend detectors to new categories while retaining previously acquired knowledge. Existing methods often adopt a class incremental learning perspective, separating feature spaces to sharpen decision boundaries. However, this separation-oriented paradigm may overlook object symbiosis in detection, where co-occurrence and occlusion introduce spatial and semantic dependencies that benefit from shared representations. Ignoring these dependencies distorts the shared representations, exacerbates confusion between old and new classes, and accelerates catastrophic forgetting. To address this, we propose Symbiosis-Inspired Knowledge
Distillation (SIKD), which explicitly leverages object symbiosis at two complementary levels. Spatial Symbiosis
Distillation (SpSD) focuses on symbiotic regions where the old model responds with high overlap to objects in the new task. It preserves generalizable old class cues, suppresses class-specific bias and redundancy, and distills the refined evidence to the new model at matched spatial locations with slot-aligned supervision. Semantic Symbiosis
Distillation (SeSD) maintains class level structure by forming confidence weighted prototypes for old classes and aligning their inter class soft ranks over the old class logits, which stabilizes the semantic
topology during adaptation. Extensive experiments demonstrate the effectiveness and superiority of the proposed method.
Read the full abstract ▾
All authors Mingyue Zeng, De Cheng, Zhipeng Xu, Huaijie Wang, Nannan Wang, Xinbo Gao
Computer Vision · Cross-listed from cs.RO
Learning Physics-Guided Residual Dynamics for Deformable Object Simulation
Shivansh Patel, Kaifeng Zhang, Sanjay Pokkali et al. (+2 more)
Simulating deformable objects is essential for a wide range of robotic manipulation applications, yet accurately predicting their dynamics remains challenging. We propose Physics-Guided Residual Dynamics (PGRD), a hybrid simulation framework that combines the advantages of physics-based and learning-based approaches. Specifically, PGRD combines an optimizable spring-mass simulator as a backbone with a learned neural network that predicts residual corrections to the physics-based predictions. We adopt a velocity-based formulation to ensure stable simulation and a sliding-window
transformer architecture to capture temporal dependencies. We show that PGRD produces more accurate results than both purely physics-based and learning-based methods on a set of diverse real-world deformable objects. We further demonstrate the utility of PGRD in two applications: manipulation planning via Model Predictive Control, including a
language-conditioned setting with a generated goal image; and interactive simulation via action-conditioned
video prediction by 3D Gaussian Splatting.
Read the full abstract ▾
All authors Shivansh Patel, Kaifeng Zhang, Sanjay Pokkali, Svetlana Lazebnik, Yunzhu Li
Computer Vision
DreamSat-Pose: Spacecraft Pose Estimation from Single-View 3D Reconstructions and Learned 2D-3D Feature Matching
Josiane Uwumukiza, Jocelyn Zhao, Giovanni Lavezzi et al. (+5 more)
6-DoF pose estimation is a critical task in autonomous rendezvous and proximity operations. In the case of an unknown target, this task becomes challenging as it shall be paired with the reconstruction of the target shape model. In this article, we propose a novel framework for single-shot shape and pose estimation of unknown spacecraft objects. Given a single image, we first reconstruct a 3D shape model of the target, then estimate the relative six-degrees-of-freedom pose by learning dense 2D-3D correspondences. The image features are extracted using a frozen DINOv3
vision transformer, while the
geometric features are computed from the reconstructed point cloud using a trainable dynamic graph convolutional neural network encoder. A dual-stream
transformer matcher refines descriptors through alternating self- and cross-
attention, producing soft correspondences that are passed to a Perspective-$n$-Point solver for pose recovery. We evaluate the method on the SPE3R
dataset and consider FoundationPose as a representative baseline for current state-of-the-art capabilities. Results show reliable pose estimates achieving 0.157 degrees mean pointing error using only a single image and reconstructed geometry, demonstrating strong generalization to unseen spacecraft.
Read the full abstract ▾
All authors Josiane Uwumukiza, Jocelyn Zhao, Giovanni Lavezzi, Giacomo Battaglia, Paolo Panicucci, Minduli C. Wijayatunga, Victor Rodriguez-Fernandez, Richard Linares
Computer Vision
CLIP-Guided Label-Free Discriminative Region Scoring for Fine-Grained Classification
Yujie Zhu
Recent
vision models such as CLIP and SAM enable training-free
segmentation and semantic encoding for fine-grained classification. A common approach is to compare the representations of segmented image regions with the text
prompt embeddings of the corresponding labels. However, it remains unclear how different local regions and CLIP-based scoring strategies affect the selection of discriminative evidence, especially when ground-truth labels are unavailable. In this paper, we propose a unified CLIP-guided label-free region scoring framework for fine-grained classification. The framework evaluates cosine similarity-based, margin-based, and
entropy-based scoring strategies using both SAM-generated masks and random crops, and introduces two label-free pseudo-label variants based on global image
embeddings and local region
embeddings. We conduct experiments on five fine-grained classification
datasets to systematically compare different region generation methods and scoring strategies. The results show that Soft Negative Margin scoring achieves the strongest performance, and pseudo-label scoring closely approximates true-label performance. Although SAM produces semantically meaningful masks, random-crop-based pseudo-label scoring consistently outperforms SAM-based scoring across all
datasets, suggesting that random crops preserve surrounding information and provide more stable semantic context when pseudo-labels are noisy. In addition, SAM masks benefit from aggregating
embeddings from all regions, whereas random crops tend to perform better with a smaller top-k subset. These findings provide new insights for fine-grained classification.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment
Dwip Dalal, Shivansh Patel, Chahit Jain et al. (+8 more)
Finetuning a pretrained
vision-
language model (VLM) on
robot demonstrations via behavior cloning (BC) has become the standard recipe for
vision-
language-action (VLA) policies. However, BC finetuning progressively overwrites the pretrained representations that support visual and semantic generalization. Co-training on web image-text data, a common remedy, applies
language and action losses to separate observations, leaving VLAs with
language-action misalignment that standard manipulation
benchmarks do not expose. We propose Anchor-Align, which augments BC with two objectives:
Vision-
Language Anchoring distills layer-wise representations from a frozen VLM copy to prevent this drift, while
Language-Action
Alignment converts each action target into a discrete motion-direction label and jointly trains
language and action prediction on the same
robot observation. We conduct real-world evaluations across eight manipulation settings on single-arm xArm7 and bimanual YAM
robots, using two VLA architectures with regression and flow-matching action heads. Across these settings, Anchor-Align consistently improves over BC on novel targets, layouts, and motion-sensitive bimanual tasks requiring coordinated control. At scale in simulation, we demonstrate consistent improvements on OOD perturbations, perceptual robustness, and long-horizon control across LIBERO-PRO, LIBERO-Plus, and CALVIN, respectively, suggesting that preserving pretrained representations and effective action learning are not fundamentally at odds. Project page: anchoralignvla.github.io
Read the full abstract ▾
All authors Dwip Dalal, Shivansh Patel, Chahit Jain, Jeonghwan Kim, Utkarsh Mishra, Yuchen Song, Alex Baratian, Hyeonjeong Ha, Heng Ji, Svetlana Lazebnik, Unnat Jain
Computer Vision
ScanFocus: A Coarse-to-Fine Framework for Spatio-Temporal Video Grounding
Kai Chen, Ming Dai, Wenxuan Cheng et al. (+1 more)
Spatio-Temporal
Video Grounding (STVG) aims to retrieve the visual trajectory of a specific object from a
video stream as described by a natural
language expression. However, most advanced methods struggle to balance global context modeling with precise boundary localization. Due to the prohibitive computational costs of processing long
videos, these approaches typically resort to low-rate temporal downsampling and implicit motion modeling. This inevitably suppresses high-frequency boundary cues and neglects the explicit inter-frame dependencies required for precise boundary delineation. To address these limitations, we present ScanFocus, a novel coarse-to-fine framework that decouples the STVG task into a global spatio-temporal scan and a local boundary focus. Specifically, we utilize a unified
vision-
language fusion encoder combined with a lightweight Deformable Semantic-Motion Fusion module to efficiently align
multimodal features and generate coarse proposals. To recover the suppressed fine-grained details, we introduce the Semantic-Guided Temporal Aggregator (SGTA) in the refinement stage. By densely sampling around coarse boundaries, SGTA explicitly models short-term temporal interactions under semantic guidance, capturing rapid motion changes for precise timestamp regression. Extensive experiments on three widely used
benchmarks demonstrate the performance superiority of our proposed method over previous approaches. Code will be released at https://github.com/TenMinutes209/ScanFocus.
Read the full abstract ▾
All authors Kai Chen, Ming Dai, Wenxuan Cheng, Wankou Yang
Computer Vision
MultiAnimate: A Unified Framework for Controllable Multi-Character Animation
Zhongyi Zhang, Guangyuan Wang, Li Hu et al. (+6 more)
Recent advances in generative models and technological innovations have significantly addressed the fundamental challenges of character image animation. However, existing approaches predominantly focus on character animation from a single reference image, substantially limiting their applicability in scenarios such as multiple character interaction animation. To fill this gap, this paper introduces MultiAnimate, a comprehensive framework that enables concurrent animation of multiple characters within a shared environment while preserving both identity consistency and spatial relationships. The framework achieves these objectives through multiple well-designed mechanisms. First, we incorporate an identity-specific reference net that enables appearance extraction from multiple reference images, distinguishing MultiAnimate from existing approaches constrained to single reference inputs. Second, we implement an identity-aware pose encoder to address the character-pose binding challenge, wherein an
attention mechanism enables the network to accurately differentiate and process multiple pose sequences during generation. Third, we introduce an interaction guider module that enhances the framework's capability to handle complex inter-character interactions by leveraging character-specific mask information, serving as an optional component that refines the pose sequences. Extensive experiments and ablation analyses demonstrate our framework's superiority in multiple character animation, particularly in scenarios involving complex motion sequences.
Read the full abstract ▾
All authors Zhongyi Zhang, Guangyuan Wang, Li Hu, Wenbo Zhou, Peng Zhang, Tianyi Wei, Weiming Zhang, Bang Zhang, Nenghai Yu
Computer Vision
AnomExpert: Identifying and Selecting Anatomical Planes for Prenatal Ultrasound Anomaly Diagnosis
Jian Wang, Yang Yang, Ziheng Pan et al. (+4 more)
Life-limiting congenital anomalies require accurate prenatal diagnosis for appropriate clinical decision-making. Prenatal ultrasound (US) examinations involve multiple anatomical planes, and diagnosis depends on identifying anatomical planes and selecting diagnostically relevant planes for each anomaly. Existing automated methods either rely on plane-level annotations or aggregate heterogeneous images without explicitly modeling these diagnostic capabilities. We propose AnomExpert, a prototype-driven framework for prenatal US anomaly diagnosis using only case-level supervision. AnomExpert introduces learnable plane prototypes to organize unordered images into latent representations corresponding to anatomical planes without requiring plane annotations. A disease-aware sparse selection mechanism further selects diagnostically relevant planes for each anomaly. Experiments on a multi-center
dataset of 3,654 cases show that AnomExpert consistently outperforms nine representative multi-instance learning methods. Using a ViT-small backbone, it achieves 86.9% accuracy and 84.2% F1-score while maintaining parameter efficiency. These findings indicate that modeling anatomical plane identification and disease-specific plane selection improves weakly supervised multi-plane prenatal US anomaly classification. The code is available at https://github.com/TIanCat/AnomExpert.
Read the full abstract ▾
All authors Jian Wang, Yang Yang, Ziheng Pan, Xiliang Zhu, Yuhan Zhang, Yanfeng Zhou, Dong Ni
Computer Vision
FM$^2$: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging
Shengchao Chen and Ting Shu
Building foundation models for medical imaging requires pooling data across institutions, yet
privacy regulations prohibit centralized aggregation. Existing
Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM$^2$, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for
vision-
language alignment. FM$^2$ equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality
Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable $O(1/\sqrt{T})$ convergence and generalization guarantees. FM$^2$ further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to
Federated Medical VQA. Experiments on our MIMH
benchmark (classification and CEL) and real-world medical VQA
datasets confirm consistent superiority over state-of-the-art
federated baselines and strong out-of-modality generalization across all three tasks.
Read the full abstract ▾
Computer Vision
RoughNet: Mapping Arctic Sea Ice Roughness Using Diffusion-Based Super-Resolution of Satellite Imagery
Tessa Cannon, Michel Tsamados, Petru Manescu et al. (+5 more)
Accurate estimation of landfast sea ice roughness is critical for
climate modeling and safe Arctic over-ice travel, yet existing approaches rely on costly airborne surveys or sparse in-situ measurements, limiting spatial coverage and operational scalability. Here we show that high-resolution sea ice topography can be reconstructed directly from optical satellite imagery using a conditional
diffusion framework. Our approach, RoughNet, learns to map 10 m Sentinel-2 multispectral images to locally normalized 1 m surface elevation residual fields, enabling fine-scale roughness characterization from widely available satellite data. Trained on airborne
LiDAR data from two Arctic regions and evaluated on an unseen third Arctic region, the model generalizes across diverse ice conditions and partially reproduces small-scale topographic structure. The best-performing model achieves an out-of-domain root mean squared error of 9 cm while preserving the statistical and spectral properties of the underlying roughness field. These results demonstrate that generative
diffusion models can recover physically meaningful surface structure from optical imagery alone, providing a scalable pathway for high-resolution sea ice mapping and roughness estimation in data-sparse environments.
Read the full abstract ▾
All authors Tessa Cannon, Michel Tsamados, Petru Manescu, Thomas Newman, Christian Haas, Veit Helm, Weibin Chen, Randall Scharien
Computer Vision
DiffGI: Differentiable Geometry Images for High-Fidelity Thin-Shell 3D Generation
Eungjune Shim, Hansol Lee and Eunjung Ju
Existing 3D generative models predominantly rely on implicit volumetric representations, which enforce watertight
topology and struggle to represent thin-shell and non-
manifold geometries such as garments. Geometry image-based approaches offer a surface-centric alternative, but existing methods rely on discrete binary occupancy maps whose resolution-dependent boundary encoding causes staircase artifacts and information loss upon downsampling, while surface reconstruction remains a non-differentiable post-processing step disconnected from the learning pipeline. To address this, we propose Differentiable Geometry Image (DiffGI), an end-to-end 3D-to-2D mapping framework that seamlessly integrates surface representation and
geometric optimization. DiffGI replaces binary maps with a continuous 2D Truncated Signed Distance Function (TSDF), which encodes boundary position at subpixel precision within a fixed grid resolution, eliminating resolution-dependent staircase artifacts even under aggressive downsampling. Building on this continuous field, we introduce a differentiable Marching Squares
algorithm based on analytical linear interpolation, allowing gradients from 3D surface losses to propagate back to the 2D latent space. Leveraging this differentiable pipeline, we train a DiffGI-VAE augmented with a geometry-aware normal rendering loss to compress complex 3D surfaces into an ultra-compact 32X32 latent space, and instantiate a
transformer-based latent
diffusion model with a flow-matching objective on top of this space for conditional 3D generation. Extensive experiments on garment and object
datasets demonstrate that our method achieves superior reconstruction fidelity and boundary precision compared to prior geometry-image and voxel-based approaches, while requiring significantly fewer computational resources.
Read the full abstract ▾