Computer Vision · Cross-listed from cs.RO
Physical Kernel: Structured Visual Latents for Dark Manipulation
Jinting Hang, Hong Li, Zhenhui Cai et al. (+2 more)
We study dark manipulation: after a brief lit Write encodes z0 = Enc(rgb), a policy pi(z) and open-loop dynamics f(z,a) complete contact-rich skills without further pixels (dark_f). On ManiSkill StackCube (n=160; seed packs 0/1000), dark_f attains 68.1% stacked on the five-rung chain (near_A -> grasped -> lifted -> on_B -> stacked), compared with 35.6% for per-step lit_reenc and 0% for freeze/encode_black. On a shared Write->HOLD protocol (n=40), occlusion and camera-aligned GT contact-neighbor masks drive lit lift from 43% to 0%, while dark_f holds 82.5%; shuffling actions inflates dynamics MSE by ~9.4x; write-time appearance shifts break encoding (night: 0% stacked), yet the same shifts during HOLD leave dark_f lift unchanged; Write length Tw is flat once the stop phase is reached, while earlier stops and write-time blur/JPEG sharply cut stacked. A dedicated pi_write reaches 35%
vision-budget stacked (n=80); matched Dreamer-style/pixel nulls without privileged geom stay at 0%. Privileged state-RSSM MPC reaches ~35% stacked with 9D dark observations – a stronger-observation null, not a matched visual baseline.
Read the full abstract ▾
All authors Jinting Hang, Hong Li, Zhenhui Cai, Zhihao Zhao, Jian He
Computer Vision
AAMBERS-UAV: Acquisition-Aware Multimodal Backbone Evaluation and Ranking for UAV Weedy Rice Segmentation
Tarek Rahman, Nazim-E-Alam, Md Kishor Morol et al. (+1 more)
UAV image collections contain spatially and temporally related frames, yet semantic-
segmentation benchmarks commonly split them at image level. Such splitting can place samples from one acquisition in both model development and testing, obscuring transfer to a genuinely new survey. Using the 734-sample WeedyRice-RGBMS-DB, we fix a 124-image target-acquisition test set and compare two protocols with identical train, validation, and test counts: target-held-out, which excludes the target acquisition from development, and target-exposed, which admits its remaining images. SegFormer-B0 is evaluated with RGB, four-band multispectral (MS), and seven-channel RGB+MS input over two fixed-split seeds. RGB is strongest under complete acquisition holdout ($0.7317\pm0.0201$ IoU), whereas RGB+MS becomes strongest after target exposure ($0.7822\pm0.0269$). A fixed-split U-Net/ResNet18 replication confirms positive exposure gains for all three inputs, but retains RGB as the best modality under both protocols. Acquisition exposure therefore increases measured performance across both evaluated backbones, while its effect on modality ranking is architecture-dependent. A supplied-split audit reveals strong near-sequential dependence, and corruption tests show that early fusion is substantially more sensitive to RGB–MS displacement than to moderate radiometric scaling. These results support acquisition-aware same-test evaluation as a necessary complement to ordinary image-level splitting in
multimodal UAV benchmarks. The code and supporting the findings of this study will be publicly released upon acceptance of the paper.
Read the full abstract ▾
All authors Tarek Rahman, Nazim-E-Alam, Md Kishor Morol, Jannatun Noor
Computer Vision
GeoContext: One Context Ladder, Two Failure Modes in Vision-Language Geolocation: Flat Reliance on User-Provided Location Context and False Confirmation of Location Claims
Yifan Zhang and Kai Wang
Visual geolocation
benchmarks typically ask a model where an image was captured without accounting for the location context that users often provide. We introduce GeoContext, a resource supporting two complementary tasks: GeoHint, open-ended localization given a true but coarse location hint, and GeoVerify, binary verification of whether an image was taken within 150 m of a claimed place. GeoContext constructs a context ladder by stratifying nearby reference points according to distance and referenceability, allowing the image to remain fixed while the supplied context varies. The
benchmark covers 109 sites in 30 cities and evaluates five
vision-
language models using 21,933 GeoHint responses and 6,270 GeoVerify responses. Our evaluation reveals three main patterns. First, hint repetition varies by only 1.5 percentage points across referenceability tiers and by less than 3 points across distance bands, while the resulting localization error increases steadily with hint distance. Second, behavior depends strongly on no-context performance: at sites with low no-context accuracy, the median ratio between localization error and hint distance is approximately 1.00, whereas at higher-accuracy sites it ranges from 0.24 to 0.69. After correcting for bias introduced by the site grouping procedure, only one of the five models retains a negative accuracy estimate when given a nearby hint. Third, in GeoVerify, no model reaches d' = 1 for decoys immediately beyond the 150 m tolerance. Model rankings also change when sensitivity is separated from response bias, and 83.8% of false acceptances are reported with confidence of at least 0.8. We release the
benchmark, construction pipeline, audit decisions, and scoring code.
Read the full abstract ▾
Computer Vision
Gaussian Linear Functional Manifold Method for Massive Point Cloud Data
Hong Zhao, Tonglin Zhang, Baijian Yang et al. (+2 more)
Reconstructing continuous terrain
manifolds from massive, unstructured airborne
LiDAR point clouds remains challenging in complex Wildland-Urban Interface (WUI) environments, where deep neural networks require costly point-wise annotations and nonparametric surface reconstruction methods often lack structural interpretability. This paper introduces the Gaussian Linear Functional
Manifold (GLFM), a physics-informed statistical framework that represents continuous surface topography using deterministic linear functional bases while modeling microscale diffuse
laser backscatter as an isotropic Gaussian process. To avoid the quadratic computational cost of exact constrained maximum likelihood estimation, we develop an algebraic singular value decomposition (SVD) rank-reduction
algorithm that enables linear-time parameter estimation and closed-form quadric classification. Evaluated on 35.2 km^2 of real-world aerial
LiDAR data, GLFM automatically filters ground points and extracts morphological features, achieving an adjusted Rand index (ARI) of 0.9933 against field-verified ground truth and outperforming four leading baselines while maintaining an out-of-core
memory footprint. The framework provides a rigorous, interpretable, and scalable foundation for large-scale point cloud analytics.
Read the full abstract ▾
All authors Hong Zhao, Tonglin Zhang, Baijian Yang, Jin Wei-Kocsis, Songlin Fei
Computer Vision
SeRV: Semantic-Aligned Residual Vector Quantization for American Sign Language Generation
Hongyu Wu, Xu Wu, Tianhao Wu et al. (+4 more)
American Sign
Language (ASL) generation remains challenging due to limited paired text-ASL motion data and the difficulty of learning motion representations both precise for reconstruction and predictable from linguistic input. Existing methods rely on motion tokenizers optimized for reconstruction, without explicit semantic supervision from paired text. As a result, the learned
tokens remain limited in supporting semantically consistent and fine-grained ASL motion generation. To address this limitation, we propose SeRV (Semantic-Aligned Residual Vector Quantization), a semantic-aligned RVQ tokenizer for ASL generation. SeRV learns a semantically structured residual
token space by combining global-level motion-text
alignment with
token-level text-conditioned supervision. Building on this tokenizer, a Hierarchical GPT predicts residual motion
tokens in a coarse-to-fine manner, generating structurally coherent and semantically aligned 3D ASL motion. We further construct a large-scale reconstructed 3D ASL motion-text
benchmark by recovering paired 3D motion from OpenASL
videos. Experiments on How2Sign and OpenASL show that SeRV outperforms six state-of-the-art baselines, with user studies assessing reconstruction quality and semantic fidelity. Samples of our reconstructed 3D ASL motion are available on our project page: https://signvideos.github.io/Demo/.
Read the full abstract ▾
All authors Hongyu Wu, Xu Wu, Tianhao Wu, Jiawei Yu, Phuc Nguyen, Jian Liu, Yi Wu
Computer Vision
RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitives
Chong Zeng, Yue Dong, Pieter Peers et al. (+2 more)
We present 'RenderFormer-V2', a unified learned
transformer-based neural rendering model, complementary to modern physics-based rendering systems, that can handle diverse light-transport effects such as caustics, volumetric scattering, environment lighting, textured and displaced surfaces and out-of-distribution materials without per-scene training or specialized code. RenderFormer-V2 models global light transport as a sequence-to-sequence transformation. Following its predecessor, RenderFormer-V2 also employs a two stage process: a view-independent stage that resolves intra-scene primitive to primitive transport, and a view-dependent stage that transforms the internal neural scene representation into image pixels. Different from RenderFormer, our model employs a novel combined windowed-
attention and rendering-informed
attention sink in the view-independent stage to improve scalability while maintaining render accuracy. To further improve versatility, RenderFormerV2 supports heterogeneous scene primitives, including environment maps and participating media, and it employs a material encoding independent of the underlying surface reflectance model that encodes material appearance via a novel neural
embedding. We demonstrate the versatility of RenderFormer-V2 on a variety of scenes and perform an extensive ablation of the improved
attention mechanism.
Read the full abstract ▾
All authors Chong Zeng, Yue Dong, Pieter Peers, Lvmin Zhang, Maneesh Agrawala
Computer Vision
Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance
Samir Char, Carles Domingo-Enrich and Randall Balestriero
Contrastive
Language-Image Pretraining (CLIP) is a building block of many machine learning applications. Scaling laws have guided resource allocation for large-scale training, yet prior work treats total CLIP model size as a single variable, without exploring how the capacity split between encoders impacts downstream performance. Here, we train multiple CLIP models with different
vision and text encoder sizes, revealing that for most
vision encoders, there is an optimal text encoder size beyond which zero-shot performance degrades—even as total parameter count increases. Exploiting this behavior yields efficient configurations that match the zero-shot performance of the standard ViT-B/16 architecture with up to 55% fewer parameters. We further show that this degradation stems from overfitting induced by the oversized text encoder, and that using modality-specific weight decay coefficients not only recovers but improves performance across all degraded configurations. A
geometric analysis reveals a trade-off in which scaling the text encoder improves
embedding uniformity but worsens cross-modal
alignment; we further show that these metrics are predictive of zero-shot performance. We hope these findings motivate CLIP architectures and training methods that counteract this degradation, a prerequisite for scaling CLIP reliably and efficiently.
Read the full abstract ▾
Computer Vision
Full-Page Optical Music Recognition of Handwritten Monophonic Scores
Adrian Rosello, Antonio Ríos-Vila, David Rizo et al. (+1 more)
Full-page end-to-end Optical Music Recognition seeks to transcribe entire music pages directly into symbolic notation, avoiding the limitations of traditional pipelines that rely on accurate staff
segmentation. Recent
Transformer-based architectures have achieved strong performance on typeset scores, relying on large-scale synthetic data for pretraining. However, their applicability to handwritten music remains largely unexplored. In this work, we study full-page transcription on handwritten monophonic collections and analyze the impact of synthetic pretraining in this setting. To investigate which factors are most relevant during pretraining, we introduce a generator capable of producing visually coherent full-page scores in both typeset and handwritten styles. Experiments on three real handwritten
datasets provide a comparative evaluation of several full-page pipelines and different synthetic pretraining strategies. The results suggest that the benefits of synthetic pretraining are primarily associated with learning structural layout conventions rather than with visual similarity to the target handwriting.
Read the full abstract ▾
All authors Adrian Rosello, Antonio Ríos-Vila, David Rizo, Jorge Calvo-Zaragoza
Computer Vision
Srijika: OpenType-Layout-Reusing Font Restyling for Nine Indic Scripts
Anil Pai
We present Srijika, a system for producing installable OpenType fonts for nine Brahmic scripts: Devanagari, Tamil, Bengali, Telugu, Kannada, Malayalam, Gujarati, Gurmukhi, and Odia. Rather than generating fonts from scratch, Srijika restyles glyph outlines from shaping-complete template fonts. It preserves the template's cmap and GSUB closure and its GPOS data under a documented metric policy, making every output a complete font by construction. This addresses a central challenge of Indic font generation: hundreds to thousands of conjuncts, half forms, and matra variants must remain mutually consistent under OpenType shaping. Srijika produces 66 TTFs: 57 curated presets and nine open-vocabulary showcase fonts. All pass the OpenType Sanitizer, while HarfBuzz and CoreText reproduce the template glyph-ID sequences on conjunct-heavy probes. A full-closure audit covering 80,915 glyphs and 54,812 anchors quantifies metric changes. Natural-
language style selection uses Lipika, a
retrieval index over approximately 650 open-license font families. A reference-conditioned latent
diffusion model redraws template glyphs in the selected style, followed by content gating, harmonization, and shaped-cluster verification with fallback to template outlines. We evaluate against no-learning baselines. On
diffusion-training-family-held-out SSIM gates, template copying outperforms generation on 50 of 56 faces. Style movement is measurable only with an internal same-model
embedding whose training corpus includes the held-out families, so these results require caution. A learned baseline, independent style metric, and human study are outside this report's scope. Our contributions are the layout-reusing formulation and pipeline, its nine-script audit and
benchmark, and a negative-results catalogue covering failed conditioning, objective choices, and data-hull limits of reference-guided restyling.
Read the full abstract ▾
Computer Vision
Facial Age Estimation for Age Fraud Detection in National ID Systems
Sharib Athar, Arka Koner, Chetan Naik et al. (+4 more)
Identity fraud during biometric enrollment and updates remains a major challenge for large-scale national identity systems. A common fraud vector is misrepresenting one's age to access age-restricted services or welfare schemes. In this work, we present SwinAge, a facial age estimation system designed for use within the Aadhaar biometric enrollment pipeline, to assist quality-check (QC) operators to flag potential age-related fraud. This is critical for a system like Aadhaar (the world's largest national identity programme), that holds about 1.5 billion unique identities, with 22.4 million new enrollments and 283 million updates in the last year. Building upon the SwinFace architecture with landmark-based similarity (warp affine)
alignment, we train on a large in-house
dataset of 1.45 million face images and evaluate on an independent, age-stratified test set of 283K images, both drawn from an ethnically diverse population of 716K unique subjects. We investigate three Aadhaar-specific operational thresholds (5, 18, and 60 years) and propose a deployment triage framework that flags suspected cases for manual review. Following NIST FATE, we report false acceptance/rejection rates (FAR/FRR) at each threshold rather than aggregate accuracy: at 1% FAR the model achieves an FRR of 3% (<5yrs), 0.4% (>18yrs) and 11.0% (>60yrs). SwinAge achieves a mean absolute error (MAE) of 2.94 years on the same test set, outperforming three zero-shot
vision language models on all
benchmarks, and improving the state-of-the-art on 5 out of 7 public
benchmark datasets. We further report per-gender errors and distill lessons for national identity programs.
Read the full abstract ▾
All authors Sharib Athar, Arka Koner, Chetan Naik, Barada P. Sabut, Tanusree Deb Barma, Anoop M. Namboodiri, Anil K. Jain
Computer Vision
Unsupervised Transfer Clustering for Mitigating Cold Start in Active Prompt Learning
André Camargo Portella, Samuel Felipe dos Santos and Jurandy Almeida
Vision-
Language Models (VLMs) are able to achieve impressive zero-shot classification performance by aligning visual and textual representations, but each new task still demands handcrafted
prompts. Active
Prompt Learning (APL) combines Active Learning (AL) and
Prompt Learning (PL) into a single framework, allowing for the usage of the VLM prior knowledge for iteratively querying the most informative images to be labeled. However, the cold-start problem is still relevant for APL methods, where the performance of the initial query can be worse than random sampling. While recent state-of-the-art APL methods mitigate with balanced sampling and
multimodal features, they rely on rigid, distance-based clustering to group these features. This simplistic approach can struggle to capture the complex, high-dimensional semantic distributions inherent to VLMs, leading to suboptimal query representativeness. Unsupervised transfer can be applied to these features as a possible alternative, since it is capable of inferring the underlying human labeling of a task without any form of supervision. This way, samples can be grouped in semantically coherent clusters. This paper proposes Unsupervised Transfer Clustering with Selective Querying (UTC+SQ), a framework that enhances a recent APL approach by leveraging state-of-the-art unsupervised transfer model. These models generate high-fidelity pseudo-labels that establish semantically meaningful clusters, allowing for the selection of more relevant samples. Experimental evaluations demonstrate that shifting from distance-based to projection-based clustering improves the representativeness of the queried subset, achieving accuracy gains in 6 of the 8
datasets tested.
Read the full abstract ▾
Computer Vision
Time-Aware Assistive Navigation
Masaki Kuribayashi, Zhongkai Shangguan and Eshed Ohn-Bar
Can interactive
vision-and-
language agents learn not just what to say but also \textbf{when} to say it? Current
language models rarely plan over whether and when to realize a real-time response to a user. However, providing accurate and timely support for human decision-making, such as when guiding visually impaired individuals through urban environments, requires careful real-time responsiveness–poorly timed responses can distract users or add unnecessary cognitive load. As a machine intelligence challenge for
Multimodal Large
Language Model (MLLM)-based
agents, we introduce a large-scale
multimodal benchmark for an
egocentric, assistive
navigation task in complex outdoor environments. Using this
benchmark, we uncover a fundamental limitation of off-the-shelf MLLMs in delivering safe and time-sensitive
navigation instructions, even with model fine-tuning on substantial amounts of data. We then demonstrate that a simple yet effective modification of the model, including direct supervision to predict the underlying reason for each instruction, yields significant performance gains across open-loop, closed-loop, and sim-to-real generalization settings. However, our analysis highlights persistent challenges in temporal
reasoning, safety-critical object awareness, and relational and distance understanding. To advance the development of scalable assistive
agents, we will release our simulation,
benchmark, and code (available at the project website: https://timeli-icra.github.io/).
Read the full abstract ▾
Computer Vision
SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution
Xingjian Ran, Xiaoye Mo, Sihao Liu et al. (+3 more)
Diverse and simulation-ready indoor scenes are essential for interactive entertainment and
embodied AI, yet their scalable generation remains challenging. Recent agentic text-to-3D scene pipelines that rely on
vision-
language models (VLMs) can generate scenes of high fidelity but require costly iterative object placement and refinement. Another mainstream paradigm, parametric image-to-3D scene models, produces scenes efficiently from strong priors learned from 2D images but often leads to imprecise and physically invalid scenes. More importantly, both paradigms struggle to output diverse scenes for a single input, making it hard for them to reflect the dynamically changing nature of real scenes. In this paper we propose SceneMosaic, a framework that combines the merits of both paradigms. It obtains the initial candidate from the learned image-based prior, and subsequently evolves the result through VLM
agents, ensuring both efficiency and physical validity. Within the
evolution process, SceneMosaic exploits the locality of natural scenes and decomposes a scene into independent local units, allowing separate
evolution within each unit before composing the global scene via Cartesian product. On SceneEval-100, SceneMosaic matches the strongest agentic baseline in semantic layout quality with a 24x speedup, substantially reduces physical violations, and receives the highest human ratings. Our code is publicly available at https://github.com/rxjfighting/SceneMosaic.
Read the full abstract ▾
All authors Xingjian Ran, Xiaoye Mo, Sihao Liu, Jianyu Zhang, Li Luo, Bo Dai
Computer Vision · Cross-listed from cs.RO
GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation
Renhang Liu, Wenzhi Zhao, Zhuo Yang et al. (+42 more)
World-action models (WAM) predict future states to guide
robot actions, enabling learning from both action-free
video and action-labeled interaction. Most inherit pretrained
video generators, leaving WAM pretraining and scaling underexplored. We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all initialized from scratch on manipulation data. It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). CoAE retains action- and instruction-relevant information under aggressive compression, while SVP produces a complete future state in one differentiable pass, so visual planning and inverse dynamics can be pretrained separately on complementary data. The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action. We evaluate pretrained checkpoints directly, without per-task fine-tuning, on 100 tasks across 20 manipulation skill groups with held-out scenes, backgrounds, lighting, and object instances. Scaling co-training data from 300 to 30,000 hours raises success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D; despite comprising less than 2% of the co-training data, G2-90D improves by 17.7 points, suggesting cross-embodiment transfer. Gains span 19/20 and 18/20 skill groups, and skill-specific coverage strongly correlates with zero-shot out-of-distribution (OOD) success (Pearson r=0.80; Spearman rho=0.85). Under the same protocol, the model grounds object, color, shape, and position references in at least 90% of trials and follows explicit instructions even when they conflict with an already-committed behavior or a conventional scene association.
Read the full abstract ▾
All authors Renhang Liu, Wenzhi Zhao, Zhuo Yang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Guanghui Ren, Youlun Peng, Rongjun Jin, Nan Wang, Sukai Wang, Xindong He, Jinyuan Feng, Ziyu Xiong, Linqing Zhong, Yifei Wei, Feng Han, Long Zhang, Da Huang, Nanshu Zhao, Chenghao Yin, Mo Wu, Zhaodong Yan, Kongtao Hu, Yuxiang Yan, Aogelijiang Niyazi, Yu Fang, Jia Zeng, Lizhu Meng, Daizhen Lv, Haoyu Cao, Zhiwen Hou, Lianjin Ye, Yuehan Niu, Zhikai Cai, Xuan Hu, Hui Min, Xiongfeng Cai, Yue Liao, Jing Wu, Soujanya Poria, Ye Li, Sanping Zhou, Maoqing Yao, AgiBot Research Team
Computer Vision
Márcus Lobo, Vitor Matias, Afonso Paiva et al. (+3 more)
Vision-
Language Models (VLMs) are reshaping computer
vision by aligning visual and textual
embeddings, allowing models to recognize visual concepts and reason about them using natural
language. Traditional 3D deep-learning models, however, are typically trained for specific tasks, such as classification,
segmentation, or detection, and do not naturally support cross-modal
retrieval from their
embedding spaces using text or images as queries. To address this issue, Contrastive
Language-Image Pretraining (CLIP)-based methods align 3D
embeddings with pretrained image and text representations, giving rise to 3D
Vision-
Language Models (3D VLMs) that support zero-shot classification, cross-modal
retrieval, and open-vocabulary recognition of 3D shapes. This tutorial provides an overview of 3D VLMs, ranging from basic definitions of 3D representations and their encoding into
embeddings to cross-modal contrastive
alignment, modern
multimodal frameworks, and 3D
Vision-Large
Language Models (3D VLLMs). We present the main definitions of contrastive learning for
multimodal embedding alignment and highlight recent advances in
language-guided 3D Gaussian splatting, 3D shape generation, and
embodied AI for robotics.
Read the full abstract ▾
All authors Márcus Lobo, Vitor Matias, Afonso Paiva, Jeová Farias, Tiago Novello, Moacir Ponti
Computer Vision
ViT3Flow: A Test-Time Training Transformer MeanFlow for Postoperative Radiograph Synthesis in Scoliosis
Rui Tang, Sicheng Yang, Moxin Zhao et al. (+6 more)
Predicting postoperative spinal morphology from preoperative radiographs could provide valuable support for scoliosis
surgical planning, but remains challenging because
surgical correction induces large spatial changes while anatomical structures must be faithfully retained. We formulate this problem as postoperative scoliosis radiograph synthesis and construct ScoliSurg, the first paired
dataset for this task, comprising 632 preoperative–postoperative whole-spine radiograph pairs with structured morphology information. We further propose ViT$^{3}$Flow, a single-NFE conditional MeanFlow framework for efficient postoperative radiograph synthesis. ViT$^{3}$Flow models
surgical correction as finite-interval generative transport and replaces conventional self-
attention with test-time-training
token mixers that perform sample-specific inner adaptation to the anatomy and deformity pattern of each case. In addition, a Spinal Morphology Extraction
Agent extracts distributions of dominant-curve region and direction from the preoperative radiograph. These distributions guide Diagnosis-Routed Interval Cross-
Attention (DRICA), which performs interval-dependent vertical, horizontal, joint, and global
retrieval from a separate preoperative
token stream. This design enables the evolving postoperative representation to incorporate spatially corresponding anatomical evidence throughout the transport process. Extensive experiments on ScoliSurg demonstrate that ViT$^{3}$Flow achieves the best performance among the compared methods in perceptual image quality, anatomical fidelity, and clinically relevant
geometric accuracy, while requiring only a single network evaluation. These results highlight the potential of ViT$^{3}$Flow for efficient and anatomically faithful postoperative radiograph synthesis in scoliosis
surgical planning.
Read the full abstract ▾
All authors Rui Tang, Sicheng Yang, Moxin Zhao, Hongqiu Wang, Guankun Wang, Lei Zhu, Hongliang Ren, Menglin Cong, Nan Meng
Computer Vision · Cross-listed from cs.LG
Capsule Lens: Locating and Tracking Concept Geometry in Model Representations
Yiming Tang, Harshvardhan Saini, Samyak Jha et al. (+3 more)
Understanding how concepts are encoded in the internal representations of machine learning models is a central problem in mechanistic interpretability, essential both for the science of deep learning and for the trustworthy deployment of increasingly capable models. Existing approaches to interpret model representations mainly map representations onto more interpretable spaces and do not directly characterize how concepts occupy representation space; various hypotheses have been proposed, but often lack of rigorous validation and largely focus on static representations. In this work, we introduce Capsule Lens, a framework that matches the region a concept occupies with a simple, trackable
geometric form, a capsule, defined by several interpretable parameters, fitted in closed form to each concept's geometry and validated on held-out samples. We apply Capsule Lens in two major settings: static and dynamic representations. On static representations, we demonstrate how to locate concept geometry across various models, and how the span and norm curves uncover important
geometric characteristics. On dynamic representations, we present three case studies tracking representation drifts induced by distinct training settings, CLIP pretraining, RL post-training on visual question answering, and RL post-training on mathematical
reasoning. These analyses reveal qualitatively different
geometric dynamics, ranging from broad network-wide restructuring in CLIP pretraining to localized and concept-specific changes in RL post-training. Our results include findings aligned with existing literature as well as novel observations. We believe Capsule Lens stands as a promising tool for locating, analyzing, and tracking concept geometry in both static and dynamic representations.
Read the full abstract ▾
All authors Yiming Tang, Harshvardhan Saini, Samyak Jha, Huaming Chen, Xufeng Duan, Dianbo Liu
Computer Vision
WorldSculpt: Generating Compositional Worlds from Grounded Videos
Muyao Niu, Jixuan He, Ruihan Yu et al. (+9 more)
We study the problem of generating a compositional 3D representation of a cluttered scene containing hundreds of objects. The goal is to represent the scene as a collection of individual object meshes placed in a shared world frame, as required by downstream applications such as gaming, AR/VR, simulation, and robotics. This task is challenging in densely cluttered scenes, where objects heavily occlude one another and each view reveals only a fraction of their geometry. Geometry-based approaches typically reconstruct the scene as a single representation and leave incomplete geometry in occluded regions, while existing compositional methods with generative priors are largely limited to relatively simple scenes. We show that complex scenes with hundreds of objects can instead be generated compositionally by adapting a strong single-object 3D generative prior to multi-view observations. We instantiate this paradigm with Pixal3D, extending it with a multi-view conditioning pathway that grounds object generation in multiple posed observations. Although the model is finetuned entirely on single objects in canonical space, it generalizes to large scenes with severe occlusion without any scene-level training, demonstrating the feasibility and scalability of this paradigm. We further introduce UE-MeshyScene, a photorealistic
benchmark of densely cluttered scenes with hundreds of objects, per-object annotations, and ground-truth meshes. Across single-object, controlled multi-object, and UE-MeshyScene evaluations, our method consistently outperforms prior approaches, with larger gains as scene complexity and occlusion increase. Finally, we demonstrate broader applicability by converting generated 3DGS worlds, such as Marble and HY-World 2.0, into compositional mesh scenes.
Read the full abstract ▾
All authors Muyao Niu, Jixuan He, Ruihan Yu, Lian Fu, Yonghao Yu, Zheng-Hui Huang, Yifan Zhan, Fengbo Lan, Yongtao Ge, Yinqiang Zheng, Kaipeng Zhang, Zhixiang Wang
Computer Vision
UniMate: One Unified Model to Animate Diverse Skeletons
Linzhan Mou, Jiahui Lei, Zhiyang Dou et al. (+4 more)
Recent advances in automatic rigging now deliver animation-ready 3D assets at scale, yet generating the motion to drive them remains a bottleneck. Existing learned animators are
topology-constrained: they rely on category-specific templates or require per-skeleton fine-tuning and reference motions at inference. We present UniMate, a unified foundation model that synthesizes articulated motion for arbitrary skeletons from a rigged 3D asset and a text
prompt, with no test-time optimization or per-skeleton retraining. UniMate introduces a
topology-aware
diffusion transformer, which integrates skeletal
topology into
attention via three mechanisms: (1) a graph-aware
attention bias from pairwise joint relations and geodesic distances; (2) a spectral rotary position
embedding generalizing RoPE to arbitrary kinematic trees via the graph
Laplacian; and (3) a global topological conditioner
attention-pooled from the rest-pose skeleton. We also curate UniML3D, 13,006 motion sequences spanning bipedal, quadrupedal, avian, marine, insectoid, serpentine, and articulated rigid objects with unified canonicalization and text pairing. Trained on this
dataset, UniMate outperforms state-of-the-art baselines in quality, generalization, and efficiency, and supports zero-shot cross-
topology transfer, in-betweening, expansion, and text-guided editing. Our project page is available at https://linzhanmou.com/unimate/.
Read the full abstract ▾
All authors Linzhan Mou, Jiahui Lei, Zhiyang Dou, Chenyue Cai, Chaoyue Song, Adam Finkelstein, Szymon Rusinkiewicz
Computer Vision
A Generalizable Feature Extractor for Alzheimer's-Related Brain MRI Tasks
Reza Rajabli and D. Louis Collins
When there is not enough labeled data to properly train deep learning models, transfer learning can help. We still do not fully understand how effective it is in neuroimaging, especially for Alzheimer's disease research. It is also not clear if these transferred models can work on new
datasets without being retrained for each specific task. We evaluate whether a compact, supervised pretrained model can serve as a reusable foundation model for downstream neuroimaging tasks. We freeze the 7.18 million weights of a 3D CNN previously trained for
brain-age prediction, and adapt it to each task using Low-Rank Adaptation (LoRA), requiring only ~1% additional trainable parameters. We evaluate generalizability in six experiments. Adapting the model to classify cognitively normal versus Dementia on ADNI gave an AUC of 0.964 on held-out folds (Experiment #1). Applying that adapted model unchanged to OASIS-3, with no retraining, gave an AUC of 0.871 (Experiment #2). Reusing its output logit together with age and a cognitive score distinguished stable from progressing MCI with an AUC of 0.828 (Experiment #3). Adapting the same backbone to predict amyloid positivity from structural MRI gave an AUC of 0.804 (Experiment #4). Finally, the same approach estimated ICV-normalized hippocampal and white matter hypointensity volumes directly from the T1w image, with R^2 of 0.80 and 0.91 respectively, tasks normally addressed with much larger U-Net networks (Experiments #5 and #6). A compact model supervised on
brain age can therefore serve as a reusable backbone, adapting to each task with ~1% additional parameters and transferring to an unseen cohort without any training. Our findings suggest that a carefully trained
brain age model can serve as an effective foundation model for Alzheimer's related tasks, even under strict data constraints.
Read the full abstract ▾
Computer Vision
From Interpretability Methods to Interpretable Models
Julien Colin, Nuria Oliver and Thomas Serre
More than a decade in, explainable AI (XAI) for computer
vision has assembled a mature toolbox: attribution, feature visualization, concept-based, and
circuit-based methods. Yet almost all of the field's effort has gone into building and comparing these methods, and little into the question they were meant to answer—how interpretable are our models, and are we making progress as they evolve? We argue for shifting the field's focus from methods to models, along two complementary lines. One is already within reach: existing tools let us characterize and compare what different models represent and compute. The other is harder, and largely neglected: whether a model can actually be understood by the humans who rely on it—the independent evaluators on whom trust and certification depend, not the experts confirming what they already expect. It can only be measured, not inferred. We review why the toolbox is mature enough to support both, survey the thin body of work comparing models, draw a parallel to systems neuroscience, and close with a model-centric XAI agenda.
Read the full abstract ▾
Computer Vision
CrossDepth: Geometry-Constrained Attention for Generalizable Multi-View Surround Depth Estimation
Samer Abualhanud and Max Mehltretter
Reliable 3D understanding of the surrounding environment is a core requirement for autonomous driving. Multi-view surround camera rigs provide broad scene coverage, but the spatially adjacent images typically overlap only minimally. Consequently, the depth of most pixels must be inferred from
monocular appearance cues. These cues can appear differently across images and may therefore be interpreted differently by the depth estimation model. We target two main sources of cross-image inconsistency: differences in camera intrinsics and the limited receptive field of each image. We address the former by conditioning the features on per-pixel camera-aware ray
embeddings, enabling the network to account for camera-dependent variations in
monocular cues. We address the latter by extending each pixel's context beyond its own image through cross-image
attention constrained to geometrically plausible regions, derived from the calibrated rig setup. The model is trained in a fully self-supervised manner based on photometric consistency. Evaluations on DDAD and nuScenes show improved overall depth accuracy and cross-image depth consistency over state-of-the-art self-supervised methods under in-domain and cross-domain evaluation. Code is available at https://abualhanud.github.io/CrossDepthPage/.
Read the full abstract ▾
Computer Vision
Think-Verify-Revise: Neuro-Symbolic Visual Reasoning with Vision-Language Models and Dynamic Logic Tensor Networks
Homayoun Afshari, Pietro Basci, Alessandro Russo et al. (+1 more)
Visual
reasoning tasks require a system to jointly perceive visual content and apply formal relational constraints—a combination that neither pure neural nor purely symbolic approaches handle well in isolation. This paper proposes a Neuro-Symbolic (NeSy) framework that closes this gap by tightly coupling a
Vision-
Language Model (VLM) for automatic First-Order Logic (FOL) rule induction with a Dynamic Logic Tensor Network (D-LTN) for differentiable rule verification, in a closed iterative feedback loop. The VLM receives a small set of labelled visual examples and proposes candidate FOL rules conforming to a strict grammar (Think); the D-LTN is automatically assembled from these rules at runtime and evaluates them grounding on CNN-produced visual
embeddings (Verify); and verification failures are fed back to guide the VLM's next hypothesis (Revise). Evaluated on the ViSudo-PC
benchmark across four visual domains (MNIST, EMNIST, KMNIST, FMNIST), the system induces valid Sudoku constraint rules using only three training examples as visual context. The proposed method achieves AUC scores matching or outperforming previous methods (NeuPSL, LTN), showing the potential for automatic rule discovery through VLM. Code is available at https://github.com/homayoun-afshari/nesy.
Read the full abstract ▾
All authors Homayoun Afshari, Pietro Basci, Alessandro Russo, Lia Morra
Computer Vision
Reflection-aware Generative Novel View Synthesis
GeonU Kim, Shin Dong-Yeon and Tae-Hyun Oh
We propose Ref-GeNVS, a training-free, reflection-aware method for generative novel view synthesis (NVS) in mirror scenes. Existing multi-view
diffusion models often fail to recognize the mirror in the scene and cannot exploit reflected content for scene generation. To fix this issue without additional training, our key idea is to treat a mirror image as two complementary views. From input images, we estimate the mirror plane and reflect camera poses to form virtual views. Based on this virtual view setup, we propose a two-stage generation method consisting of Mirror-gated
attention and Reflection injection, which enables reflection-consistent NVS by explicitly leveraging reflection relationships in a multi-view
diffusion model. Ref-GeNVS inherits the strong generalizability of the multi-view
diffusion backbone, while it does not require finetuning. On synthetic and real scenes including mirrors, Ref-GeNVS outperforms recent generative NVS methods by generating reflection-consistent and contextually coherent novel views, revealing scene structure visible only through mirrors. Project page: https://kim-geonu.github.io/Ref-GeNVS/
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies
Vivek Chavan, Pengtao Xie, Yahuan Shi et al. (+3 more)
Visuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced. We study this behavior as a problem of conditional visual grounding: the visual target required for successful control changes with the manipulation phase and, in more complex tasks, with the observed task state. Using Action Chunking with
Transformers (ACT), we systematically introduce distractor objects and receptacles with controlled color and shape similarity and localize failures to picking and placement. We find that distractor sensitivity is specific to both the type of visual similarity and the manipulation stage. Guided by this diagnosis, we evaluate distractor augmentation, phase-dependent
attention regularization, and appearance-based visual prompting as complementary interventions for improving target selection while preserving spatial information required for control. These interventions substantially improve robustness in simulation and on a physical UR3e. We further examine the same failure pattern in a pretrained
vision-
language-action policy on a state-conditioned instrument-handling task, where the observed state of a medical instrument determines the correct destination. Together, the results show that visual distractors can cause incorrect object or destination selection even when the underlying manipulation skill remains intact, and that explicitly improving target selection can substantially recover performance across distinct visuomotor policy-learning regimes.
Read the full abstract ▾
All authors Vivek Chavan, Pengtao Xie, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger
Computer Vision · Cross-listed from cs.RO
Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation
Vivek Chavan, Yahuan Shi, Oliver Heimann et al. (+2 more)
Vision-
language-action (VLA) models can execute short manipulation skills, but remain brittle in long-horizon procedures requiring persistent task state, dependency-aware
reasoning, conditional decisions, and reliable grounding. We investigate a neuro-symbolic framework that combines learned VLA control with explicit task graphs and
multimodal procedural
memory. Task graphs encode action dependencies, valid transitions, and branch conditions, while
memory maintains the active step, completed actions, textual context, and task-relevant visual evidence. Together, these structures guide object selection, destination grounding, subgoal dispatch, and verification of expected state transitions. Human demonstrations provide additional spatial and temporal guidance through gaze or saliency cues. To isolate their effect on policy learning, our initial study bypasses cross-view gaze transfer and directly annotates pseudo-gaze in
robot-view teleoperation
videos. The resulting guidance is used during VLA fine-tuning and inference. We study two long-horizon manipulation domains, workspace clearing and
surgical-instrument handling, which require ordered execution, visually grounded decisions, and conditional branching. We evaluate correct-object and destination selection, subtask completion, task progress, step-order consistency, complete-task success, and procedural or execution mistakes. This work positions structured symbolic
reasoning and demonstration-derived visual guidance as complementary mechanisms for reliable long-horizon VLA manipulation.
Read the full abstract ▾
All authors Vivek Chavan, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger
Computer Vision
MEOX: Compact Multimodal Mixture-of-Experts for Earth Observation
Mohanad Albughdadi
Recent advances in Earth Observation representation learning accommodate heterogeneous sensors and missing observations, often through larger architectures. We present MEOX (
Multimodal Earth Observation with eXperts), a
multimodal masked autoencoder with a 2.939 million-parameter encoder and 3.115 million parameters in total. Sensor-specific adapters, explicit validity signals, and a shared sparse-expert block preserve modality-dependent processing before a learned patch-wise fusion. Four metadata
tokens then accompany a single spatial sequence through fourteen further encoder blocks. Shared expert projections with private low-rank residuals constrain parameter growth, while rotary
attention supports downstream spatial grids different from pretraining. The model is pretrained on 1.228 million MMEarth64 samples using modality-balanced masked reconstruction and structured sensor dropout. Frozen transfer is evaluated on six GEO-Bench tasks at both 64 and 224 pixels. The model reaches 64.42% mean intersection-over-union on cashew
segmentation at 64 pixels and 90.56% average accuracy on EuroSAT at 224 pixels, exceeding the corresponding reported CSMoE results. BigEarthNet finetuning reaches 72.95% micro-average precision. Routing diagnostics distinguish expert participation, spatial dependence, modality association, and functional contribution. A held-out WorldCover probe measures a 0.64-percentage-point benefit from metadata, while
retrieval separates same-sensor semantics from cross-sensor
alignment. These results demonstrate sensor-flexible representation learning and strong task transfer using a compact parameter budget.
Read the full abstract ▾
Computer Vision
Lightweight Vision Transformer Compression for On-Device Plant Disease Detection in Resource-Constrained Agricultural Field Conditions
Mahadev Sunil Kumar, Bhavika Gondi, Desaisetty Venkata Satya Sai Swapnith et al. (+6 more)
Chilli (Capsicum annuum) is one of India's most economically significant crops, yet its productivity is persistently threatened by diseases that are difficult to identify without expert intervention. While
Vision Transformers (ViTs) have achieved high classification accuracy, their large computational footprint makes deployment on resource constrained devices challenging. Existing compression approaches typically address
pruning, quantization, and knowledge
distillation in isolation, leaving the potential benefits and interactions of their combined application insufficiently explored. We propose a unified
Vision Transformer compression framework that combines Hessian-Balanced Adaptive Block
Pruning (H-BAC), guided by second-order sensitivity estimation, with quantization and
attention-based knowledge
distillation. To systematically identify the most effective configuration within each compression family, each technique is first evaluated independently through controlled ablation studies, after which the best-performing components are integrated into a sequential deployment pipeline tailored to real-world agricultural constraints. On a chilli 3-class village-split
dataset with a genuine cross-village, cross-device out-of-distribution test split, the resulting compressed models match or exceed the 95.13% FP32 baseline's accuracy, alongside 74-98% model size reduction, and the fully integrated compression pipeline achieves a 54.5x size reduction (327.42 MB to 6.01 MB) at 95.13 +/- 2.32% accuracy across four tested configurations. A direct comparison further reveals that, on this
dataset, a directly-trained student of the same final size, without
pruning or
distillation, reaches comparable accuracy of 94.87%, at the same 6.01 MB INT8 size, indicating where H-BAC and knowledge
distillation are, and are not yet shown to be, worth their computational cost.
Read the full abstract ▾
All authors Mahadev Sunil Kumar, Bhavika Gondi, Desaisetty Venkata Satya Sai Swapnith, Gangireddy Rahul Jogi, Sudheesh Manalil, Arnab Raha, Amitava Mukherjee, Parthasarathy Seethapathy, G. Gopakumar
Computer Vision · Cross-listed from cs.RO
RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?
Zhenxuan Fan, Bo Zhang, Yutong Lin et al. (+9 more)
Vision-
Language-Action (VLA) models have shown promising progress in
language-conditioned robotic manipulation. However, existing
datasets and
benchmarks mainly evaluate task completion under predefined settings, offering limited insight into model
reasoning under increasing spatial and procedural complexity. We introduce RoboSPA (Robot Spatial-Procedural Assessment), a large-scale robotic manipulation
dataset and
benchmark for diagnosing
embodied reasoning in VLA models. RoboSPA focuses on two core dimensions, Fine-Grained Spatial
Reasoning and Long-Horizon Procedural Planning, covering 10 task categories and 56 base tasks. Each task is instantiated across five difficulty levels, yielding 280 variants with increasing spatial ambiguity and procedural complexity. We collect 527K trajectories across multiple embodiments and diverse scenes. Beyond binary success rate, RoboSPA introduces diagnostic metrics for more detailed evaluation. Experiments on representative VLA models show that current systems still struggle with complex spatial relations, precise low-level execution, and
memory-intensive planning. These results establish RoboSPA as a challenging diagnostic
benchmark for developing more capable, reliable, and generalizable
embodied agents. Our data and code are available at https://github.com/fanzhenxuan/RoboSPA.
Read the full abstract ▾
All authors Zhenxuan Fan, Bo Zhang, Yutong Lin, Yuqian Yuan, Juekai Lin, Liang Liang, Zhuoyi Huang, Wenqiao Zhang, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang
Computer Vision
Scalable Detection of Fossil Palynomorphs in Multifocal Digital Microscopy Images
Abbas Shaikh, Praise Mayor, Patrick Ainlay-Vazquez et al. (+7 more)
Palynomorphs (microscopic, organic-walled fossils such as pollen, spores, and dinoflagellates) are important high-resolution records of past climates and are critical to the study of ancient ecosystems. Existing methods rely on manual analysis of high-resolution, multifocal digital microscopy images, which is slow and time-consuming and requires researchers to compromise on the scale of their investigations. To the best of our knowledge, our work proposes the first ever scalable end-to-end pipeline for automated palynomorph detection in whole slide images that addresses this bottleneck through: (1) efficient methods for decomposing and compressing digitized multifocal microscope slide images into tractable 2-dimensional tiles for analysis; (2) benchmarking modern object detection models, including RF-DETR, for the detection of palynomorphs, achieving an AP@50 of 0.879; (3) an efficient
algorithm for the synthesis of detection outputs across large-scale, high-resolution images; and (4) an I/O optimization resulting in faster inference time. Our methods drastically reduce the time required for palynomorph detection in a single slide from often days of manual inspection to under one hour of automated analysis, enabling palynological research at a substantially greater scale.
Read the full abstract ▾
All authors Abbas Shaikh, Praise Mayor, Patrick Ainlay-Vazquez, Aditya Viswanathan, Teon Golden, Eric Zhang, Ingrid C. Romero, Alexander E. White, Scott Wing, Arko Barman