Computer Vision
RecGen3D: Reconstruction-Guided 3D Generation in a Shared Canonical Space
Zhisheng Huang, Jiahao Chen, Cheng Lin et al. (+10 more)
Sparse-view 3D modeling represents a fundamental tension between reconstruction fidelity and generative plausibility. While feed-forward reconstruction excels in efficiency and input
alignment, it often lacks the global priors needed for structural completeness. Conversely,
diffusion-based generation provides rich
geometric details but struggles with multi-view consistency. We present RecGen3D, a framework that combines these two paradigms into a cooperative system. To overcome inherent conflicts in coordinate spaces, 3D representations, and training objectives, we align both models within a shared canonical space. We employ decoupled cooperative learning, which maintains stable training while enabling seamless collaboration during inference. Specifically, the reconstruction module is adapted to provide canonical
geometric anchors, while the
diffusion generator leverages latent-augmented conditioning to refine and complete the
geometric structure. Experimental results demonstrate that RecGen3D achieves superior fidelity and robustness, outperforming existing methods in creating complete and consistent 3D models from sparse observations.
Read the full abstract ▾
All authors Zhisheng Huang, Jiahao Chen, Cheng Lin, Chenyu Hu, Hanzhuo Huang, Zhengming Yu, Mengfei Li, Yuheng Liu, Zekai Gu, Zibo Zhao, Yuan Liu, Xin Li, Wenping Wang
Computer Vision
Prime Once, then Reprogram Locally: An Efficient Alternative to Black-Box Service Model Adaptation
Yunbei Zhang, Chengyi Cai, Feng Liu et al. (+1 more)
Adapting closed-box service models (i.e., APIs) for target tasks typically relies on reprogramming via Zeroth-Order Optimization (ZOO). However, this standard strategy is known for extensive, costly API calls and often suffers from slow, unstable optimization. Furthermore, we observe that this paradigm faces new challenges with modern APIs (e.g., GPT-4o). These models can be less sensitive to the input perturbations ZOO relies on, thereby hindering performance gains. To address these limitations, we propose an Alternative efficient Reprogramming approach for Service models (AReS). Instead of direct, continuous closed-box optimization, AReS initiates a single-pass interaction with the service API to
prime an amenable local pre-trained encoder. This priming stage trains only a lightweight layer on top of the local encoder, making it highly receptive to the subsequent glass-box (white-box) reprogramming stage performed directly on the local model. Consequently, all subsequent adaptation and inference rely solely on this local proxy, eliminating all further API costs. Experiments demonstrate AReS's effectiveness where prior ZOO-based methods struggle: on GPT-4o, AReS achieves a +27.8% gain over the zero-shot baseline, a task where ZOO-based methods provide little to no improvement. Broadly, across ten diverse
datasets, AReS outperforms state-of-the-art methods (+2.5% for VLMs, +15.6% for standard VMs) while reducing API calls by over 99.99%. AReS thus provides a robust and practical solution for adapting modern closed-box models.
Read the full abstract ▾
All authors Yunbei Zhang, Chengyi Cai, Feng Liu, Jihun Hamm
Computer Vision · Cross-listed from cs.RO
Efficient Equivariant Transformer for Self-Driving Agent Modeling
Scott Xu, Dian Chen, Kelvin Wong et al. (+3 more)
Accurately modeling
agent behaviors is an important task in self-driving. It is also a task with many
symmetries, such as equivariance to the order of
agents and objects in the scene or equivariance to arbitrary roto-translations of the entire scene as a whole; i.e., SE(2)-equivariance. The
transformer architecture is a ubiquitous tool for modeling these
symmetries. While standard self-
attention is inherently permutation equivariant, explicit pairwise relative positional encodings have been the standard for introducing SE(2)-equivariance. However, this approach introduces an additional cost that is quadratic in the number of
agents, limiting its scalability to larger scenes and batch sizes. In this work, we propose DriveGATr, a novel
transformer-based architecture for
agent modeling that achieves SE(2)-equivariance without the computational cost of existing methods. Inspired by recent advances in
geometric deep learning, DriveGATr encodes scene elements as multivectors in the 2D projective
geometric algebra $\mathbb{R}^*_{2,0,1}$ and processes them with a stack of equivariant
transformer blocks. Crucially, DriveGATr models
geometric relationships using standard
attention between multivectors, eliminating the need for costly explicit pairwise relative positional encodings. Experiments on the Waymo Open Motion
Dataset demonstrate that DriveGATr is comparable to the state-of-the-art in traffic simulation and establishes a superior Pareto front for performance vs computational cost.
Read the full abstract ▾
All authors Scott Xu, Dian Chen, Kelvin Wong, Chris Zhang, Kion Fallah, Raquel Urtasun
Computer Vision
Reinforcing Consistency in Video MLLMs with Structured Rewards
Yihao Quan, Zeru Shi, Jinman Zhao et al. (+1 more)
Multimodal large
language models (MLLMs) have achieved remarkable progress in
video understanding. However, seemingly plausible outputs often suffer from poor visual and temporal grounding: a model may fabricate object existence, assign incorrect attributes, or collapse repeated events while still producing a globally reasonable caption or answer. We study this failure mode through a compositional consistency audit that decomposes a caption into supporting factual and temporal claims, investigating whether a correct high-level prediction is actually backed by valid lower-level evidence. Our top-down audit reveals that even correct root relational claims often lack reliable attribute and existence support. This indicates that standard sentence-level supervision is a weak proxy for faithful
video understanding. Furthermore, when turning to reinforcement learning (RL) for better
alignment, standard sentence-level rewards often prove too coarse to accurately localize specific grounding failures. To address this, we replace generic sentence-level rewards with a structured reward built from factual and temporal units. Our training objective integrates three complementary components: (1) an instance-aware scene-graph reward for factual objects, attributes, and relations; (2) a temporal reward for event ordering and repetition; and (3) a
video-grounded VQA reward for hierarchical self-verification. Across temporal, general
video understanding, and
hallucination-oriented
benchmarks, this objective yields consistent gains on open-source backbones. These results suggest that structured reward shaping is a practical route to more faithful
video understanding.
Read the full abstract ▾
All authors Yihao Quan, Zeru Shi, Jinman Zhao, Ruixiang Tang
Computer Vision
Nonlinear Methods for Analyzing Pose in Behavioral Research
Carter Sale, Margaret C. Macpherson, Gaurav Patil et al. (+4 more)
Advances in markerless pose estimation have made it possible to capture detailed human movement in naturalistic settings using standard
video, enabling new forms of behavioral analysis at scale. However, the high dimensionality, noise, and temporal complexity of pose data raise significant challenges for extracting meaningful patterns of coordination and behavioral change. This paper presents a general-purpose analysis pipeline for human pose data, designed to support both linear and nonlinear characterizations of movement across diverse experimental contexts. The pipeline combines principled preprocessing, dimensionality reduction, and recurrence-based time series analysis to quantify the temporal structure of movement dynamics. To illustrate the pipeline's flexibility, we present three case studies spanning facial and full-body movement, 2D and 3D data, and individual versus multi-
agent behavior. Together, these examples demonstrate how the same analytic workflow can be adapted to extract theoretically meaningful insights from complex pose time series.
Read the full abstract ▾
All authors Carter Sale, Margaret C. Macpherson, Gaurav Patil, Kelly Miles, Rachel W. Kallen, Sebastian Wallot, Michael J. Richardson
Computer Vision
Better Rigs, Not Bigger Networks: A Body Model Ablation for Gaussian Avatars
Derek Austin
Recent 3D Gaussian splatting methods built atop SMPL achieve remarkable visual fidelity while continually increasing the complexity of the overall training architecture. We demonstrate that much of this complexity is unnecessary: by replacing SMPL with the Momentum Human Rig (MHR), estimated via SAM-3D-Body, a minimal pipeline with no learned deformations or pose-dependent corrections achieves the highest reported PSNR and competitive or superior LPIPS and SSIM on PeopleSnapshot and ZJU-MoCap. To disentangle pose estimation quality from body model representational capacity, we perform two controlled ablations: translating SAM-3D-Body meshes to SMPL-X, and translating the original
dataset's SMPL poses into MHR both retrained under identical conditions. These ablations confirm that body model expressiveness has been a primary bottleneck in avatar reconstruction, with both mesh representational capacity and pose estimation quality contributing meaningfully to the full pipeline's gains.
Read the full abstract ▾
Computer Vision
EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generation
Abhishek Saroha, Huajian Zeng, Xingxing Zuo et al. (+2 more)
Understanding and predicting object motion from
egocentric video is fundamental to
embodied perception and interaction. However, generating physically consistent 6DoF trajectories remains challenging due to occlusions, fast motion, and the lack of explicit physical
reasoning in existing generative models. We present EgoFlow, a flow-matching framework that synthesizes realistic and physically plausible trajectories conditioned on
multimodal egocentric observations. EgoFlow employs a hybrid Mamba-
Transformer-Perceiver architecture to jointly model temporal dynamics, scene geometry, and semantic intent, while a gradient-guided inference process enforces differentiable physical constraints such as collision avoidance and motion smoothness. This combination yields coherent and controllable motion generation without post-hoc filtering or additional supervision. Experiments on real-world
datasets HD-EPIC, EgoExo4D, and HOT3D show that EgoFlow outperforms
diffusion-based and
transformer baselines in accuracy, generalization, and physical realism, reducing collision rates by up to 79%, and strong generalization to unseen scenes. Our results highlight the promise of flow-based generative modeling for scalable and physically grounded
egocentric motion understanding.
Read the full abstract ▾
All authors Abhishek Saroha, Huajian Zeng, Xingxing Zuo, Daniel Cremers, Xi Wang
Computer Vision
LESV: Language Embedded Sparse Voxel Fusion for Open-Vocabulary 3D Scene Understanding
Fusang Wang, Nathan Piasco, Moussab Bennehar et al. (+3 more)
Recent advancements in open-vocabulary 3D scene understanding heavily rely on 3D Gaussian Splatting (3DGS) to register
vision-
language features into 3D space. However, we identify two critical limitations in these approaches: the spatial ambiguity arising from unstructured, overlapping Gaussians which necessitates probabilistic feature registration, and the multi-level semantic ambiguity caused by pooling features over object-level masks, which dilutes fine-grained details. To address these challenges, we present a novel framework that leverages Sparse Voxel Rasterization (SVRaster) as a structured, disjoint geometry representation. By regularizing SVRaster with
monocular depth and normal priors, we establish a stable
geometric foundation. This enables a deterministic, confidence-aware feature registration process and suppresses the semantic bleeding artifact common in 3DGS. Furthermore, we resolve multi-level ambiguity by exploiting the emerging dense
alignment properties of the AM-
RADIO foundation model, avoiding the computational overhead of hierarchical training methods. Our approach achieves state-of-the-art performance on Open Vocabulary Point Cloud Understanding, and highly competitive results on 3D and 2D Object
Retrieval benchmarks.
Read the full abstract ▾
All authors Fusang Wang, Nathan Piasco, Moussab Bennehar, Luis Roldão, Dzmitry Tsishkou, Fabien Moutarde
Computer Vision
GRAZE: Grounded Refinement and Motion-Aware Zero-Shot Event Localization
Syed Ahsan Masud Zaidi, Lior Shamir, William Hsu et al. (+2 more)
American football practice generates
video at scale, yet the interaction of interest occupies only a brief window of each long, untrimmed clip. Reliable biomechanical analysis, therefore, depends on spatiotemporal localization that identifies both the interacting entities and the onset of contact. We study First Point of Contact (FPOC), defined as the first frame in which a player physically touches a tackle dummy, in unconstrained practice footage with camera motion, clutter, multiple similarly equipped athletes, and rapid pose changes around impact. We present GRAZE, a training-free pipeline for FPOC localization that requires no labeled tackle-contact examples. GRAZE uses Grounding DINO to discover candidate player-dummy interactions, refines them with motion-aware temporal
reasoning, and uses SAM2 as an explicit pixel-level verifier of contact rather than relying on detection confidence alone. This separation between candidate discovery and contact confirmation makes the approach robust to cluttered scenes and unstable grounding near impact. On 738 tackle-practice
videos, GRAZE produces valid outputs for 97.4% of clips and localizes FPOC within $\pm$ 10 frames on 77.5% of all clips and within $\pm$ 20 frames on 82.7% of all clips. These results show that frame-accurate contact onset localization in real-world practice footage is feasible without task-specific training.
Read the full abstract ▾
All authors Syed Ahsan Masud Zaidi, Lior Shamir, William Hsu, Scott Dietrich, Talha Zaidi
Computer Vision
AffordTissue: Dense Affordance Prediction for Tool-Action Specific Tissue Interaction
Aiza Maksutova, Lalithkumar Seenivasan, Hao Ding et al. (+5 more)
Surgical action automation has progressed rapidly toward achieving surgeon-like dexterous control, driven primarily by advances in learning from demonstration and
vision-
language-action models. While these have demonstrated success in table-top experiments, translating them to clinical deployment remains challenging: current methods offer limited predictability on where instruments will interact on tissue surfaces and lack explicit conditioning inputs to enforce tool-action-specific safe interaction regions. Addressing this gap, we introduce AffordTissue, a
multimodal framework for predicting tool-action specific tissue affordance regions as dense heatmaps during cholecystectomy. Our approach combines a temporal
vision encoder capturing tool motion and tissue dynamics across multiple viewpoints,
language conditioning enabling generalization across diverse instrument-action pairs, and a DiT-style decoder for dense affordance prediction. We establish the first tissue affordance
benchmark by curating and annotating 15,638
video clips across 103 cholecystectomy procedures, covering six unique tool-action pairs involving four instruments (hook, grasper, scissors, clipper) and their associated tasks: dissection, grasping, clipping, and cutting. Experiments demonstrate substantial improvement over
vision-
language model baselines (20.6 px ASSD vs. 60.2 px for Molmo-VLM), showing that our task-specific architecture outperforms large-scale foundation models for dense
surgical affordance prediction. By predicting tool-action specific tissue affordance regions, AffordTissue provides explicit spatial
reasoning for safe
surgical automation, potentially unlocking explicit policy guidance toward appropriate tissue regions and early safe stop when instruments deviate outside predicted safe zones.
Read the full abstract ▾
All authors Aiza Maksutova, Lalithkumar Seenivasan, Hao Ding, Jiru Xu, Chenhao Yu, Chenyan Jing, Yiqing Shen, Mathias Unberath
Computer Vision
IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentation
Nermin Samet, Gilles Puy and Renaud Marlet
This paper presents a new method for the zero-shot open-vocabulary semantic
segmentation (OVSS) of 3D automotive
lidar data. To circumvent the recognized image-text modality gap that is intrinsic to approaches based on
Vision Language Models (VLMs) such as CLIP, our method relies instead on image generation from text, to create prototype images. Given a 3D network distilled from a 2D
Vision Foundation Model (VFM), we then label a point cloud by matching 3D point features with 2D image features of these prototypes. Our method is state-of-the-art for OVSS on nuScenes and SemanticKITTI. Code, pre-trained models and generated images are available at https://github.com/valeoai/IGLOSS.
Read the full abstract ▾
Computer Vision
Perceptual misalignment of texture representations in convolutional neural networks
Ludovica de Paolis, Fabio Anselmi, Alessio Ansuini et al. (+1 more)
Mathematical modeling of visual textures traces back to Julesz's intuition that texture perception in humans is based on local correlations between image features. An influential approach for texture analysis and generation generalizes this notion to linear correlations between the nonlinear features computed by convolutional neural networks (CNNs), compiled into Gram
matrices. Given that CNNs are often used as models for the visual system, it is natural to ask whether such "texture representations" spontaneously align with the textures' perceptual content, and in particular whether those CNNs that are regarded as better models for the visual system also possess more human-like texture representations. Here we quantify the perceptual content captured by feature correlations computed for a diverse pool of CNNs, and we compare it to the models' perceptual
alignment with the mammalian visual system as measured by
Brain-Score. Surprisingly, we find that there is no connection between conventional measures of CNN quality as a model of the visual system and its
alignment with human texture perception. We conclude that texture perception involves mechanisms that are distinct from those that are commonly modeled using approaches based on CNNs trained on object recognition, possibly depending on the integration of contextual information.
Read the full abstract ▾
All authors Ludovica de Paolis, Fabio Anselmi, Alessio Ansuini, Eugenio Piasini
Computer Vision
Regularizing Attention Scores with Bootstrapping
Neo Christopher Chung and Maxim Laletin
Vision transformers (ViT) rely on
attention mechanism to weigh input features, and therefore
attention scores have naturally been considered as explanations for its decision-making process. However,
attention scores are almost always non-zero, resulting in noisy and diffused
attention maps and limiting interpretability. Can we quantify uncertainty measures of
attention scores and obtain regularized
attention scores? To this end, we consider
attention scores of ViT in a statistical framework where independent noise would lead to insignificant yet non-zero scores. Leveraging statistical learning techniques, we introduce the bootstrapping for
attention scores which generates a baseline distribution of
attention scores by resampling input features. Such a bootstrap distribution is then used to estimate significances and posterior probabilities of
attention scores. In natural and medical images, the proposed
Attention Regularization approach demonstrates a straightforward removal of spurious
attention arising from noise, drastically improving shrinkage and sparsity. Quantitative evaluations are conducted using both simulation and real-world
datasets. Our study highlights bootstrapping as a practical regularization tool when using
attention scores as explanations for ViT. Code available: https://github.com/ncchung/AttentionRegularization
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
SECURE: Stable Early Collision Understanding via Robust Embeddings in Autonomous Driving
Wenjing Wang, Wenxuan Wang and Songning Lai
While deep learning has significantly advanced accident anticipation, the robustness of these safety-critical systems against real-world perturbations remains a major challenge. We reveal that state-of-the-art models like CRASH, despite their high performance, exhibit significant instability in predictions and latent representations when faced with minor input perturbations, posing serious reliability risks. To address this, we introduce SECURE - Stable Early Collision Understanding Robust
Embeddings, a framework that formally defines and enforces model robustness. SECURE is founded on four key attributes: consistency and stability in both prediction space and latent feature space. We propose a principled training methodology that fine-tunes a baseline model using a multi-objective loss, which minimizes divergence from a reference model and penalizes sensitivity to
adversarial perturbations. Experiments on DAD and CCD
datasets demonstrate that our approach not only significantly enhances robustness against various perturbations but also improves performance on clean data, achieving new state-of-the-art results.
Read the full abstract ▾
Computer Vision
Human Pose Estimation in Trampoline Gymnastics: How to Improve Performance on Extreme Poses
Léa Drolet-Roy, Victor Nogues, Bérenger Chedal-Anglay et al. (+4 more)
Trampoline gymnastics involves extreme human poses and uncommon viewpoints, on which state-of-the art pose estimation models tend to under-perform. We demonstrate that this problem can be addressed by fine-tuning a pose estimation model on a combination of real extreme poses and domain-specific synthetic poses (STP). We generate STP from motion capture recordings of trampoline routines. We propose a pipeline to fit noisy motion capture data to a parametric human model, then generate multi-view realistic images with high-fidelity keypoint labels. The fine-tuned ViTPose model tested on real multi-view images exhibits accuracy improvements in 2D which translate to improved 3D triangulation. In 2D, we obtain a performance similar to state-of-the-art models on the MS COCO validation set while evaluating on significantly more challenging data, bridging the performance gap between common and extreme poses. In 3D, we reduce the MPJPE by 46.1 mm with our best model, which represents an improvement of 42.7% compared to the pretrained ViTPose model. Our code and data are available at https://github.com/VisionICLab/trampoline_syn_data.
Read the full abstract ▾
All authors Léa Drolet-Roy, Victor Nogues, Bérenger Chedal-Anglay, Sylvain Gaudet, Eve Charbonneau, Mickaël Begon, Lama Séoud
Computer Vision
ViTs for Action Classification in Videos: An Approach to Risky Tackle Detection in American Football Practice Videos
Syed Ahsan Masud Zaidi, William Hsu and Scott Dietrich
Early identification of hazardous actions in contact sports enables timely intervention and improves player safety. We present a method for detecting risky tackles in American football practice
videos and introduce a substantially expanded
dataset for this task. Our work contains 733 single-athlete-dummy tackle clips, each temporally localized around first point contact and labeled with a strike zone component of the standardized Assessment for Tackling Technique (SATT-3), extending prior work that reported 178 annotated
videos. Using a
Vision transformer-based model with imbalance-aware training, we obtain risky recall of 0.67 and Risky F1 of 0.59 under crossvalidation. Relative to the previous baseline in a smaller subset (risky recall of 0.58; Risky F1 0.56 ), our approach improves risky recall by more than 8% points on a much larger
dataset. These results indicate that the
vision transformer-based
video analysis, coupled with careful handling of class imbalance, can reliably detect rare but safety-critical tackling patterns, offering a practical pathway toward coach-centered injury prevention tools.
Read the full abstract ▾
Computer Vision
Sparse Spectral LoRA: Routed Experts for Medical VLMs
Omid Nejati Manzari, Hojat Asgariandehkordi, Taha Koleilat et al. (+2 more)
Large
vision-
language models (VLMs) excel on general
benchmarks but often lack robustness in medical imaging, where heterogeneous supervision induces cross-
dataset interference and sensitivity to data regime (i.e., how the supervisory signals are mixed). In realistic clinical workflows, data and tasks arrive sequentially, so naive continual training further leads to catastrophic forgetting. To address these challenges, we propose MedQwen, a parameter-efficient medical VLM that couples a spectrally routed Mixture-of-Experts (MoE) with a theoretically grounded scaling rule that aligns low-rank updates with a full-rank, fully fine-tuned MoE, without changing the base architecture. Concretely, we initialize each expert from non-overlapping singular value decomposition (SVD) segments of the pretrained weight and introduce a residual compensation and scaling scheme to enable stable expert specialization and consistent routing under distribution shift. Across 23 medical
datasets covering visual question answering, report generation, radiology classification, and
hallucination mitigation, MedQwen achieves strong, reliable performance: it approaches full fine-tuning on zero-shot classification with 339$\times$ fewer trainable parameters, and reduces sequential forgetting to $\sim$5% where strong baselines degrade by $>$20-50%.
Read the full abstract ▾
All authors Omid Nejati Manzari, Hojat Asgariandehkordi, Taha Koleilat, Yiming Xiao, Hassan Rivaz
Computer Vision
Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering
Marco Morini, Sara Sarto, Marcella Cornia et al. (+2 more)
Knowledge-based Visual Question Answering (KB-VQA) requires
Multimodal Large
Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence. However,
retrieval often introduces noisy and partially relevant content, while images contain distracting visual regions, causing pretrained MLLMs to overlook the evidence that actually supports the answer. To address this, we introduce Look Twice (LoT), a training-free inference-time framework that turns the model's own internal
attention into an explicit
multimodal evidence-selection mechanism. LoT first leverages the model's internal
attention patterns to identify query-relevant image regions and textual sentences, filters
attention sinks and distracting content, and reformulates the input to explicitly highlight the selected evidence before answer generation. The method requires no parameter updates, auxiliary models, or architectural modifications. Across four KB-VQA
benchmarks and ten off-the-shelf MLLMs ranging from 2B to 38B parameters, LoT improves every evaluated backbone, with average gains of up to +12.5 accuracy points. It also provides further gains when combined with established context-refinement strategies, yielding additional improvements over already refined inputs. These results establish LoT as a general and effective mechanism for enabling pretrained MLLMs to exploit available
multimodal evidence more accurately. Source code is publicly available at https://aimagelab.github.io/LoT/.
Read the full abstract ▾
All authors Marco Morini, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Computer Vision · Cross-listed from cs.GR
Non-Rigid 3D Shape Correspondences: From Foundations to Open Challenges and Opportunities
Aleksei Zhuravlev, Lennart Bastian, Dongliang Cao et al. (+12 more)
Estimating correspondences between deformed shape instances is a long-standing problem in computer graphics; numerous applications, from texture transfer to statistical modelling, rely on recovering an accurate correspondence map. Many methods have thus been proposed to tackle this challenging problem from varying perspectives, depending on the downstream application. This state-of-the-art report is geared towards researchers, practitioners, and students seeking to understand recent trends and advances in the field. We categorise developments into three paradigms: spectral methods based on functional maps, combinatorial formulations that impose discrete constraints, and deformation-based methods that directly recover a global
alignment. Each school of thought offers different advantages and disadvantages, which we discuss throughout the report. Meanwhile, we highlight the latest developments in each area and suggest new potential research directions. Finally, we provide an overview of emerging challenges and opportunities in this growing field, including the recent use of
vision foundation models for zero-shot correspondence and the particularly challenging task of matching partial shapes.
Read the full abstract ▾
All authors Aleksei Zhuravlev, Lennart Bastian, Dongliang Cao, Nafie El Amrani, Paul Roetzer, Viktoria Ehm, Riccardo Marin, Hiroki Nishizawa, Shigeo Morishima, Christian Theobalt, Nassir Navab, Daniel Cremers, Florian Bernard, Zorah Lähner, Vladislav Golyanik
Computer Vision
Camouflage-aware Image-Text Retrieval via Expert Collaboration
Yao Jiang, Zhongkuan Mao, Xuan Wu et al. (+2 more)
Camouflaged scene understanding (CSU) has attracted significant
attention due to its broad practical implications. However, in this field, robust image-text cross-modal
alignment remains under-explored, hindering deeper understanding of camouflaged scenarios and their related applications. To this end, we focus on the typical image-text
retrieval task, and formulate a new task dubbed “camouflage-aware image-text
retrieval” (CA-ITR). We first construct a dedicated camouflage image-text
retrieval dataset (CamoIT), comprising $\sim$10.5K samples with multi-granularity textual annotations.
Benchmark results conducted on CamoIT reveal the underlying challenges of CA-ITR for existing cutting-edge
retrieval techniques, which are mainly caused by objects' camouflage properties as well as those complex image contents. As a solution, we propose a camouflage-expert collaborative network (CECNet), which features a dual-branch visual encoder: one branch captures holistic image representations, while the other incorporates a dedicated model to inject representations of camouflaged objects. A novel confidence-conditioned graph
attention (C\textsuperscript{2}GA) mechanism is incorporated to exploit the complementarity across branches. Comparative experiments show that CECNet achieves $\sim$29% overall CA-ITR accuracy boost, surpassing seven representative
retrieval models. The
dataset and code will be available at https://github.com/jiangyao-scu/CA-ITR.
Read the full abstract ▾
All authors Yao Jiang, Zhongkuan Mao, Xuan Wu, Keren Fu, Qijun Zhao
Computer Vision · Cross-listed from cs.AI
HippoCamp: Benchmarking Contextual Agents on Personal Computers
Zhe Yang, Shulin Tian, Kairui Hu et al. (+9 more)
We present HippoCamp, a new
benchmark designed to evaluate
agents' capabilities on
multimodal file management. Unlike existing
agent benchmarks that focus on tasks like web interaction, tool use, or software automation in generic settings, HippoCamp evaluates
agents in user-centric environments to model individual user profiles and search massive personal files for context-aware
reasoning. Our
benchmark instantiates device-scale file systems over real-world profiles spanning diverse modalities, comprising 42.4 GB of data across over 2K real-world files. Building upon the raw files, we construct 581 QA pairs to assess
agents' capabilities in search, evidence perception, and multi-step
reasoning. To facilitate fine-grained analysis, we provide 46.1K densely annotated structured trajectories for step-wise failure diagnosis. We evaluate a wide range of state-of-the-art
multimodal large
language models (MLLMs) and agentic methods on HippoCamp. Our comprehensive experiments reveal a significant performance gap: even the most advanced commercial models achieve only 48.3% accuracy in user profiling, struggling particularly with long-horizon
retrieval and cross-modal
reasoning within dense personal file systems. Furthermore, our step-wise failure diagnosis identifies
multimodal perception and evidence grounding as the primary bottlenecks. Ultimately, HippoCamp exposes the critical limitations of current
agents in realistic, user-centric environments and provides a robust foundation for developing next-generation personal AI assistants.
Read the full abstract ▾
All authors Zhe Yang, Shulin Tian, Kairui Hu, Shuai Liu, Hoang-Nhat Nguyen, Yichi Zhang, Zujin Guo, Mengying Yu, Zinan Zhang, Jingkang Yang, Chen Change Loy, Ziwei Liu
Computer Vision · Cross-listed from cs.LG
LAtent Phase Inference from Short time sequences using SHallow REcurrent Decoders (LAPIS-SHRED)
Yuxuan Bao, Xingyue Zhang and J. Nathan Kutz
Reconstructing full spatio-temporal dynamics from sparse observations in both space and time remains a central challenge in complex systems, as measurements can be spatially incomplete and can be also limited to narrow temporal windows. Yet approximating the complete spatio-temporal trajectory is essential for mechanistic insight and understanding, model calibration, and operational decision-making. We introduce LAPIS-SHRED (LAtent Phase Inference from Short time sequence using SHallow REcurrent Decoders), a modular architecture that reconstructs and/or forecasts complete spatiotemporal dynamics from sparse sensor observations confined to short temporal windows. LAPIS-SHRED operates through a three-stage pipeline: (i) a SHRED model is pre-trained entirely on simulation data to map sensor time-histories into a structured latent space, (ii) a temporal sequence model, trained on simulation-derived latent trajectories, learns to propagate latent states forward or backward in time to span unobserved temporal regions from short observational time windows, and (iii) at deployment, only a short observation window of hyper-sparse sensor measurements from the true system is provided, from which the frozen SHRED model and the temporal model jointly reconstruct or forecast the complete spatiotemporal trajectory. The framework supports bidirectional inference, inherits data assimilation and multiscale reconstruction capabilities from its modular structure, and accommodates extreme observational constraints including single-frame terminal inputs. We evaluate LAPIS-SHRED on six experiments spanning complex spatio-temporal physics: turbulent flows, multiscale propulsion physics, volatile combustion transients, and satellite-derived environmental fields, highlighting a lightweight, modular architecture suited for operational settings where observation is constrained by physical or logistical limitations.
Read the full abstract ▾
Computer Vision
TRACE: High-Fidelity 3D Scene Editing via Tangible Reconstruction and Geometry-Aligned Contextual Video Masking
Jiyuan Hu, Zechuan Zhang, Zongxin Yang et al. (+1 more)
Existing 3D Gaussian Splatting (3DGS) editing methods primarily focus on appearance modification and often struggle to support flexible geometry editing while preserving structural integrity and scene-consistent appearance. To address this limitation, we present TRACE, a mesh-guided 3DGS editing framework that automatically aligns explicit 3D geometry with Gaussian scenes and decouples
Geometric Anchoring from Appearance Harmonization. First, Multi-view 3D-Anchor Synthesis, trained on our MV-TRACE
dataset for scene-coherent object addition and modification, generates geometrically aligned editing anchors, while Tangible Geometry
Alignment (TGA) performs coarse-to-fine mesh-scene registration. Then, Contextual
Video Masking (CVM) integrates projected 3D anchors into an autoregressive
video diffusion pipeline, harmonizing their appearance with the surrounding scene while maintaining multi-view consistency. We evaluate TRACE on eight held-out scenes across six editing categories. TRACE completes each edit in approximately 10 minutes on a single NVIDIA RTX Pro 6000 GPU. Extensive experiments demonstrate consistent improvements over existing methods in editing versatility, structural integrity, semantic
alignment, multi-view consistency, and visual quality.
Read the full abstract ▾
All authors Jiyuan Hu, Zechuan Zhang, Zongxin Yang, Yi Yang
Computer Vision
Neural Harmonic Textures for High-Quality Primitive Based Neural Reconstruction
Jorge Condor, Nicolas Moenne-Loccoz, Merlin Nimier-David et al. (+3 more)
Primitive-based methods such as 3D Gaussian Splatting have recently become the state-of-the-art for novel-view synthesis and related reconstruction tasks. Compared to neural fields, these representations are more flexible, adaptive, and scale better to large scenes. However, the limited expressivity of individual primitives makes modeling high-frequency detail challenging. We introduce Neural Harmonic Textures, a neural representation approach that anchors latent feature vectors on a virtual scaffold surrounding each primitive. These features are interpolated within the primitive at ray intersection points. Inspired by
Fourier analysis, we apply periodic activations to the interpolated features, turning alpha blending into a weighted sum of harmonic components. The resulting signal is then decoded in a single deferred pass using a small neural network, significantly reducing computational cost. Neural Harmonic Textures yield state-of-the-art results in real-time novel view synthesis while bridging the gap between primitive- and neural-field-based reconstruction. Our method integrates seamlessly into existing primitive-based pipelines such as 3DGUT, Triangle Splatting, and 2DGS. We further demonstrate its generality with applications to 2D image fitting and semantic reconstruction.
Read the full abstract ▾
All authors Jorge Condor, Nicolas Moenne-Loccoz, Merlin Nimier-David, Piotr Didyk, Zan Gojcic, Qi Wu
Computer Vision · Cross-listed from cs.HC
True (VIS) Lies: Analyzing How Generative AI Recognizes Intentionality, Rhetoric, and Misleadingness in Visualization Lies
Graziano Blasilli and Marco Angelini
This study investigates the ability of
multimodal Large
Language Models (LLMs) to identify and interpret misleading visualizations, and recognize these observations along with their underlying causes and potential intentionality. Our analysis leverages concepts from visualization rhetoric and a newly developed taxonomy of authorial intents as explanatory lenses. We formulated three research questions and addressed them experimentally using a
dataset of 2,336 COVID-19-related tweets, half of which contain misleading visualizations, and supplemented it with real-world examples of perceptual, cognitive, and conceptual errors drawn from VisLies, the IEEE VIS community event dedicated to showcasing deceptive and misleading visualizations. To ensure broad coverage of the current LLM landscape, we evaluated 16 state-of-the-art models. Among them, 15 are open-weight models, spanning a wide range of model sizes, architectural families, and
reasoning capabilities. The selection comprises small models, namely Nemotron-Nano-V2-VL (12B parameters), Mistral-Small-3.2 (24B), DeepSeek-VL2 (27B), Gemma3 (27B), and GTA1 (32B); medium-sized models, namely Qianfan-VL (70B), Molmo (72B), GLM-4.5V (108B), LLaVA-NeXT (110B), and Pixtral-Large (124B); and large models, namely Qwen3-VL (235B), InternVL3.5 (241B), Step3 (321B), Llama-4-Maverick (400B), and Kimi-K2.5 (1000B). In addition, we employed OpenAI GPT-5.4, a frontier proprietary model. To establish a human perspective on these tasks, we also conducted a user study with visualization experts to assess how people perceive rhetorical techniques and the authorial intentions behind the same misleading visualizations. This allows comparison between model and expert behavior, revealing similarities and differences that provide insights into where LLMs align with human judgment and where they diverge.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
A ROS 2 Wrapper for Florence-2: Multi-Mode Local Vision-Language Inference for Robotic Systems
J. E. Domínguez-Vidal
Foundation
vision-
language models are becoming increasingly relevant to robotics because they can provide richer semantic perception than narrow task-specific pipelines. However, their practical adoption in
robot software stacks still depends on reproducible middleware integrations rather than on model quality alone. Florence-2 is especially attractive in this regard because it unifies captioning, optical character recognition, open-vocabulary detection, grounding and related
vision-
language tasks within a comparatively manageable model size. This article presents a ROS 2 wrapper for Florence-2 that exposes the model through three complementary interaction modes: continuous topic-driven processing, synchronous service calls and asynchronous actions. The wrapper is designed for local execution and supports both native installation and Docker container deployment. It also combines generic JSON outputs with standard ROS 2 message bindings for detection-oriented tasks. A functional validation is reported together with a throughput study on several GPUs, showing that local deployment is feasible with consumer grade hardware. The repository is publicly available here: https://github.com/JEDominguezVidal/florence2_ros2_wrapper
Read the full abstract ▾
Computer Vision
Open-Set Supervised 3D Anomaly Detection: An Industrial Dataset and a Generalisable Framework for Unknown Defects
Hanzhe Liang, Luocheng Zhang, Junyang Xia et al. (+7 more)
Although self-supervised 3D anomaly detection assumes that acquiring high-precision point clouds is computationally expensive, in real manufacturing scenarios it is often feasible to collect a limited number of anomalous samples. Therefore, we study open-set supervised 3D anomaly detection, where the model is trained with only normal samples and a small number of known anomalous samples, aiming to identify unknown anomalies at test time. We present Open-Industry, a high-quality industrial
dataset containing 15 categories, each with five real anomaly types collected from production lines. We first adapt general open-set anomaly detection methods to accommodate 3D point cloud inputs better. Building upon this, we propose Open3D-AD, a point-cloud-oriented approach that leverages normal samples, simulated anomalies, and partially observed real anomalies to model the probability density distributions of normal and anomalous data. Then, we introduce a simple Correspondence Distributions Subsampling to reduce the overlap between normal and non-normal distributions, enabling stronger dual distributions modeling. Based on these contributions, we establish a comprehensive
benchmark and evaluate the proposed method extensively on Open-Industry as well as established
datasets including Real3D-AD and Anomaly-ShapeNet.
Benchmark results and ablation studies demonstrate the effectiveness of Open3D-AD and further reveal the potential of open-set supervised 3D anomaly detection.
Read the full abstract ▾
All authors Hanzhe Liang, Luocheng Zhang, Junyang Xia, HanLiang Zhou, Bingyang Guo, Yingxi Xie, Can Gao, Ruiyun Yu, Jinbao Wang, Pan Li
Computer Vision · Cross-listed from eess.IV
AdaLoRA-QAT: Adaptive Low-Rank and Quantization-Aware Segmentation
Prantik Deb, Srimanth Dhondy, N. Ramakrishna et al. (+3 more)
Chest X-ray (CXR)
segmentation is an important step in computer-aided diagnosis, yet deploying large foundation models in clinical settings remains challenging due to computational constraints. We propose AdaLoRA-QAT, a two-stage fine-tuning framework that combines adaptive low-rank encoder adaptation with full quantization-aware training. Adaptive rank allocation improves parameter efficiency, while selective mixed-precision INT8 quantization preserves structural fidelity crucial for clinical reliability. Evaluated across large-scale CXR
datasets, AdaLoRA-QAT achieves 95.6% Dice, matching full-precision SAM decoder fine-tuning while reducing trainable parameters by 16.6\times and yielding 2.24\times model compression. A Wilcoxon signed-rank test confirms that quantization does not significantly degrade
segmentation accuracy. These results demonstrate that AdaLoRA-QAT effectively balances accuracy, efficiency, and structural trust-worthiness, enabling compact and deployable foundation models for medical image
segmentation. Code and pretrained models are available at: https://prantik-pdeb.github.io/adaloraqat.github.io/
Read the full abstract ▾
All authors Prantik Deb, Srimanth Dhondy, N. Ramakrishna, Anu Kapoor, Raju S. Bapi, Tapabrata Chakraborti
Computer Vision
Looking into a Pixel by Nonlinear Unmixing – A Generative Approach
Maofeng Tang and Hairong Qi
Due to the large footprint of pixels in remote sensing imagery, hyperspectral unmixing (HU) has become an important and necessary procedure in hyperspectral image analysis. Traditional HU methods rely on a prior spectral mixing model, especially for nonlinear mixtures, which has largely limited the performance and generalization capacity of the unmixing approach. In this paper, we address the challenging problem of hyperspectral nonlinear unmixing (HNU) without explicit knowledge of the mixing model. Inspired by the principle of generative models, where images of the same distribution can be generated as that of the training images without knowing the exact probability distribution function of the image, we develop an invertible mixing-unmixing process via a bi-directional GAN framework, constrained by both the cycle consistency and the linkage between linear and nonlinear mixtures. The combination of cycle consistency and linear linkage provides powerful constraints without requiring an explicit mixing model. We refer to the proposed approach as the linearly-constrained CycleGAN unmixing net, or LCGU net. Experimental results indicate that the proposed LCGU net exhibits stable and competitive performance across different
datasets compared with other state-of-the-art model-based HNU methods.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Toward Personalized Darts Training: A Data-Driven Framework Based on Skeleton-Based Biomechanical Analysis and Motion Modeling
Zhantao Chen, Dongyi He, Jin Fang et al. (+4 more)
As sports training becomes more data-driven, traditional dart coaching based mainly on experience and visual observation is increasingly inadequate for high-precision, goal-oriented movements. Although prior studies have highlighted the importance of release parameters, joint motion, and coordination in dart throwing, most quantitative methods still focus on local variables, single-release metrics, or static template matching. These approaches offer limited support for personalized training and often overlook useful movement variability. This paper presents a data-driven dart training assistance system. The system creates a closed-loop framework spanning motion capture, feature modeling, and personalized feedback. Dart-throwing data were collected in markerless conditions using a Kinect 2.0 depth sensor and an optical camera. Eighteen kinematic features were extracted from four biomechanical dimensions: three-link coordination, release velocity, multi-joint angular configuration, and postural stability. Two modules were developed: a personalized optimal throwing trajectory model that combines historical high-quality samples with the minimum jerk criterion, and a motion deviation diagnosis and recommendation model based on z-scores and hierarchical logic. A total of 2,396 throwing samples from professional and non-professional athletes were collected. Results show that the system generates smooth personalized reference trajectories consistent with natural human movement. Case studies indicate that it can detect poor trunk stability, abnormal elbow displacement, and imbalanced velocity control, then provide targeted recommendations. The framework shifts dart evaluation from deviation from a uniform standard to deviation from an individual's optimal control range, improving personalization and interpretability for darts training and other high-precision target sports.
Read the full abstract ▾
All authors Zhantao Chen, Dongyi He, Jin Fang, Xi Chen, Yishuo Liu, Xiaozhen Zhong, Xuejun Hu