Computer Vision
Mind the Gaps: A Curated Benchmark for Form Field Detection
Iheb Brini, Omar Moured, Hamza Gbada et al. (+1 more)
Form Field Detection (FFD) is a fundamental component of document understanding systems, enabling applications ranging from large-scale industrial digitization to accessible form interaction for automated analysis. Unlike conventional object detection tasks, FFD is inherently challenging because fields are often defined by layout structure and whitespace rather than visible foreground content. Existing large-scale
datasets frequently rely on heuristic annotation pipelines, resulting in noisy and inconsistent labels that hinder reliable evaluation. In this work, we introduce mini-CommonForms, a carefully curated FFD
benchmark with consistent, high-quality annotations, and present a detailed evaluation of state-of-the-art detection approaches. The
benchmark is designed to support reproducible research in document automation and accessibility-oriented applications.
Dataset and code are available at https://github.com/moured/mini-commonforms
Read the full abstract ▾
All authors Iheb Brini, Omar Moured, Hamza Gbada, Elisa Barney
Computer Vision
Which Terrain Is Better? Preference Learning with VLM Prototypes for Off-Road Traversability Ranking
Ji-Hoon Hwang, Jisung Bae, E-In Son et al. (+3 more)
In
vision-based off-road
navigation, a
robot needs to know not only which obstacles to avoid but also which terrain is better. The first is handled by freespace detection or semantic
segmentation. The second is usually answered with a traversability score, but no universal ground truth exists for such a score, so perception falls back on a predefined value per semantic class or a freespace confidence. These scores say what a region is, not which region a
robot should prefer. We therefore formulate this preference as visual traversability ranking, an ordering of visible terrain that can be supervised by comparisons between two regions. Standard annotations do not label preference, but they imply its direction. We present TravPro, which converts these annotations into ordered region pairs and fits a small readout on frozen
vision–
language model (VLM) patch
tokens to these pairs. The
tokens are clustered once into a fixed prototype bank, and the readout learns a preference score per prototype. The readout is then applied to every patch and serves as a teacher that turns sparse comparisons into dense preference pseudo-labels without pixel-wise annotation. An RGB student distills these maps into a dense terrain-preference map together with a non-ground mask that excludes obstacles and background from the ranking. On five unseen domains, TravPro reaches a mean pairwise accuracy of 0.915 against 0.783 for the strongest baseline, producing an ordering sensitive to surface condition that a per-class value cannot represent. The same VLM and the same supervision yield no such ordering when the VLM is prompted and the supervision is used as dense targets; what matters is how they are used.
Read the full abstract ▾
All authors Ji-Hoon Hwang, Jisung Bae, E-In Son, Dong-Wook Kim, Jung-Taak Kim, Seung-Woo Seo
Computer Vision
Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms
Yueyan Li, Haibo Wang, Caixia Yuan et al. (+1 more)
Despite impressive visual quality, state-of-the-art
video diffusion models often generate content that violates real-world physical laws. While existing solutions rely on external priors or specialized data, we investigate the root cause by exploring the internal mechanisms of these models. Specifically, we present the first interpretability study on the ”motion planning” process of text-to-
video diffusion models, revealing how motion trajectories form during early denoising stages. Building upon the ”first shape, then details” finding, we combine cross-
attention trajectory patterns with causal head contributions to identify a specific subset of
attention heads driving motion planning. Further, our self-
attention analysis shows that Rotary Position
Embedding (RoPE) induces excessive spatial
attention decay. This causes early candidate regions to prematurely lock into physically implausible positions, suppressing reasonable trajectories in adjacent frames and triggering generation failure modes. To address this fundamental flaw, we propose a lightweight architectural modification that scales the frequency of RoPE across different denoising steps. This strategy reduces excessive
attention decay, helping the model explore better candidate regions to establish coherent physical motion. Finally, training-free and training-based experiments confirm the effectiveness of our approach in enhancing the physical commonsense of generated
videos.
Read the full abstract ▾
All authors Yueyan Li, Haibo Wang, Caixia Yuan, Xiaojie Wang
Computer Vision
Reassessing Global Gradient-Norm Imbalance in BLIP Fine-Tuning Across Physical Domains
Kiran Naseer, Samreen Azhar and Dwarikanath Mahapatra
Imbalanced gradient magnitudes between the visual and
language pathways of a
vision-
language model are often treated as a defect to be corrected. We test that premise for one family of correction, deliberately excluding adaptive, signal-driven schemes (e.g. BalGrad, OGM, PMR, CGGM), which are a mechanistically distinct class outside this study's scope. Measuring the
language-to-visual gradient-norm ratio, reported in parameter-normalised form, across nine fine-tuning conditions, three seeds, and three captioning
datasets spanning distinct physical domain shifts –
underwater, aerial, radiological – we find imbalance magnitude varies markedly across domains with no predictable ordering. A plain learning-rate reduction cuts imbalance substantially and lands within a few BLEU points of the best method on every
dataset. Staged freezing reduces the ratio on every domain yet never ranks first; a schedule-only control isolates freezing as the cause on one
dataset but not the other two. Forcing the two gradient groups to equal magnitude drives per-parameter imbalance close to zero on every domain, yet is both the best result in the study and the worst placement among full fine-tuning methods, on different
datasets, with identical settings. Reductions in gradient-norm ratio do not consistently predict captioning performance across domains, and how a given level of balance is reached matters as much as the level itself. As a secondary finding, a commonly reused LoRA configuration applied to BLIP silently adapts zero visual parameters; correcting it improves BLEU-4 on all three
datasets.
Read the full abstract ▾
Computer Vision
Compact Low-Cost Hyperspectral Imaging via Angular-to-Spectral Diversity Conversion
Kazuma Fujiwara, Takuya Funatomi, Kazuya Kitano et al. (+2 more)
Snapshot hyperspectral imaging avoids sequential scanning, but systems that jointly achieve stable reconstruction, low cost, and compact optics remain limited. We present a snapshot hyperspectral imaging system based on angular-to-spectral diversity conversion. A tapered kaleidoscope creates replicated views with distinct incidence directions, and a directly attached birefringent filter converts them into view-channel-dependent spectral transmittances, yielding complementary measurements that better condition the inverse problem for more stable single-shot spectral reconstruction. The system preserves a simple pixel-wise linear model for fast non-learning-based reconstruction and uses only off-the-shelf components without relay optics or cascaded modules. We select the birefringent filter configuration using a condition-number-based criterion and validate the system on both synthetic and real data.
Read the full abstract ▾
All authors Kazuma Fujiwara, Takuya Funatomi, Kazuya Kitano, Yuki Fujimura, Yasuhiro Mukaigawa
Computer Vision
Beyond UV Mapping: Mesh Texture Compression via Surface-Aligned Texture Fields
Jianqiang Wang, Junhui Hou, Siyu Ren et al. (+2 more)
Mesh texture compression typically relies on 2D UV atlases, whose chart discontinuities and mapping overhead can limit coding efficiency. To tackle this challenge, we introduce TexF, a surface-aligned texture field that organizes texture attributes in sparse voxels derived from the mesh surface. This representation supports high-resolution textures while preserving local 3D correlations for compression and enabling direct surface queries. For bitstream compression, TexF reuses established 3D attribute codecs, with voxel locations reconstructed from the decoded mesh without separate transmission. For GPU-resident compression, we develop 3DNTC, which combines quantized hash features with a lightweight decoder for random-access reconstruction at surface positions. Differentiable rendering enables image-space refinement of both voxel attributes and compressed neural fields. Experiments on the MPEG and AOM mesh compression
benchmarks demonstrate improved average rate-distortion performance over representative UV-based methods for both bitstream and GPU-resident compression. 3DNTC also supports real-time rendering.
Read the full abstract ▾
All authors Jianqiang Wang, Junhui Hou, Siyu Ren, Weiyao Lin, Wenping Wang
Computer Vision
PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding
Siru Zhong, Qiongyan Wang, Xiaohui Lv et al. (+5 more)
Long-
video understanding must capture transient visual evidence under strict
token budgets, yet existing methods compress frames, append
memory tokens, or alter internal key-value (KV) caches. We introduce Prefix-Steered Recurrent
Memory (PREM), a
memory-
token-free framework for frozen
vision-
language models (VLMs). PREM separates
video ingestion from query answering: a recurrent writer distills visual streams into a compact 256 KiB multi-slot associative state, while a question-conditioned readout adds
memory-derived key/value (K/V) steering modulations to existing non-visual
prompt prefixes during prefill. This enables write-once, query-many inference without extra
prompt tokens or decoding recurrence. Across six long-
video benchmarks in offline and streaming end-of-stream settings, PREM consistently outperforms frozen baselines at every evaluated visual budget. Under a constrained budget of 16 frames, PREM improves macro-average accuracy by 3.06% on Qwen2.5-VL-3B, with gains of 11.0% on action antonym identification and 9.9% on localized needle
retrieval. These gains require tuning 0.24% of backbone parameters at 0.03 GiB of peak GPU
memory overhead.
Read the full abstract ▾
All authors Siru Zhong, Qiongyan Wang, Xiaohui Lv, Yuzheng Zhuang, Shuai Tao, Wulong Liu, Haohuan Fu, Yuxuan Liang
Computer Vision
PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models
Weihan Cai, Hao Tan, Xinping Gao et al. (+2 more)
Prompt learning efficiently adapts
vision-
language models (VLMs) to downstream tasks, but gains on seen classes often come at the expense of generalization to unseen classes. To address this limitation, we propose
prompt ensembling with training-free routing (PETR), whose key innovation is a carefully designed dual-
prompt architecture: two complementary
prompts are learned from different data and objectives to emphasize seen class discrimination and unseen-class generalization, respectively. During training, both
prompts are fine-tuned using a shared frozen CLIP backbone, and statistical information is collected from the training set logits. At inference time, we determine the similarity of each test sample to seen data, and route the sample to the most appropriate
prompt branch. To the best of our knowledge, this is the first
prompt tuning framework that performs training-free adaptive routing based on statistical similarity. This design provides an interpretable routing signal and avoids common MoE-style routing pathologies, such as router training instability and load imbalance. Extensive experiments on 11
benchmark datasets demonstrate that our framework consistently outperforms previous methods on both seen and unseen classes, achieving new state-of-the-art results.
Read the full abstract ▾
All authors Weihan Cai, Hao Tan, Xinping Gao, Shibiao Xu, Jun Wan
Computer Vision
StyleAT: Defending Face Recognition Against Semantic Attacks
Ben Shapira, Roi Cohen, Shang-Tse Chen et al. (+1 more)
With face-recognition models now embedded in everyday authentication and surveillance, recent works have pinpointed a critical weakness: these models remain acutely vulnerable to
adversarial semantic edits. I.e., adversarially produced semantic alterations to the input, such as slight aging or pose changes, can induce misclassifications. Certain existing attacks are powerful, but they can be computationally costly, rendering them inadequate for developing defenses (e.g., through
adversarial training). To fill the gap, we introduce BoundStyle, a potent semantic attack operating in StyleGAN's rich latent space to maximize misclassification rates. Notably, BoundStyle achieves high attack success rates while being ${\sim}{\times}9.5$ faster than existing state-of-the-art attacks, making it suitable for
adversarial training. Building on BoundStyle, we develop StyleAT, an efficient
adversarial training scheme that incorporates low-budget attack variants yet defends against stronger and unseen semantic attacks. We evaluate on two
datasets unseen during training and seven models, and find that StyleAT boosts robust accuracy against state-of-the-art attacks and outperforms common defenses in various settings.
Read the full abstract ▾
All authors Ben Shapira, Roi Cohen, Shang-Tse Chen, Mahmood Sharif
Computer Vision
Collapse, Not Complexity: Failure-Conditioned Decomposition Repair for End-to-End Document Parsing
Xingyu Lin and Dehui Du
End-to-end document parsers increasingly offer an optional
reasoning mode for complex pages. On a 180-page
entropy-stratified discovery sample with one frozen 4B checkpoint, complexity is the wrong decision variable.
Reasoning lowers mean quality by 2.21 Overall at 1.54x
tokens; a preregistered input-only model cannot predict its signed benefit (held-out AUROC 0.47, indistinguishable from chance). The benefit concentrates on pages whose ordinary pass has already collapsed, and they do not look complex: shared collapses have lower layout
entropy than healthy ones yet consume 19x the
tokens as degenerate repetition that doubling the budget does not cure. Switching modes rarely repairs them: 83% recur under
reasoning. We instead detect collapse from the ordinary-pass trace, decompose the page by projection, and re-parse each region. Repair gains 1.40 Overall (95% CI [0.68, 2.16]) at 1.13x
tokens, replicates across three checkpoints, and, with all parameters frozen, gains 2.41 (CI [1.64, 3.46]) on the remaining 1,175
benchmark pages.
Read the full abstract ▾
Computer Vision
An Efficient and Effective Watermarking Scheme for the Protection of the Intellectual Property Rights of Video Generative Models
Wenhong Huang, Jianwei Fei, Benedetta Tondi et al. (+2 more)
The rapid development of
video generative models (VGMs) has enabled the generation of highly realistic synthetic
videos, raising concerns about the intellectual property rights (IPR) of these models. In particular, two closely related forensic tasks remain largely unaddressed: synthetic
video verification (determining whether a
video was generated by a protected VGM) and model ownership verification (determining whether a suspect VGM is an unauthorized copy of a protected VGM). In this paper, we propose a new in-generation watermarking scheme that can address the two verification tasks. First, a novel
video watermarking network named VidMark is presented, which incorporates a two-scale discrete wavelet transform (DWT) decomposition and a global temporal
attention block (GTAB) to enhance watermark robustness and imperceptibility. Second, we present a decoder-guided fine-tuning procedure. By leveraging the frozen VidMark decoder, this process enables VGMs to synthesize
videos carrying an imperceptible, robust, and model-specific watermark. Finally, two verification frameworks are established to perform synthetic
video verification and model ownership verification. Extensive experiments on representative VGMs demonstrate that the proposed scheme achieves over 99% watermark extraction accuracy and 100% verification accuracy on both tasks, with negligible impact on
video generation quality. Furthermore, the watermarks exhibit strong robustness against a comprehensive range of
video-level and model-level attacks.
Read the full abstract ▾
All authors Wenhong Huang, Jianwei Fei, Benedetta Tondi, Bin Ma, Fangjun Huang
Computer Vision
G6D: Geometric Learning-Free RGB-D 6D Pose Solver for Robotic Manipulation
Yixuan Liang, William Chen, Yunan Wang et al. (+4 more)
6D object pose estimation is fundamental to robotic manipulation and automation. Recent zero-shot methods have significantly improved generalization to unseen objects, but most still rely on large-scale pretrained models with substantial GPU computation and
memory demands. These requirements complicate deployment on robotic platforms where perception, planning, and control share limited computational resources, while learned intermediate representations offer limited
geometric interpretability for task-specific adaptation. To address these limitations, we propose G6D, a learning-free, geometry-driven RGB-D 6D pose solver. Given an RGB-D observation, an object instance mask, camera intrinsics, and a CAD model, G6D generates pose hypotheses through template-based
geometric matching and refines them using silhouette and depth consistency, forming a purely geometry-driven pose estimation paradigm. This paradigm requires neither pretrained visual models nor target-specific training and preserves interpretable
geometric representations throughout pose estimation. Moreover, adjustable hypothesis counts provide flexible accuracy-computation trade-offs, while a CPU-only configuration supports deployment without GPU resources. Experiments on LineMOD and five BOP19
datasets demonstrate advanced performance. Real-world pick-and-place experiments further demonstrate G6D's applicability to robotic manipulation. The complete project is publicly available at https://ai4control.github.io/G6D-Project-Page .
Read the full abstract ▾
All authors Yixuan Liang, William Chen, Yunan Wang, Jizhou Yan, Zhao Jin, Changling Liu, Chuxiong Hu
Computer Vision
MaskVLA: Visual Masking Against Trajectory Overfitting of Vision-Language-Action Model
Yuxuan Jiang, Jiaying Huang, Ge Wang et al. (+9 more)
Vision-
Language-Action (VLA) models integrate
vision-
language understanding with executable
robot actions, enabling end-to-end learning for
robot control. However, our empirical analysis reveals that existing models exhibit severe trajectory overfitting when finetuned on limited
datasets. To guide the model in effectively utilizing wrist camera information, we propose MaskVLA, a masking-based fine-tuning strategy. By randomly masking a small portion of the main camera's visual information, the model is guided to autonomously learn more fine-grained, task-relevant, and effective visual features. This process leads to the emergence of robust policies, thereby enhancing the model's capability to tackle complex manipulation tasks and improving its generalization performance. Our method has been comprehensively evaluated on RoboTwin 2.0, achieving an average success rate improvement of 23.2% and 16.8% compared to $\pi_0$ and OpenVLA-OFT, respectively. Furthermore, experiments on real-world ALOHA
robots also demonstrate the effectiveness of our approach.
Read the full abstract ▾
All authors Yuxuan Jiang, Jiaying Huang, Ge Wang, Shenhao Yan, Jiahao Yang, Chengsi Yao, Qi Liu, Qing Zhao, Shuguang Cui, Yiming Zhao, Yatong Han, Zhen Li
Computer Vision
Transferring Visual Explanations: How Cross-Architecture Knowledge Distillation Affects Model Interpretability
Aleks Czufarow and Ihor Babin
Deploying efficient neural networks is essential in resource-constrained environments, yet compact models often sacrifice interpretability - a critical in safety-critical domains such as autonomous driving and medicine. This study investigates whether Knowledge
Distillation transfers the spatial feature attribution of a large teacher network to a compact student. To assess the influence of the KD scheme on interpretability, we distill a ResNet-152 teacher into a ResNet-34 student on ImageNet-1K across five configurations by systematically varying the
distillation temperature and soft-label loss weight. Models are evaluated on top-1 accuracy, along with two interpretability metrics: Relevance Mass Accuracy and Relevance Rank Accuracy. These metrics are computed via Grad-CAM heatmaps benchmarked against ground-truth object masks. Our results show that top-1 accuracy ranges from 71.6% to 74.0%. For Grad-CAM, RMA ranges from 7.7% to 9.7% and RRA from 7.3% to 10.1%; for Guided Grad-CAM, RMA ranges from 16.1% to 18.6% and RRA from 15.9% to 21.5%. Interpretability proves far more sensitive to the soft-label weight than to the temperature: keeping the student anchored to hard labels preserves both accuracy and coarse localization, whereas weighting the teacher heavily degrades both. Fine-grained attribution, however, fell below the undistilled baseline in every configuration tested, indicating that logit
distillation transmits where a model attends more readily than the pixel-level structure of that
attention. We evaluate 12 cross-architecture combinations of convolutional and
transformer-based models, revealing that the inheritance of fine-grained spatial
reasoning is fundamentally bottlenecked by the student's intrinsic structural biases. To our knowledge, this is the first application of this interpretability-aware evaluation framework - previously used for neural network
pruning - to KD.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Resolution-Flexible Decoding for Hybrid Neural Video Representations
Taiga Hayami, Masaya Takabe and Hiroshi Watanabe
Neural
video representations (NVRs) represent
videos using neural network parameters and, in hybrid formulations, frame-wise latent
embeddings. Although hybrid NVRs can improve reconstruction quality by using content-adaptive latent
embeddings, their latent spatial sizes and decoder upsampling schedules are tied to the target frame resolution. For high-resolution
videos, this dependency may require large and non-uniform upsampling factors and can affect the parameter allocation between the latent
embeddings and the decoder. In this paper, we propose a resolution-flexible decoder framework for hybrid NVRs. The decoder is constructed from uniform \(2\times\) upsampling stages, whose target feature sizes are obtained by tracing the spatial resolution backward from the final output resolution. After each upsampling stage, the feature map is aligned with the target size by minimal padding or cropping when necessary. To support this progressive decoding process, we further use intermediate reconstruction supervision and a reconstruction-difficulty-aware frame sampling strategy based on recent frame-wise losses. The framework preserves the basic representation format of hybrid NVRs and can therefore be applied to different backbones. Experiments on the UVG
dataset show that the proposed approach improves reconstruction quality over the corresponding NVR baselines.
Read the full abstract ▾
Computer Vision
Modeling Clinical Workflow for SYNTAX Scoring from Coronary Angiography Videos
Suzhong Fu, Jingqi Dong, Xuan Ding et al. (+4 more)
The SYNTAX score is a clinically established tool for assessing anatomical lesion complexity in coronary artery disease and guiding subsequent treatment. However, automated SYNTAX scoring is commonly formulated as a direct regression problem from coronary angiography
videos to patient-level scores. In this work, we reformulate SYNTAX scoring as a vessel segment identity-preserving anatomical
reasoning problem and propose a hierarchical modeling framework that explicitly aligns learning with the clinical workflow. Our approach maintains vessel segment identity across frames and views, estimates stenosis severity at the segment level, and aggregates evidence hierarchically according to coronary anatomy. Simultaneously, to address the scarcity of domain-specific data, we integrate and complete multiple public coronary angiography
datasets, constructing a large-scale resource featuring completed vessel
segmentation and derived structural annotations. Experiments demonstrate that vessel segment-level stenosis
embedding enhances explanatory power and reduces prediction variability compared to baseline models, with the R^2 score improving by 0.201 and dev STD decreasing by 18.4%. These results highlight the necessity of structure-aligned modeling for reliable and stable automated SYNTAX scoring from multi-view coronary angiography
videos. The GitHub link is https://github.com/VersaceSu7/SYNTAX_score_777.
Read the full abstract ▾
All authors Suzhong Fu, Jingqi Dong, Xuan Ding, Rui Sun, Yiming Yang, Shuguang Cui, Zhen Li
Computer Vision
SewFusion: Tailored Generation of Topology and Panel-Level Geometry for Sewing Patterns
Jiaxin Lin, Xiao Pan, Hangjie Yuan et al. (+3 more)
Generating sewing patterns from images and text requires modeling a heterogeneous representation composed of discrete
topology and continuous geometry. Existing methods mainly follow two paradigms:
diffusion-based methods enable holistic geometry generation by converting the entire pattern into a continuous representation, but weaken discrete
topology modeling; in contrast, autoregressive methods preserve discrete
topology through next-
token prediction, but tie continuous geometry regression to
token-level hidden states with limited panel-level context. To bridge this gap, we propose SewFusion, a unified autoregressive framework that adopts tailored generation mechanisms for discrete
topology and panel-level continuous geometry, using next-
token prediction for the former and flow matching for the latter. To support panel-level continuous geometry generation, we introduce a Panel Geometry VAE that learns a fixed-size latent space for variable-length panel geometry, together with Panel Geometry Flow for latent generation. We further propose Panel-Forcing to reduce the training–inference mismatch in
topology context and improve robustness to
topology prediction errors. Extensive experiments on SewFactory and GCD-MM demonstrate that SewFusion consistently outperforms previous state-of-the-art methods across various settings, achieving +6.36% Panel Accuracy, +11.30% Stitch Accuracy, and -1.90 Vertex L2 error in the image-text-based generation setting.
Read the full abstract ▾
All authors Jiaxin Lin, Xiao Pan, Hangjie Yuan, Luyan Liang, Wan Li, Daquan Feng
Computer Vision
Towards robust multimodal 3D object detection via visual foundation models
Ziying Song, Lin Liu, Hongyu Pan et al. (+4 more)
Multimodal 3D object detection is fundamental to robust perception in autonomous driving because it integrates complementary information from
LiDAR and camera sensors. However, existing methods often fail to maintain robustness under out-of-distribution (OOD) corruptions caused by sensor noise, adverse weather, and environmental changes. To address this problem, we propose RoboDistill, a robust and generalizable
multimodal 3D object detection framework that leverages visual foundation models (VFMs), such as the Segment Anything Model (SAM). First, we introduce SAM-AD, a domain-specific pretraining strategy that fine-tunes SAM on autonomous-driving imagery to extract feature representations with rich semantic information. Second, we design the AD Feature Pyramid Network (AD-FPN) to refine and upsample SAM features at multiple scales for seamless fusion with
LiDAR features. Third, we develop the Depth-Guided Wavelet
Attention (DGWA) module, which suppresses high-frequency sensor noise while preserving critical contextual information. Finally, we introduce KD Fusion, in which the pretrained SAM-AD serves as a teacher that distills high-quality visual knowledge into a lightweight point-cloud network, thereby improving robustness under noisy conditions. Extensive experiments across 27 challenging OOD corruption settings show that RoboDistill generally delivers stronger or competitive detection performance and robustness relative to representative state-of-the-art methods. This work bridges the gap between VFMs and 3D object detection and advances robust
multimodal perception for real-world autonomous-driving applications.
Read the full abstract ▾
All authors Ziying Song, Lin Liu, Hongyu Pan, Shaoqing Xu, Lei Yang, Mingzhe Guo, Caiyan Jia
Computer Vision
PosEviLoc: Position-Conditioned Spatial Evidence for Language-Based 3D Localization
Tianyi Shang, Yike Shi and Zhenyu Li
Language-based 3D localization retrieves the point-cloud submap containing a target position from descriptions of nearby objects and their spatial relations. Existing methods typically compress queries and submaps into global descriptors, potentially obscuring object-level semantics and cross-description spatial coherence. We propose Position-Conditioned Evidence Localization (PosEviLoc), a query-position-aware framework for coarse text-to-point-cloud localization. Instead of relying on global matching, PosEviLoc evaluates each candidate submap using explicit semantic and spatial evidence. It models direction as a relation jointly determined by an object position and a hypothetical query position. The resulting Query-Position Spatial Evidence Field (QSEF) measures the fraction of query descriptions supported at each hypothetical position, explicitly capturing their agreement without using the ground-truth query pose to construct the evidence field. A Multi-Level Evidence Readout (MER) summarizes this evidence in a compact representation, which a lightweight MLP converts into a
retrieval score. Across five
benchmarks, PosEviLoc outperforms MNCL by an average of 17 percentage points in Recall@1. When used as a plug-and-play reranker, it improves MNCL by an average of 16 percentage points. Moreover, PosEviLoc introduces substantially fewer parameters and achieves faster inference speed than existing methods.
Read the full abstract ▾
Computer Vision
GeoBalance: Geometry-Aware Monitoring and Reconstruction with Asymmetric Optimization for Balanced Multimodal Learning
Zechang Xiong, Da Li, Rong Yin et al. (+7 more)
Multimodal classifiers can converge to modality-dominant solutions in which one modality dominates the joint prediction, suppressing the learning of others. Existing balancing methods mainly adjust losses, gradients, or modality contributions, largely treating modality imbalance as an optimization problem while implicitly treating the weak modality as under-optimized but representationally intact. In this work, we find that this assumption does not always hold, as persistent modality dominance can induce a representation-level collapse of the weak modality, which we term
manifold modality collapse (MMC). MMC manifests as a coupled
geometric degradation in which weak-modality representations collapse onto fewer directions within each class and become less separable across classes. Motivated by this observation, we propose GeoBalance, a geometry-aware framework that monitors these two
geometric properties and reconstructs the weak modality representation only when it exhibits signs of MMC. Once triggered, GeoBalance uses a fixed Simplex-ETF class scaffold and spectral regularization to restore class separation while preventing collapse onto a few feature directions. To preserve reconstruction during joint training, asymmetric gradient projection removes the joint-gradient component conflicting with reconstruction, leaving non-conflicting optimization unchanged. Extensive experiments across six
multimodal benchmarks demonstrate great improvements over competitive balancing methods, validating its effectiveness.
Read the full abstract ▾
All authors Zechang Xiong, Da Li, Rong Yin, Kexin Tang, Biao Yang, Pengyuan Li, Wenkang Kong, Yulan Hu, Shengyu Zhu, Hao Peng
Computer Vision
GARO: Geometry-Aware Redundancy Optimization for Real-Time and High-Fidelity Dynamic Gaussian Splatting
Huiwen Xue, Kaixing Zhao, Zuheng Ming et al. (+1 more)
Novel view synthesis is a key task for dynamic scene reconstruction, where high rendering speed is essential for applications such as virtual reality. Existing deformable Gaussian Splatting methods achieve high-fidelity dynamic scene modeling, but still face limitations in
memory usage and rendering efficiency due to the large number of redundant Gaussians. To address these challenges, we propose Geometry-Aware Redundancy Optimization (GARO), a unified redundancy measurement framework in the adaptive density control stage of the traditional dynamic scene reconstruction pipeline. This framework first selects low-gradient candidates using an optimization activity assessment strategy, and then evaluates
geometric complexity through low
curvature analysis to further filter and prune redundant points, resulting in a compact and expressive Gaussian representation. Extensive experiments on synthetic and real-world
datasets demonstrate that GARO achieves robust trade-offs between quality and speed, with PSNR remaining stable and rendering speed improved by 2x, validating the efficiency and effectiveness of GARO.
Read the full abstract ▾
All authors Huiwen Xue, Kaixing Zhao, Zuheng Ming, Tingcheng Li
Computer Vision
Detecting Phone-Induced Pedestrian Distraction via a Multimodal Fusion Transformer
Yuanzhe Li, Hounian Liu, Xiaotong Chang et al. (+1 more)
The increasing reliance on mobile phones has made phone-induced pedestrian distraction increasingly prevalent. Activities such as texting, watching
videos, and making phone calls have become significant contributors to traffic accidents. Reliable detection of pedestrian distraction is essential for autonomous vehicles, as it improves situational awareness and enables timely risk assessment, thereby supporting safe motion planning and vehicle control. We propose a
multimodal fusion
Transformer (MFT) for detecting phone-induced pedestrian distraction. MFT jointly extracts skeletal dynamics from body pose keypoints and visual appearance features from pedestrian images, effectively leveraging the complementary information provided by the two modalities. A cross-modal
attention module is proposed to capture inter-modal dependencies through multi-head cross-
attention, facilitating effective fusion of complementary information across the two modalities. Then, a temporal
attention fusion module, implemented with a
Transformer encoder, is employed to capture temporal dependencies. MFT is trained and evaluated on a manually annotated
dataset comprising 287 pedestrian instances with 20,741 images. Extensive experiments demonstrate that MFT attains an overall accuracy of 95%, exceeding the performance of six baseline approaches by 6%.
Read the full abstract ▾
All authors Yuanzhe Li, Hounian Liu, Xiaotong Chang, Yidi Huang
Computer Vision
Pay More Attention To Text In High-Resolution MLLMs
Zhongkuan Mao, Wenzhuo Zhao, Xianjie Liu et al. (+7 more)
Failures of high-resolution MLLMs are commonly attributed to a visual problem, motivating zooming, cropping, and related visual interventions to recover fine-grained evidence or suppress interference. Yet recent studies suggest that relevant visual evidence is already encoded in intermediate representations, indicating that visual-side improvements alone insufficient. This raises a natural question: does the remaining bottleneck lie in the text that guides visual search? We identify a previously overlooked linguistic bottleneck: questions formulated for answering do not necessarily specify the visual evidence required for localization. To address this mismatch, we introduce EviSpec, a training-free compiler that derives complementary evidence specifications while preserving the original question for final
reasoning. We further validate it through matched-control experiments that isolate the roles of evidence specification and localization. With the search budget fixed, structured evidence specifications yield an 8.6% relative gain over generic requests. With evidence geometry matched, the evidence localized by EviSpec yields a 14.8% relative gain over random evidence. Together, these controls isolate the benefit of specifying what evidence to seek rather than merely expanding visual access. Across all five MLLMs, EviSpec consistently improves upon the corresponding baseline on each of the three
benchmarks, yielding average relative gains of 10.4%, 8.8%, and 12.4% on V\textsuperscript{*}Bench, HR-Bench-4K, and HR-Bench-8K, respectively. Beyond high-resolution
reasoning, EviSpec also achieves state-of-the-art performance on VQA and
hallucination-focused
benchmarks.
Read the full abstract ▾
All authors Zhongkuan Mao, Wenzhuo Zhao, Xianjie Liu, Yidong Wang, Zhao Gao, Ronghao Xian, Yao Jiang, Yi Zhang, Liangjian Wen, Keren Fu
Computer Vision
CE$^4$L: Continual Ego, Exo, and Ego-Exo Learning
Hongwei Yan, Kanglei Zhou, Yuchen Liu et al. (+3 more)
Perception for
embodied agents is
video-based, often multi-view (ego, exo, or both), and inherently continual, with simultaneous task and viewpoint shifts. Yet continual learning (CL) remains dominated by exo-only recognition tasks, obscuring behavior under these real-world coupled shifts. We introduce Continual Ego, E}xo, and Ego-Exo Learning (CE$^4$L), a unified multi-view CL
benchmark spanning four representative tasks: cross-view referenced skill assessment, temporal action
segmentation, cross-view association, and action anticipation & planning. CE$^4$L highlights challenges largely absent in prior CL
benchmarks, including cross-view correspondence, view-dependent asynchrony, and heterogeneous semantic objectives. To this end, we propose
Video Incremental Subspace-routed Task Adapters (VISTA), a parameter-efficient baseline method that stores task-specific updates in lightweight adapters and performs training-free routing via residual distance to task-specific whitened subspaces estimated from second-order statistics. Extensive experiments demonstrate the significantly varied efficacy of representative CL methods across CE$^4$L settings, while VISTA is consistently competitive and achieves state-of-the-art overall performance. Our source code for
benchmarks and methods is available at https://github.com/AnAppleCore/CE4L .
Read the full abstract ▾
All authors Hongwei Yan, Kanglei Zhou, Yuchen Liu, Qingyu Shi, Yi Zhong, Liyuan Wang
Computer Vision · Cross-listed from cs.RO
Cognitive Action Reasoning for Proactive Robots from Human-Centered Multimodal Observations
Zhihao Gu, Kechao Zhu, Yuanfeng Wu et al. (+7 more)
Robots operating in human-centered environments are typically designed to execute explicit instructions, and most
robot-learning
datasets likewise pair observations with task instructions or low-level actions. Although recent work has begun to explore proactive
embodied assistance, existing resources target different settings and action levels, leaving real-world human-centered
multimodal decision-making underexplored. We formulate this problem as Proactive
Robot Action
Reasoning (ProRobo), an upstream cognitive decision problem in which a
robot must determine which action to take based on
multimodal human and environmental cues without explicit action instructions. To support ProRobo, we introduce ProAction, a real-world
multimodal dataset containing 10K samples of visual observations, audio signals, and text inputs across 12 daily-life scenarios in five common scenes. To construct cognitively grounded high-level action supervision, we develop a two-stage human-in-the-loop pipeline that combines appraisal-guided candidate generation with Affective Theory-of-Mind-guided human refinement, explicitly incorporating contextual judgment about human states, urgency, feasibility, and potential risk into action annotation. Based on this supervision, we
benchmark representative
Multimodal Large
Language Models (MLLMs) and introduce MMC2Act, a reference model that implicitly learns the mapping from
multimodal observations to cognitively grounded high-level actions. Experiments across modality settings, subject-disjoint generalization, cross-
dataset transfer, and human evaluation show that general-purpose MLLMs struggle with proactively
reasoning high-level actions from
multimodal cues, whereas training on ProAction substantially improves performance.
Read the full abstract ▾
All authors Zhihao Gu, Kechao Zhu, Yuanfeng Wu, Mohan Liu, Ankit Kumar Shaw, ChenDong Hong, Xuanyu Chen, Dengchen Mei, Xu Tianyi, Lin Wang
Computer Vision
Algebraic Consistency Alone Does Not Certify Temporal Structure in Latent Action Models
Di Wen, Ruodi Zhang, Kailun Yang et al. (+1 more)
Latent action models infer a code for the transition between two frames of action-free
video. Recent methods regularise this code to compose additively and reverse antisymmetrically, and report order-of-magnitude reductions in the resulting errors as a label-free certificate that the code has captured temporal structure. We show that this conclusion does not follow. Reconstruction drives the decoded transition toward a difference of state features, for which both identities hold for any pairing, a solution the metric cannot distinguish from one encoding nuisance state or a coordinate convention. Across five source domains, a trained but unconstrained counterpart already achieves 83-97% of the reduction relative to an untrained anchor. The residual fold is governed as much by the decoder family as by what is learned. A constrained model retrained after its temporal pairing is destroyed still reaches, in each domain, a lower error than the unconstrained model on real data. Downstream, preserving the temporal pairing yields no consistent advantage on LIBERO-GOAL or LIBERO-SPATIAL, and across the tested arms the code's mean linear action decodability falls as the algebraic error improves. We also test the most direct repair, a violation-contrastive objective that requires the
algebra to fail on destroyed pairings: in the tested configurations it yields only a marginal separation within the reconstruction budget, on training and test triples alike. We recommend a validation protocol that these methods currently lack: a baseline-corrected metric, retraining on destroyed pairings, and a seed-budget analysis.
Read the full abstract ▾
All authors Di Wen, Ruodi Zhang, Kailun Yang, Kunyu Peng
Computer Vision
Rule-Constrained Assignment for Cue-Ball Identification in Broadcast Snooker
Yuxin Cao, Wei Song, Yuezhong Wu et al. (+1 more)
Accurate cue-ball identification is essential for metric analysis of broadcast snooker. Existing systems evaluate each candidate independently against a fixed white prototype and reject candidates above an appearance threshold. Under broadcast conditions, illumination changes can make colored balls appear white, while intrusions from players and equipment can obscure the cue ball or introduce competing candidates. We formulate cue-ball identification as a rule-constrained assignment problem that jointly assigns detected candidates to the bounded snooker inventory: one cue ball, up to 15 reds, and six colors with known spots. The cue ball is selected by the incremental cost of assigning each candidate to the white slot, and the conventional appearance test follows as the one-slot case. On 419 hand-annotated shots, our method improves identity accuracy from 88.1% to 95.5%, and from 80.5% to 95.2% on held-out venues. Within CueLift, our metric state-recovery system, the assignment expands coverage from 36.2% to 55.4% over 6,241 scorable shots. Assigning an estimate to every shot in a separate evaluation on 2,529 shots preserves this advantage.
Read the full abstract ▾
All authors Yuxin Cao, Wei Song, Yuezhong Wu, Jin Song Dong
Computer Vision
PhysReflect: Geometry and Perception Guided Diffusion for Physically-Plausible Mirror Reflections
Shuheng Ge, Hongwei Ren, Li Zhang et al. (+1 more)
Diffusion models generate high-quality images, yet often violate the physical laws governing mirror reflections. Reflections often suffer from
geometric aberrations, including positional offsets, directional misalignment, proportional imbalance, and structural distortion. These failures remain evident even in contemporary state-of-the-art generative systems. Existing methods itigate this problem through synthetic data scaling or auxiliary depth conditioning, yet their merely reliance on latent-space noise reconstruction losses as implicit supervision prevents direct enforcement of reflection-specific
geometric and perceptual constraints. To bridge this gap, we present PhysReflect, a geometry and perception guided
diffusion framework that decodes the predicted clean latent into pixel space at each training step and applies annealed supervision through two complementary differentiable objectives. The
Geometric Loss enforces mirror-induced spatial consistency through sparse epipolar correspondence and dense boundary projection
alignment, where a SAM2-based TwinTrack mechanism provides stable in-mirror localization for boundary-aware supervision. The Perceptual Loss preserves reflected appearance by combining Semantic Consistency Loss, which maintains reflected identity and appearance via DINOv2 features, and Lighting Consistency Loss, which regularizes depth, surface-normal, and illumination coherence under
monocular geometry priors. Experiments on synthetic and real-world
benchmarks show that PhysReflect outperforms prior mirror-reflection methods in
geometric, perceptual, and physical-plausibility metrics, as well as qualitative visual results.
Read the full abstract ▾
All authors Shuheng Ge, Hongwei Ren, Li Zhang, Xiangqian Wu
Computer Vision
GAPS: Generative Active Pseudo-view Selection for Sparse-View 3D Gaussian Splatting
Hongfei Zhu, Haochen Deng, Sitao Zhang et al. (+1 more)
Novel view synthesis from sparse observations is severely under-constrained. Although 3D Gaussian Splatting (3DGS) enables real-time rendering, it produces floaters, broken geometry, and washed-out backgrounds when trained with few views. We propose an alternating optimization framework that uses a pre-trained image
diffusion model to generate geometrically consistent pseudo-views for additional 3DGS supervision. Generation is constrained by depth-conditioned ControlNet, IP-Adapter style transfer, LoRA scene adaptation, and img2img structural anchoring. We introduce Generative Active Pseudo-view Selection (GAPS) to balance reconstruction informativeness and generative reliability when choosing target views. Its annealing schedule shifts from conservative interpolation early in training to exploratory extrapolation later, gradually covering unobserved regions. A dual-criterion admission gate and uncertainty-weighted losses reject unreliable generations, while density-adaptive DropGaussian reduces overfitting in complex scenes. On LLFF with 3/6/9 views, our method improves average PSNR over vanilla 3DGS by 0.40/0.89/0.70 dB. On Mip-NeRF 360 with 12/24 views, the gains are 1.18/0.80 dB. SSIM improves and LPIPS decreases in every setting. Ablations show that active selection and density-adaptive regularization are both necessary; only the full method reduces LPIPS below the no-pseudo-view baseline on unbounded 360-degree scenes.
Read the full abstract ▾
All authors Hongfei Zhu, Haochen Deng, Sitao Zhang, Ling Zhou
Computer Vision
HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detection
Junwen Chen and Keiji Yanai
Human-object interaction (HOI) detection requires grounding an interacting human-object pair and recognizing the verb that links them, often under severe long-tail supervision. Recent methods improve accuracy with stronger detectors and
vision-
language priors, but many still stack heavy
transformer encoders, intricate denoising schedules, or post-hoc semantic calibration on top of the detector. We present HOIBlender, an efficient HOI detector named after its core design principle: blending detector-grounded visual
tokens, spatial subject-object
reasoning, and BLIP-2 semantic priors inside one lightweight decoding pipeline. HOIBlender builds on an RF-DETR/LW-DETR-style foundation with a DINOv2 backbone and selects top-$K$ image-conditioned
tokens directly from the multi-scale projector as subject and object candidates, removing the dedicated encoder stage retained by prior HOI methods. A dual-stage decoder first stabilizes human-object geometry and then performs verb and HOI classification through progressive BLIP-2 prior fusion, with classifier weights initialized from BLIP-2 text
embeddings for long-tail categories. Grouped-query training further enriches optimization without increasing inference cost. Across three model scales (Nano, Small, 2XL), HOIBlender consistently outperforms SOV-STG-VLA and Hybrid-SOV-VLA on HICO-DET, reaching $44.49$ Default Full mAP in only $9$ training epochs while maintaining competitive latency and parameter budgets. These results show that lightweight detection, structured spatial-semantic decoding, and deeply integrated
vision-
language priors can be blended into a single efficient HOI pipeline.
Read the full abstract ▾