Computer Vision
Cross-Lingual Learning within Arabic Script for Low-Resource HTR
Sana Al-azzawi, Elisa Barney and Marcus Liwicki
Handwritten Text Recognition (HTR) with limited labeled data remains a challenging problem, particularly for Arabic-script
languages. Although modern sequence-based recognizers perform well in high-resource settings, their accuracy degrades sharply as training data becomes scarce. Arabic-script
languages share a common writing system with substantial character overlap, motivating cross-lingual learning as a strategy to mitigate data scarcity. We conduct a controlled line-level study of cross-lingual joint training for Arabic-script HTR under low-resource regimes (number of samples K = 100, 500, 1000 labeled lines) on Arabic (KHATT), Urdu (NUST-UHWR) and Persian (PHTD). CRNN and
Vision Transformer-based HTR-VT models are trained on the union of multiple related Arabic-script
datasets to mitigate the data scarcity and are evaluated on individual target
languages. Both architectures benefit from cross-
language training under low-resource conditions. CRNN remains more effective under extremely limited target-
language data, whereas the benefits of cross-
language training for HTR-VT become less consistent as larger amounts of target-
language data become available. On Persian (PHTD), joint training achieves a Character Error Rate (CER) of 9.99 , surpassing previously reported results despite not using the full available training data. On an additional Urdu
dataset (UNHD), joint training reduces CER from 17.20 to 14.45.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.SY
Observability Conditions and Filter Design for Visual Pose Estimation via Dual Quaternions
Nicholas B. Andrews and Kristi A. Morgansen
This paper presents a dual quaternion framework for 6-DOF visual target tracking that addresses key limitations of perspective-n-point (P$n$P) solvers: sensitivity to noise and outliers, and inability to propagate estimates through measurement dropouts. A nonlinear observability analysis is performed using a Lie algebraic approach, deriving sufficient conditions for local observability under two sensing modalities: relative position vector and unit vector measurements. For the unit vector case, the classical collinear feature point degeneracy of the perspective-three-point problem is recovered through rank analysis of the observability codistribution
matrix, providing a control-theoretic interpretation of a previously
geometric result. A dual quaternion Lie group unscented Kalman filter is then developed, directly modeling relative dynamics without assumptions about cooperative measurements or slowly-varying motion. Simulations demonstrate improved pose estimation accuracy and robustness to occlusions compared to an off-the-shelf P$n$P solver. Results are broadly applicable to visual-inertial
navigation, simultaneous localization and mapping, and P$n$P solver development.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
How Can One Choose the Best CAM-Based Explainability Method for a CNN Model?
Daniel da Silva Costa, Pedro Nuno de Souza Moura and Adriana C. F. Alvim
In recent years, several advances have been observed in Deep Learning with surprising results. Models in this area have been increasingly used in numerous applications, including those sensitive to human life, which require clear explanations and justifications. Various explainability methods have been proposed, but not many metrics to evaluate these methods. The most commonly used metric is the Intersection over Union (IoU). However, due to the characteristics of the results of the explainability methods, called saliency maps, which do not have a known shape, we hypothesise that there must be a better metric that allows one to find an explainability method that produces results that best resemble the human perception. We propose using different metrics to assess the similarity between human perception and the explanation saliency maps to find a better metric. An investigation was conducted employing a subset of the Chihuahuas images from ImageNet
dataset. Several CAM-based explainability methods were used to generate saliency maps for each chihuahua image.
Alignment was measured by applying distance metrics between the bounding box of human annotations and the saliency maps produced by each explainability method. Rankings of the best saliency maps were created using the results of the distance metrics and compared to the ranking obtained using people's choice, collected through crowdsourcing, of the best explanation saliency maps for each selected image. Comparison between rankings was performed using the Rank-Biased Overlap (RBO) metric. The results indicate the feasibility of our method to find the explainability method that best resembles human perception. In our experiments, the two metrics that best resemble human perception corresponded to Manhattan and Correlation. Besides, the best explainability methods regarding human perception were LayerCAM, Score-CAM, and IS-CAM.
Read the full abstract ▾
Computer Vision
From Concept to Capability: Evaluating 3D Gaussian Splatting for Synthetic Scene Editing in Autonomous Driving
Ali Nouri, Yifei Zhang, Yifan Zhang et al. (+5 more)
The perception of an Autonomous Driving System (ADS) critically depends on relevant, comprehensive, and diverse
datasets to ensure its safety while operating in the environment. Field data collection lacks completeness with respect to the list of rare but still possible safety-related scenarios needed for the development, verification, and validation of the ADS. 3D Gaussian Splatting (3DGS) has shown promising capabilities for the reconstruction and editing of scenes based on data collected by cameras and
LiDAR sensors. However, the industrial fidelity evaluation of reconstructions is underexplored, which is crucial when employing such methods in safety-related systems, especially for ADS. This becomes more challenging as ADS operates in a dynamic, uncontrolled environment with limited viewpoints and often partially occluded objects. This paper addresses this gap by proposing and implementing a framework (Fig. 1) to systematically analyze the capabilities and limitations of 3DGS for use in the reconstruction of safety-related scenes. It focuses on the quality of reconstruction for vehicles and pedestrians, which are the two most critical object classes for ADS. Our findings provide industry insights into the fidelity degradation of reconstructions from multiple novel viewpoints, both lateral and longitudinal, enabling the integration of these methods into real-world industrial AD software development and testing pipelines.
Read the full abstract ▾
All authors Ali Nouri, Yifei Zhang, Yifan Zhang, Tayssir Bouraffa, Zhennan Fei, Zijian Han, Håkan Sivencrona, Anders Heyden
Computer Vision
ProtoFair: Fair Self-Supervised Contrastive Learning via Pseudo-Counterfactual Pairs
Marah Halawa and Olaf Hellwich
Self-supervised learning methods learn high-quality visual representations, yet recent studies show that these representations often capture demographic biases present in the training data. Existing
fairness-aware methods address this by redesigning the self-supervised objective itself, limiting portability across the rapidly evolving landscape of self-supervised learning (SSL) frameworks. We propose ProtoFair, a
fairness-aware contrastive loss designed to work alongside existing SSL objectives without modifying them. ProtoFair leverages unsupervised prototype clustering to identify pseudo-counterfactual pairs: samples sharing the same cluster assignment but belonging to different sensitive groups. By pulling these content-matched, cross-group samples together in the
embedding space, ProtoFair encourages the encoder to learn representations that are invariant to the sensitive attribute. The method requires only sensitive attribute annotations, no target labels, and integrates seamlessly with both SimCLR and SupCon. Experiments on CelebA and UTKFace demonstrate consistent
fairness improvements while maintaining competitive accuracy.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization
Yavuz Yarici and Ghassan AlRegib
Deploying
multimodal models in real-world scenarios requires generalization to new environments where recording conditions differ from training, a challenge known as
multimodal domain generalization (MMDG). Standard architectures employ separate encoders for each modality and a fusion module, training the system end-to-end by optimizing on the fused features. In this paper, we identify that such joint optimization causes encoders to exploit cross-modal co-occurrences, statistical relationships between modalities that arise from source-specific recording conditions, rather than learning domain-invariant features. We term this failure mode Fusion Overfitting. To address this, we propose Modality-
Entropy Regularization for Domain Generalization (MER-DG), which maximizes the
entropy of each encoder's feature distribution to preserve feature diversity. MER-DG is architecture-agnostic and integrates into existing
multimodal frameworks as an additive loss term. Extensive experiments on EPIC-Kitchens and HAC
benchmarks demonstrate average improvements of approximately 5% over standard fusion and approximately 2% over state-of-the-art methods.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Sonar-GPS Fusion for Seabed Mapping in Turbid Shallow Waters with an Autonomous Surface Vehicle
Yisheng Zhang, Michael Xu, Alan Williams et al. (+3 more)
Accurate seabed mapping is essential for habitat monitoring and infrastructure inspection. In turbid, shallow coastal waters, such as shellfish aquaculture farms, the effectiveness of traditional optical methods is limited. Autonomous surface vehicles (ASVs) equipped with forward-looking sonar (FLS) offer a promising alternative. However, existing sonar-based systems face challenges in achieving fine resolution mapping over long trajectories due to low-resolution positioning measurements and accumulated drift over long trajectories. In this paper, we present a drift-resilient seabed mapping framework that integrates local FLS frame
alignment using the
Fourier-Mellin transform (FMT) with global trajectory optimization based on an extended Kalman filter (EKF) that fuses global positioning system (GPS), inertial measurement unit (IMU), and compass data. A variance-based image blending strategy is used to further reduce visual artifacts in overlapping regions. Field trials on a structured oyster farm site show that our framework helps reduce drift in RMSE by 9.5% relative to the FMT-only baseline. This framework also enables sub-meter reconstruction accuracy and preservation of high-resolution textures needed for oyster inventory estimation within the mapped areas.
Read the full abstract ▾
All authors Yisheng Zhang, Michael Xu, Alan Williams, Matthew Gray, Nare Karapetyan, Miao Yu
Computer Vision · Cross-listed from cs.AR
ViM-Q: Scalable Algorithm-Hardware Co-Design for Vision Mamba Model Inference on FPGA
Shengzhe Lyu, Yuhan She, Patrick S. Y. Hung et al. (+2 more)
Vision Mamba (ViM) models offer a compelling efficiency advantage over
Transformers by leveraging the linear complexity of State Space Models (SSMs), yet efficiently deploying them on FPGAs remains challenging. Linear layers struggle with dynamic activation outliers that render static quantization ineffective, while uniform quantization fails to capture the weight distribution at low bit-widths. Furthermore, while associative scan accelerates SSMs on GPUs, its
memory access patterns are misaligned with the streaming dataflow required by FPGAs. To address these challenges, we present ViM-Q, a scalable
algorithm-hardware co-design for end-to-end ViM inference on the edge. We introduce a hardware-aware quantization scheme combining dynamic per-
token activation quantization and per-channel smoothing to mitigate outliers, alongside a custom 4-bit per-block Additive Power-of-Two (APoT) weight quantization. The models are deployed on a runtime-parameterizable FPGA accelerator featuring a linear engine employing a Lookup-Table (LUT) unit to replace multiplications with shift-add operations, and a fine-grained pipelined SSM engine that parallelizes the state dimension while preserving sequential recurrence. Crucially, the hardware supports runtime configuration, adapting to diverse dimensions and input resolutions across the ViM family. Implemented on an AMD ZCU102 FPGA, ViM-Q achieves an average 4.96x speedup and 59.8x energy efficiency gain over a quantized NVIDIA RTX 3090 GPU baseline for low-batch inference on ViM-tiny. This co-design shows a viable path for deploying ViM models on resource-constrained edge devices.
Read the full abstract ▾
All authors Shengzhe Lyu, Yuhan She, Patrick S. Y. Hung, Ray C. C. Cheung, Weitao Xu
Computer Vision
Exploring Data-Free LoRA Transferability for Video Diffusion Models
Yuchen Wang, Wenliang Zhong, Lichen Bai et al. (+6 more)
Video diffusion models leveraging step
distillation or causal
distillation have achieved remarkable performance. However, adapting existing LoRAs to these variants remains a critical challenge due to weight space mismatches. We observe that direct application leads to style degradation and structural collapse, yet the underlying mechanisms remain poorly understood. To fill this gap, we delve into the weight space and identify that the incompatibility stems from spectral interference within shared functional clusters defined over singular subspaces. Specifically, our analysis reveals that while both paradigms respect spectral
rigidity, they establish conflicting routing pathways that clash through constructive overload or destructive cancellation. To address this issue, we propose Cluster-Aware Spectral Arbitration (CASA), a data-free framework that dynamically arbitrates between safeguarding the target's
manifold and restoring LoRA
alignment based on spectral density. Extensive experiments demonstrate that CASA effectively mitigates artifacts and revives LoRA functionality. Our code is available at https://github.com/Noahwangyuchen/CASA
Read the full abstract ▾
All authors Yuchen Wang, Wenliang Zhong, Lichen Bai, Zikai Zhou, Shitong Shao, Bojun Cheng, Shuo Chen, Shuo Yang, Zeke Xie
Computer Vision
CADFS: A Big CAD Program Dataset and Framework for Computer-Aided Design with Large Language Models
Vladislav Pyatov, Gleb Bobrovskikh, Saveliy Galochkin et al. (+6 more)
We introduce CADFS, a data-centric framework that enables large
vision-
language models to generate complex CAD design histories. Existing generative CAD systems are restricted to sketch-extrude operations due to simplified representations and limited
datasets. We address this by introducing a FeatureScript-based representation and constructing a
dataset of 450k real-world CAD models spanning 15 modeling operations. We obtain the
dataset via a new pipeline that reconstructs clean, executable FeatureScript programs and provides
multimodal annotations. Fine-tuning a VLM on this representation yields state-of-the-art results in text-conditioned CAD generation and image-based reconstruction, producing more accurate, diverse, and feature-rich designs than prior frameworks. Ablations show that each individual component of our framework, i.e., the FeatureScript representation, the extended operation set, and representation-aligned textual descriptions, significantly improves performance. Our framework substantially broadens the complexity and realism achievable in generative CAD. The CADFS framework and the new
dataset are available at https://voyleg.github.io/cadfs/.
Read the full abstract ▾
All authors Vladislav Pyatov, Gleb Bobrovskikh, Saveliy Galochkin, Nikita Boldyrev, Oleg Voynov, Alexander Filippov, Gonzalo Ferrer, Peter Wonka, Evgeny Burnaev
Computer Vision
SimPB++: Simultaneously Detecting 2D and 3D Objects from Multiple Cameras
Yingqi Tang, Zhaotie Meng, Erkang Cheng et al. (+1 more)
Simultaneous perception of 2D objects in perspective view and 3D objects in Bird's Eye View (BEV) is challenging for multi-camera autonomous driving. Existing two-stage pipelines use 2D results only as a one-time cue for 3D detection. We propose SimPB++, which simultaneously detects 2D objects in perspective and 3D objects in BEV from multiple cameras. It unifies both tasks into an end-to-end model with a hybrid decoder architecture, coupling multi-view 2D and 3D decoders interactively. Two novel modules enable deep interaction: Dynamic Query Allocation adaptively assigns 2D queries to 3D candidates, and Adaptive Query Aggregation refines 3D representations using multi-view 2D features, forming a cyclic 3D-2D-3D refinement. For multi-view 2D detection, we use Query-group
Attention for intra-group communication. We also design a Crop-and-Scale strategy for long-range perception and a Propagating Denoising strategy with an auxiliary RoI detector. SimPB++ supports mixed supervision with 2D-only and fully annotated data, reducing reliance on expensive 3D labels. Experiments show state-of-the-art performance on nuScenes for both tasks and strong long-range detection (up to 150m) on Argoverse2.
Read the full abstract ▾
All authors Yingqi Tang, Zhaotie Meng, Erkang Cheng, Haibin Ling
Computer Vision
EAPFusion: Intrinsic Evolving Auxiliary Prior Guidance for Infrared and Visible Image Fusion
Zhenyu Sun, Luobin Zhang, Axi Niu et al. (+2 more)
Infrared-visible image fusion aims to create an information-rich fused image by integrating the complementary thermal saliency from infrared sensing and fine textures from visible imaging. Such accurate fusion is essential for real-world perception applications in complex scenes, including nighttime autonomous driving, search and rescue, and surveillance, and can further benefit downstream tasks such as semantic
segmentation. However, most existing fusion methods rely upon static trained weights that cannot adapt to scene-specific content at inference time, and often suffer from a granularity mismatch when coarse auxiliary semantics are injected, which makes it difficult to simultaneously highlight targets and preserve details. In this work, we propose EAPFusion to address these issues by using self-evolving intrinsic priors instead of relying on external auxiliary models. Concretely, EAPFusion maintains a compact set of intrinsic priors and progressively updates them across scales. These evolved priors are utilized to dynamically generate convolutional kernels, shifting the paradigm from fixed, pre-trained filters to instance-adaptive parameters via prior-conditioned dynamic convolution. Furthermore, we design a channel-level fusion module that shuffles and interleaves infrared and visible channels, applying local channel mixing to boost cross-modal complementarity. Experiments on different
datasets, including cross-
dataset evaluation and semantic
segmentation, show that the proposed method achieves state-of-the-art quantitative and qualitative fusion results, and consistently boosts downstream performance. Code is coming soon.
Read the full abstract ▾
All authors Zhenyu Sun, Luobin Zhang, Axi Niu, Haishen Wang, Qingsen Yan
Computer Vision
SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?
Jongmin Shin, Ka Young Kim, Eunki Cho et al. (+2 more)
Purpose:
Vision-
language models (VLMs) have shown promising performance in
surgical visual question answering (VQA). However, existing
surgical VQA
datasets often contain linguistic shortcuts, where question phrasing implicitly constrains the answer space. It remains unclear whether reported performance reflects visual understanding or reliance on such linguistic shortcuts. Methods: We introduce SurgCheck, a diagnostic
benchmark for quantifying linguistic shortcut reliance in
surgical VQA. SurgCheck employs a paired-question design in which each
surgical frame is associated with an original question containing entity names and a less-biased counterpart that removes these names while preserving identical visual content and ground-truth answers. The resulting performance gap provides a diagnostic signal of shortcut reliance. To ensure that the less-biased question remains well-defined even without entity names, four grounding cues are incorporated: bounding box, arrow, spatial position, and periphrasis. We evaluate both general-purpose and
surgical-specific VLMs under zero-shot and fine-tuned settings on SurgCheck. To evaluate open-ended zero-shot responses, we introduce an LLM-as-a-judge evaluation protocol. Results: Using SurgCheck, we observe consistent performance degradation on less-biased questions across five VLMs, despite identical visual inputs. Text-only ablation reveals minimal performance drops for action and target prediction, indicating that action and target prediction is largely driven by linguistic shortcuts rather than visual
reasoning. Conclusion: SurgCheck provides a controlled diagnostic framework that exposes failure modes masked by linguistic bias in existing
surgical VQA
benchmarks. Our findings demonstrate that strong
benchmark performance does not necessarily imply faithful visual understanding, underscoring the need for bias-aware evaluation in
surgical VQA.
Read the full abstract ▾
All authors Jongmin Shin, Ka Young Kim, Eunki Cho, Seong Tae Kim, Namkee Oh
Computer Vision
Behavior-Grounded Lane Representation Learning for Multi-Task Traffic Digital Twins
Rei Tamaru, Pei Li and Bin Ran
Traffic digital twins are powerful tools for advanced traffic management, and most systems are built on static
geometric representations. However, these representations fail to capture the dynamic functional semantics required for behavior-aware
reasoning, such as how a lane operates under complex traffic conditions. To address this gap, we introduce GeoLaneRep, a behavior-grounded lane representation learning framework for traffic digital twins. GeoLaneRep jointly encodes static lane geometry, observed vehicle trajectories, and operational descriptors into a shared, cross-camera semantic
embedding. The encoder is trained with a joint objective combining contrastive cross-camera
alignment, auxiliary role supervision, and temporal anomaly detection. Across 16 roadside cameras and 132 lanes, the learned
embeddings achieve a $0.004$ lateral-rank error and an edge-role F1 of $1.000$ in zero-shot cross-camera matching, and an AUROC of $0.991$ for window-level anomaly detection. We further show that the same behavioral
embeddings can condition a
diffusion-based generator to synthesize lane geometries that satisfy targeted operational specifications, with $87.9\%$ overall specification accuracy across 38 lane groups. GeoLaneRep thus provides a semantic interface between roadside observations and downstream digital twin tasks, supporting cross-camera transfer, behavior-aware monitoring, and goal-directed lane synthesis. The framework is openly available at https://github.com/raynbowy23/GeoLaneRep.
Read the full abstract ▾
Computer Vision
Divide and Conquer: Decoupled Representation Alignment for Multimodal World Models
Junyuan Xiao, Dingkang Liang, Xin Zhou et al. (+9 more)
Emerging multi-modal world models attempt to jointly generate
videos across diverse modalities (e.g., RGB, depth, and mask), yet they fail to fully exploit the rich priors of existing foundation models. We propose $M^2$-REPA, the first representation
alignment method tailored for multi-modal
video generation. Our key insight is that foundation models trained on different modality spaces naturally capture distinct domain-specific priors, acting as complementary "experts." Specifically, we first decouple modality-specific features from the
diffusion model's intermediate representations, then align each with its corresponding expert foundation model. To this end, we design two synergistic objectives: a multi-modal representation
alignment loss that enforces feature-to-expert matching, and a modality-specific decoupling regularization that encourages complementarity across different modalities. This design enables joint optimization, fully exploiting priors from multiple foundation models. Extensive experiments demonstrate that our method significantly outperforms baselines in visual quality and long-term consistency.
Read the full abstract ▾
All authors Junyuan Xiao, Dingkang Liang, Xin Zhou, Yixuan Ye, Tongtong Su, Guangmo Yi, Bin Xia, Qiang Lyu, Shurui Shi, Jun Huang, Jianlou Si, Wenming Yang
Computer Vision
AFFormer: Adaptive Feature Fusion Transformer for V2X Cooperative Perception under Channel Impairments
Xi Zhou, Tao Huang, Qing-Long Han et al. (+2 more)
Accurate 3D object detection is essential for ensuring the safety of autonomous vehicles. Cooperative perception, which leverages vehicle-to-everything (V2X) communication to share perceptual data, enhances detection but is vulnerable to channel impairments, such as noise, fading, and interference. To strengthen the reliability of intelligent transportation systems, this work improves the robustness of V2X cooperative perception under communication conditions that reflect common channel impairments. This paper proposes an Adaptive Feature Fusion
Transformer (AFFormer), a
Transformer-based framework that mitigates the adverse effects of corrupted features by modeling temporal, inter-
agent, and spatial correlations. AFFormer introduces three key modules: Multi-
Agent and Temporal Aggregation for context-aware fusion across
agents and over time, Dual Spatial
Attention for efficient modeling of spatial dependencies, and Uncertainty-Guided Fusion for
entropy-driven refinement of fused features. A teacher-student knowledge
distillation strategy further enhances robustness by aligning fused features with reliable early-collaboration supervision. AFFormer is validated on the V2XSet and DAIR-V2X
datasets, where it consistently outperforms existing methods under both ideal and impaired communication conditions, demonstrating improved robustness to communication-induced feature degradation while maintaining a competitive efficiency-accuracy trade-off.
Read the full abstract ▾
All authors Xi Zhou, Tao Huang, Qing-Long Han, Rana Abbas, Mostafa Rahimi Azghadi
Computer Vision
Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts
Hongkun Pan, Yuwei Wu, Wanyi Hong et al. (+8 more)
Multimodal large
language models (MLLMs) have shown considerable potential in chart understanding and
reasoning tasks. However, they still struggle with high information density (HID) charts characterized by multiple subplots, legends, and dense annotations due to three major challenges: (1) limited fine-grained perception results in the omission of critical visual cues; (2) redundant or noisy visual information undermines the performance of
multimodal reasoning; (3) lack of adaptive deep
reasoning relative to the amount of visual information. To tackle these challenges, we present a novel focus-driven fine-grained chart
reasoning model, Chart-FR1, to improve perception, focusing efficiency, and adaptive deep
reasoning on HID charts. Specifically, we propose Focus-CoT, a visual focusing chain-of-thought that enhances fine-grained perception by explicitly linking
reasoning steps to key visual cues, such as local image regions and OCR signals. Building on this, we introduce Focus-GRPO, a focus-driven reinforcement learning
algorithm with an information-efficiency reward that compresses redundant visual information for efficient focusing, and an adaptive KL penalty mechanism that enables flexible control over
reasoning depth as more visual cues are discovered. Furthermore, to fill the gap in
benchmarks for HID charts, we build HID-Chart, a challenging
benchmark with an information-density metric designed to evaluate fine-grained chart
reasoning capabilities. Extensive experiments on multiple chart
benchmarks demonstrate that Chart-FR1 outperforms state-of-the-art MLLMs in chart understanding and
reasoning. Code is available at https://github.com/phkhub/Chart-FR1.
Read the full abstract ▾
All authors Hongkun Pan, Yuwei Wu, Wanyi Hong, Shenghui Hu, Qitong Yan, Yi Yang, Rufei Han, Changju Zhou, Minfeng Zhu, Dongming Han, Wei Chen
Computer Vision
BadmintonGRF: A Multimodal Dataset and Benchmark for Markerless Ground Reaction Force Estimation in Badminton
Kuoye Niu, Jianwei Li, Shengze Cai et al. (+6 more)
Multimodal resources for non-periodic court sports with laboratory-grade sensing remain scarce: few publicly pair instrumented ground reaction force (GRF) with high-frame-rate multi-view
video, limiting markerless load estimation in realistic training settings. BadmintonGRF records eight synchronized RGB views at ~120 FPS, four Kistler force plates, and Vicon motion capture (C3D) without hardware genlock across modalities;
alignment combines human-verified events, automated quality assurance, and per-camera time offsets with uncertainty metadata. Tier 1 distributes pose, time-aligned GRF, metadata, and splits under CC BY-NC 4.0, enabling the primary
benchmark without raw RGB or C3D; we report a Tier 1 task that maps 2D pose to GRF. Tier 2 provides raw RGB and C3D under controlled access for studies that require appearance or full kinematics. The public release contains 17,425 impact-segment archives in the 10-subject
benchmark tree (156 instrumented trials; raw multi-view RGB alone exceeds 1 TB);
benchmark loader gates retain 12,867 view-specific instances and 1,732 unique impacts after multi-view deduplication. We are not aware of prior public badminton corpora that combine this sensing layout with audited
video–GRF
alignment for impact-centric GRF estimation. We distribute preprocessing code, leave-one-subject-out splits, ten reference baselines, and optional late fusion (one deterministic test-time pass per instance; no test-time augmentation), with a within-trial diagnostic in the supplementary material.
Read the full abstract ▾
All authors Kuoye Niu, Jianwei Li, Shengze Cai, Yong Ma, Mengyao Jia, Lishun Shen, Zhenheng Zhang, Yuxin Peng, Xian Song
Computer Vision · Cross-listed from cs.IT
Evolving Token Communication with Parametric Memory Network
Weixuan Chen and Qianqian Yang
Token communication has emerged as a promising framework for efficient wireless transmission by representing source data as compact semantic
tokens. However, transmitting full semantic
tokens still incurs considerable communication overhead. In this paper, we propose an evolving semantic
token communication system with a parametric
memory network over
MIMO fading channels. Specifically, only an equal-length prefix of each semantic
token is transmitted, which reduces transmission cost while preserving a consistent
token structure for receiver-side recovery. At the receiver, a parametric
memory network is introduced to reconstruct the missing suffix information from the received
token prefixes, where semantic
memory is stored implicitly in the network parameters. To realize this design, full semantic
tokens are first organized into a codebook, and truncated
tokens are paired with the codeword labels of their corresponding full
tokens. Based on these
token-label pairs, kNN-based teacher distributions are constructed to fine-tune a pretrained GPT-2-based recovery module, which learns to infer the codeword distribution of each incomplete
token and recover the corresponding complete semantic
token. In addition, an online
evolution strategy is developed to periodically update the parametric
memory network and the entire system using newly observed test samples, thereby improving adaptability under distribution shifts. Experimental results demonstrate that the proposed method consistently outperforms the existing evolving
memory benchmark under different channel conditions and channel bandwidth ratios, with up to 1.09 dB PSNR improvement.
Read the full abstract ▾
Computer Vision
Decouple and Cache: KV Cache Construction for Streaming Video Understanding
Zhanzhong Pang, Dibyadip Chatterjee, Fadime Sener et al. (+1 more)
Streaming
video understanding requires processing unbounded
video streams with limited
memory and computation, posing two key challenges. First, continuously constructing new and evicting old key-value(KV) caches is required for unbounded streams. Secondly, due to the high cost of collecting and training on unbounded streams, models must learn from short sequences while generalizing to long streams. Existing streaming VideoVLLMs fail to scale to unbounded
video streams or focus on cache reuse strategies, leaving the impact of cache construction underexplored. In this paper, we propose Decoupled Streaming Cache(DSCache), a training-free cache construction mechanism that adapts pretrained offline models to streaming settings. DSCache maintains a cumulative past KV cache while constructing a separate instant cache on-demand, decoupled from past caches to preserve the informativeness of recent inputs. To enable position extrapolation beyond the training length, DSCache further incorporates a position-agnostic encoding strategy, ensuring KV caches to support unseen positions and preventing position overflow. Experiments on Streaming
Video QA
benchmarks demonstrate DSCache's state-of-the-art performance, with an average 2.5% accuracy gains over prior methods.
Read the full abstract ▾
All authors Zhanzhong Pang, Dibyadip Chatterjee, Fadime Sener, Angela Yao
Computer Vision
High-Fidelity Mobile Avatars with Pruned Local Blendshapes
Youyi Zhan, He Wang, Tianjia Shao et al. (+1 more)
We propose a method to reconstruct high-fidelity human avatars from multi-view
video that can run on mobile devices. Many works can model high-quality Gaussian-based full-body avatars from multi-view
video. However, these methods require heavy computation to obtain pose-dependent appearance, making deployment on mobile devices very difficult. Recent methods distill from pretrained models and model pose-dependent nonlinear Gaussian attributes by linearly combining global pose features with blendshapes. Although they can run on mobile devices, they suffer some loss of detail. We observe that nearby Gaussians are often highly correlated within a local region of the body, and can be linearly modeled with less error. Therefore, we use local linear blendshapes in small body parts to capture global nonlinear changes of Gaussian attributes. To further reduce computation and model size, we propose to remove blendshapes for Gaussians whose attributes change little, yielding a minimal blendshape representation. Our method is an end-to-end training method without a pretrained model. To make it run on multiple devices, we implement our method using WebGPU. Experiments show that our method can render high-quality human avatars with better details, and can reach 120 FPS at 2K resolution on mobile devices.
Read the full abstract ▾
All authors Youyi Zhan, He Wang, Tianjia Shao, Kun Zhou
Computer Vision
DP-SfM: Dual-Pixel Structure-from-Motion without Scale Ambiguity
Lilika Makabe, Kohei Ashida, Hiroaki Santo et al. (+2 more)
Multi-view 3D reconstruction, namely, structure-from-motion followed by multi-view stereo, is a fundamental component of 3D computer
vision. In general, multi-view 3D reconstruction suffers from an unknown scale ambiguity unless a reference object of known size is present in the scene. In this article, we show that multi-view images captured using a dual-pixel (DP) sensor can automatically resolve the scale ambiguity, without requiring a reference object or prior calibration. Specifically, the defocus blur observed in DP images provides sufficient information to determine the absolute scale when paired with depth maps (up to scale) recovered from multi-view 3D reconstruction. Based on this observation, we develop a simple yet effective linear method to estimate the absolute scale, followed by the intensity-based optimization stage that aligns the left and right DP images by shifting them back toward each other using cross-view blur kernels. Experiments demonstrate the effectiveness of the proposed approach across diverse scenes captured with different cameras and lenses. Code and data are available at https://github.com/lilika-makabe/dp-sfm-tpami.git
Read the full abstract ▾
All authors Lilika Makabe, Kohei Ashida, Hiroaki Santo, Fumio Okura, Yasuyuki Matsushita
Computer Vision
Disentangled Anatomy-Disease Diffusion (DADD) for Controllable Ulcerative Colitis Progression Synthesis
Umut Dundar and Alptekin Temizel
Synthesizing longitudinal medical images at controllable disease stages while preserving patient-specific anatomy is hindered by the
entanglement of pathological textures and structural features. We address this challenge for ulcerative colitis (UC) endoscopy, where severity follows a continuous ordinal progression along the Mayo Endoscopic Score (MES). Our framework, Disentangled Anatomy-Disease
Diffusion (DADD), conditions a latent
diffusion model on two complementary
embeddings: a pretrained image encoder for patient anatomy and a separately trained ordinal embedder for cumulative disease severity. Since image
embeddings inevitably capture disease information, we introduce a Feature Purifier, a cross-
attention-based erasure mechanism that identifies and suppresses disease-correlated channels, yielding purified anatomical representations. These cleaned anatomy
tokens and target disease
tokens are injected into the denoising network via a Triple-Pathway Cross-
Attention mechanism with resolution-dependent routing gates. This architecture leverages the U-Net hierarchy, in which different network depths encode global structure versus fine-grained pathological texture. Furthermore, we introduce Delta Steering, a training-free directional signal derived from the ordinal
embeddings that enables explicit, single-pass control over disease transitions at inference without requiring additional forward passes. Validated on the LIMUC
dataset, our approach produces high-fidelity images across all severity levels and effectively rebalances skewed class distributions, enhancing performance for downstream classification tasks. The
dataset is available at zenodo.org/records/5827695 and the code base at github.com/umutdundar99/progressive-stable-
diffusion
Read the full abstract ▾
Computer Vision
GeoSAE: Geometric Prior-Guided Layer-Wise Sparse Autoencoder Annotation of Brain MRI Foundation Models
Favour Nerrise, Lucy Yin, Mohammad H. Abbasi et al. (+2 more)
Brain MRI foundation models learn rich representations of anatomy, but interpreting what clinical information they encode remains an open problem. Standard sparse autoencoders (SAEs) suffer from severe feature collapse in deep
transformer layers, and in Alzheimer's disease (AD) research, aging confounds nearly every clinical variable, making naive annotation unreliable. We propose GeoSAE, a geometry-guided SAE framework that uses the foundation model's learned
manifold structure to prevent feature collapse and annotates each surviving feature via age-deconfounded partial correlations. Applied to ~14k T1-weighted MRI scans from the Alzheimer's Disease Neuroimaging Initiative (ADNI) and the Australian Imaging biomarkers and Lifestyle (AIBL)
datasets, GeoSAE identifies a compact, fully interpretable feature set that predicts mild cognitive impairment (MCI)-to-AD conversion (AUC 0.746) using only 2% of the
embedding dimensions, while comorbidity-annotated features achieve only chance-level performance. The identified features replicate across cohorts without retraining (r=0.97) and localize to neuroanatomically distinct regions consistent with Braak staging. This shows that geometry-guided SAEs can extract interpretable, biomarkers from frozen
brain MRI foundation models.
Read the full abstract ▾
All authors Favour Nerrise, Lucy Yin, Mohammad H. Abbasi, Kilian M. Pohl, Ehsan Adeli
Computer Vision
Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering
Yun Xing, Hanyuan Liu, Jiahao Nie et al. (+1 more)
Large
Multimodal Models (LMMs) have recently demonstrated their proficiency in holistic visual comprehension. However, most of them struggle to tackle region-level perception guided by visual
prompts, especially for cases where multiple regions are referred simultaneously, or scenarios where global contexts are necessary for precise visual referring. We introduce Contextual Latent Steering (CSteer), a training-free approach for guiding general LMMs to refer multiple regions contextually, without expensive fine-tuning or architectural modifications. CSteer starts with pre-computing contextual vectors that implicitly represent visual referring behaviors, such as differentiation among regions and
attention to global contexts, followed by representation editing during inference time. Experimental results on multiple
datasets indicate that general LMMs with CSteer outperform tailored referring LMMs in most cases, suggesting a promising solution in training-free, and setting new state-of-the-art for this field. Code is available at https://github.com/xing0047/csteer.git.
Read the full abstract ▾
All authors Yun Xing, Hanyuan Liu, Jiahao Nie, Shijian Lu
Computer Vision
Hybrid Visual Telemetry for Bandwidth-Constrained Robotic Vision: A Pilot Study with HEVC Base Video and JPEG ROI Stills
Natalia Trukhina and Vadim Vashkelis
Bandwidth-constrained robotic and surveillance systems often rely on a single compressed
video stream to support both continuous scene awareness and downstream machine perception. In practice, this creates a mismatch: low-bitrate
video can preserve motion and coarse context, but often loses the fine local detail needed for reliable object recognition and decision-making. Motivated by a hybrid architecture in which low-resolution
video supports dynamic scene understanding while eventdriven high-detail regions of interest (ROIs) support close-up identification and analytics, this paper formalizes a two-channel visual telemetry scheme in which a continuous low-bitrate
video stream is augmented by selectively transmitted high-detail still ROIs. This first paper does not attempt to prove the superiority of a new still-image codec. Instead, it establishes the hybrid transmission paradigm itself using a practical and reproducible codec stack: x265/HEVC for the base
video stream and JPEG stills for ROI refinement. We formulate the problem as bitrate-constrained information selection for robotic
vision and define an experimental protocol in which
video-only and hybrid schemes are compared under matched total communication budgets. The study is designed around
UAV-oriented
datasets, two practical bitrate regimes, several ROI triggering policies, and object-level classification refinement on selectively transmitted ROI stills. The resulting paper lays the methodological foundation for a second-stage investigation of JPEG AI as the semantic still-image channel within the same hybrid architecture.
Read the full abstract ▾
Computer Vision
Cross-Domain Adversarial Augmentation: Stabilizing GANs for Medical and Handwriting Data Scarcity
Md. Sohanuzzaman Soad, Mahady Al Hady, S M Rafiuddin Rifat et al. (+1 more)
Generative
Adversarial Networks (GANs) can help overcome data scarcity in computer
vision tasks by generating additional training samples. In this work, we explore generative data augmentation in two low-resource domains: Bangla handwritten character recognition and chest X-ray image analysis. We use DCGAN-based models trained on 64x64 images to generate synthetic samples and evaluate their quality using Inception Score (IS), Fréchet Inception Distance (FID), and visualization methods such as t-SNE and UMAP. To measure practical usefulness, we train image classifiers using real data and a combination of real and synthetic data. Experimental results show that synthetic augmentation improves data diversity and consistently increases classification performance in limited-data settings. We also investigate training stability techniques, including gradient penalty and spectral normalization, and perform ablation studies on synthetic-to-real data ratios and sample filtering strategies. In addition, we discuss challenges related to medical image evaluation,
dataset licensing, and
privacy concerns of synthetic data. Our approach is simple, reproducible, and provides a strong baseline for generative augmentation in resource-constrained imaging applications.
Read the full abstract ▾
All authors Md. Sohanuzzaman Soad, Mahady Al Hady, S M Rafiuddin Rifat, Sudip Ghose
Computer Vision
Embody4D: A Generalist Data Engine for Embodied 4D World Modeling
Peiyan Tu, Hanxin Zhu, Jingwen Sun et al. (+6 more)
Embodied agents require robust and comprehensive 3D spatiotemporal representations to support spatial
reasoning, manipulation understanding, and downstream decision making. However, existing
robot data are typically captured from fixed or sparse viewpoints, providing only partial and view-dependent observations, which limits multi-view perception and generalization across viewpoints. Given the difficulty of collecting additional viewpoints in real-world settings, we propose Embody4D, a dedicated
video-to-
video world model for
embodied scenarios to bridge this observation gap by transforming a
monocular robot video into novel-view
videos from flexible target camera viewpoints. First, to tackle training data scarcity, we introduce a 3D-aware compositional synthesis pipeline to curate a heterogeneous
dataset compositing cross-embodiment robotic arms with diverse backgrounds, promoting broad generalization. Second, to enforce
geometric stability, we devise a latent confidence-aware expert modulation strategy, which estimates the reliability of warped latent priors and adaptively routes regions to copy, repair, or inpaint experts for spatiotemporally consistent 4D generation. Finally, to enhance the fidelity of the manipulation, we incorporate an interaction-aware
attention mechanism that explicitly attends to the robotic interaction regions. Extensive experiments show that Embody4D achieves state-of-the-art performance on visual evaluation
benchmarks, while both simulated and real-world robotic experiments further demonstrate its effectiveness as a robust data engine for synthesizing high-fidelity, view-consistent
videos that empower downstream robotic planning and learning.
Read the full abstract ▾
All authors Peiyan Tu, Hanxin Zhu, Jingwen Sun, Shaojie Ren, Cong Wang, Yuyan Xu, Jiayi Luo, Xiaoqian Cheng, Zhibo Chen
Computer Vision
MedScribe: Clinically Grounded CT Reporting through Agentic Workflows
Giuseppe A. Orlando, Paolo Papotti, Maria A. Zuluaga et al. (+2 more)
Vision-
language models (VLMs) have shown potential for automated radiology report generation, yet existing approaches rely on global
embedding compression of volumetric data, often leading to hallucinated findings and limited anatomical grounding in 3D CT imaging. We introduce MedScribe, a hypothesis-driven framework that reformulates report generation as an iterative evidence acquisition process rather than a single-pass encoding task. MedScribe models reporting as a sequential decision process in which a large
language model dynamically invokes pathology-specific diagnostic tools to extract localized volumetric features. These structured features are used to query a multidimensional
retrieval space aligned with pathology-specific textual evidence. By explicitly accumulating quantitative evidence prior to synthesis, the framework enforces fine-grained grounding and reduces unsupported claims. Without task-specific fine-tuning, MedScribe improves clinical accuracy, factual consistency, and interpretability on CT-RATE and RadChestCT compared to state-of-the-art 2D and 3D VLMs, demonstrating the value of hypothesis-driven
reasoning for reliable medical image reporting.
Read the full abstract ▾
All authors Giuseppe A. Orlando, Paolo Papotti, Maria A. Zuluaga, Olivier Humbert, Marco Lorenzi
Computer Vision · Cross-listed from cs.LG
Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time
Itai Allouche and Joseph Keshet
Multimodal large
language models (MLLMs) achieve strong performance on
vision- and audio-
language tasks, yet can generate responses that conflict with the given visual or auditory inputs, a problem known as
multimodal hallucinations. Prior work suggests that this occurs when models rely more on textual cues and learned
language patterns than on evidence from the perceptual input. To obtain a more direct account of this imbalance, we apply Layer-wise Relevance Propagation (LRP), which attributes predictions to individual input
tokens, and use the resulting relevance scores to analyze and mitigate
hallucinations. First, we examine whether this imbalance leads to
multimodal hallucinations. We find that
hallucinations often arise when the model relies less on perceptual inputs, and that changing this reliance affects its predictions. We further leverage LRP and propose a training-free framework that shifts relevance toward perceptual
tokens by optimizing key-value representations during decoding, without modifying model parameters or requiring training data. We call this method Learning Inference-time Modality Enhancement (LIME). Despite using no spatial or temporal supervision, LIME concentrates relevance on query-relevant regions. We evaluate LIME across multiple
multimodal benchmarks in both
vision and audio domains, demonstrating consistent reductions in
hallucinations and enhanced grounding while preserving generation quality.
Read the full abstract ▾