Computer Vision
Pose2Muscle: Structured Spatio-Temporal Decoding for Discrete Muscle Activity Estimation from Human Pose
Yuepeng Chen, Jiehong Shi, Kaili Zheng et al. (+4 more)
Muscle activity is fundamental to human movement, and understanding its patterns is critical for injury prevention and rehabilitation. Conventional muscle activity monitoring relies on specialized sensors such as surface electromyography, which limits its practicality for long-term real-world use. Existing studies suggest that muscle-related information can be inferred from human pose. However, the substantial gap between externally observable pose and internal muscle activation, limits the accuracy and generalization of current approaches. In this study, we propose Pose2Muscle, a pose-driven framework for discrete muscle activity estimation without requiring sEMG signals at inference time. Instead of directly regressing continuous sEMG signals, Pose2Muscle reformulates muscle estimation as a structured prediction problem over discrete muscle activity states, yielding a more stable and interpretable target space. The framework combines multi-scale spatio-temporal
attention to capture motion patterns at complementary spatial and temporal scales with a directed acyclic graph-based decoder that maintains multiple candidate muscle-state hypotheses and performs structured trajectory inference over time. To support this task, we construct PoseEMG-43, a synchronized pose-sEMG
dataset containing 2,992 movement instances from 43 daily-life actions performed by 14 participants. Experiments show that Pose2Muscle consistently outperforms representative
retrieval- and pose-based baselines. It achieves an Adjacent-level Accuracy of 86.36% and a Pearson correlation coefficient of 0.8821 under the Random Split, and 63.97% and 0.6795, respectively, under the Subject-Level Split. These results demonstrate the feasibility of inferring structured muscle-state patterns from human pose and suggest the potential of Pose2Muscle for muscle-aware movement analysis when direct physiological sensing is impractical
Read the full abstract ▾
All authors Yuepeng Chen, Jiehong Shi, Kaili Zheng, Boyi Zhang, Chenyi Guo, Ji Wu, Xiangling Fu
Computer Vision
PDA++: Field-Aligned Planning and Scene-Adaptive Insertion in Remote Sensing
Xianchi Dong, Yingyan Hou, Chao Ren et al. (+6 more)
Remote sensing recognition is often constrained by scarce observations of rare targets and costly annotations, making realistic synthetic augmentation particularly valuable for few-shot and long-tailed scenarios. Object insertion provides an efficient way to increase target diversity while preserving authentic background scenes, but realistic insertion in overhead imagery requires the generated target to adapt coherently to its surrounding environment. To this end, we propose PDA++, a unified environment-aware object insertion framework organized as Plan, Decouple, and Assimilate. Planning determines scene-compatible poses through an affordance field that combines
geometric clearance with structure- and scale-aware cues. Decoupling introduces a pose-conditioned background that provides precise spatial guidance together with target-scene context, allowing the reference object to preserve its identity while adapting to the target observation. This construction also naturally provides pixel-level masks for
segmentation augmentation. Assimilation further improves local coherence by aligning multi-scale texture distributions through optimal transport. On the optical
benchmark, PDA++ achieves a whole-image FID of 6.28 and improves average few-shot recognition mAP50 by 17.69 points, corresponding to a 28.8% relative gain over the real-data baseline. On SAR imagery, it improves ship detection by 4.10 mAP50 points and remains effective under cross-
dataset transfer and amorphous-target insertion. Code is available at https://github.com/lisheyu972/PDA_PLUS.
Read the full abstract ▾
All authors Xianchi Dong, Yingyan Hou, Chao Ren, Wanxuan Lu, Zihan Wei, Hongfeng Yu, Yixiao Wang, Chubo Deng, Xian Sun
Computer Vision
Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model
Haoyu Zhao, Zihao Zhao, Tianyu Deng et al. (+10 more)
Recent Omni-Modal Generative Models (Omni-Models) have advanced content generation toward unified modeling of text, images,
video, and audio. MiniMax-H3 exemplifies this transition by combining
multimodal context understanding with joint audio-visual generation in a shared latent framework. Its unified architecture raises a fundamental question: Can
multimodal alignment improve the model's world
reasoning, and what new evaluation paradigms do omni-modal inputs enable? To investigate this question, this work introduces a comprehensive evaluation framework organized around four complementary dimensions of physical world
reasoning. Unlike existing evaluation frameworks for
video generation and world models, which are often constrained by limited input modalities and evaluation settings where
prompts closely match the target
video content, our evaluation is specifically designed to exploit the
multimodal inputs of Omni-Model. We construct a diverse set of novel tasks that require models to integrate complementary information across modalities. Specifically, we consider four scenarios, including implicit
prompts paired with multiple frames, audio-image, prefix-
videos, and audio-
video inputs. Every single modality provides only partial evidence about the underlying event, requiring the model to jointly reason over the complementary semantic cues to infer latent event states and future dynamics. Across 517 evaluation instances, MiniMax-H3 achieves an overall success rate of 41.97%.
Video-based Decision
Reasoning yields the highest success rate at 56.00%, while Audio-based Disambiguation
Reasoning is the weakest, reaching only 27.40%. These results indicate that effective
multimodal integration remains key to fully exploiting the benefits of diverse input modalities. The project is available at https://github.com/gulucaptain/MiniMax-H3-Reason.
Read the full abstract ▾
All authors Haoyu Zhao, Zihao Zhao, Tianyu Deng, Ziqin Xu, Zihao Zhang, Xudong Wang, Jinxiang Guo, Chen Gao, Ziyi Ye, Yeying Jin, Jiaxi Gu, Zuxuan Wu, Shuicheng Yan
Computer Vision
Visual Autoregressive Priors for RAW-to-sRGB Image Signal Processing
Tailai Chen, Xiaotong Luo, Yuan Gao et al. (+2 more)
RAW-to-sRGB image signal processing (ISP) must recover perceptually faithful colors and fine details from sensor measurements, often under imperfect spatial
alignment and missing camera metadata. This paper presents, to the best of our knowledge, the first application of visual autoregressive (VAR) next-scale prediction over a discrete image codebook to the RAW-to-sRGB ISP task. We adapt a frozen 1.10\,B-parameter VAR backbone for RAW-conditioned ISP with only 32.93\,M trainable parameters (2.99%), and propose a frequency-decomposed color loss that separately supervises low-frequency tone via wavelet LL cosine similarity and chromatic edges via detail-band $\ell_1$. On the Zurich RAW-to-sRGB
benchmark, the method improves PSNR-Y from 21.31 to 21.89\,dB and reduces LPIPS from 0.276 to 0.218 on the full 1,204-image test set. Diagnostic experiments show that the VAR prior preserves structure well, but continuous color transfer remains the dominant bottleneck: oracle affine correction recovers 3.8\,dB, while learned color heads yield marginal gains.
Read the full abstract ▾
All authors Tailai Chen, Xiaotong Luo, Yuan Gao, Xin Jin, Wenjun Zeng
Computer Vision
Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM
Victor Bercy, Martyna Poreba, Michal Szczepanski et al. (+1 more)
Vision Transformers (ViTs) have achieved state-of-the-art performance across a range of computer
vision tasks, mainly thanks to the self-
attention mechanism. However, its complexity, increasing quadratically with the number of
tokens, remains the major obstacle to ViT efficiency and deployment at scale.
Token merging reduces this cost by aggregating redundant
tokens. Yet existing methods are typically evaluated within a single architecture, leaving open whether their effectiveness stems from the merging mechanism itself or from the specific decoder they are paired with. We extend Graph-Guided
Token Merging (G2TM), a single module inserted early in a ViT-based network, beyond its original Segmenter setting. We evaluate G2TM across three semantic
segmentation frameworks (Segmenter, SETR, EoMT) and three decoder families (Linear,
Transformer-, convolution-based), as well as standard ViT image classification. Our results show that G2TM's behavior and accuracy-efficiency trade-off are consistent across every tested architecture for a given backbone size, indicating that its effectiveness is a property of the encoder rather than the decoder. G2TM also generalizes well to image classification, achieving an even smaller degradation in accuracy compared to semantic
segmentation. We further find that G2TM's optimal hyperparameters, resulting in a consistent drop in GFLOPs of 22-47% and an increase in throughput by up to 74% for
segmentation models on ADE20K
dataset, depend primarily on the backbone's pre-training recipe and on the target
dataset, rather than on the decoder choice.
Read the full abstract ▾
All authors Victor Bercy, Martyna Poreba, Michal Szczepanski, Samia Bouchafa
Computer Vision
MS-RFD: Multi-Signal Release Frame Detection in Hammer Throw from Reconstructed 3D Trajectories
Ahmed Endris Hasen, Nikolaos Passalis, Tomi Vanttinen et al. (+1 more)
Recent advances in artificial intelligence and computer
vision are reshaping sports performance analysis by enabling automated detection, tracking, and performance analysis. In hammer throw, performance is strongly determined by the kinematic conditions at release, particularly release speed, release angle, and release height. However, identifying the release instant from
video typically requires manual frame-by-frame inspection, which is subjective and cumbersome in real-world training scenarios. In this paper, we present a fully automatic multi-signal release frame detection (MS-RFD) method for hammer throw using reconstructed 3D hammer trajectories. The proposed method integrates four complementary kinematic signals: speed dynamics, angular velocity transition, radial distance relative to the rotation center, and post-release trajectory linearity. These signals are fused to score and verify candidate release frames. MS-RFD is evaluated through the throwing-distance estimation error obtained from the release parameters estimated at the detected frame. An ablation study analyzes the contribution of each signal and compares alternative candidate selection strategies. The results show that speed dynamics and radial expansion provide the strongest signals for release frame detection, while angular velocity and post-release linearity provide smaller refinements.
Read the full abstract ▾
All authors Ahmed Endris Hasen, Nikolaos Passalis, Tomi Vanttinen, Jenni Raitoharju
Computer Vision · Cross-listed from cs.RO
M2Tok: Multi-head Multi-codebook Discrete Action Tokenization for Vision-Language-Action Models
Chunpu Xu, Zhixuan Liang, Yuhao Zhang et al. (+6 more)
Recent advancements have successfully adapted autoregressive
language models to process
multimodal signals, such as images and actions. Since raw action signals are continuous, effective tokenization is essential to map high-dimensional inputs into compact discrete
tokens for autoregressive processing. However, existing discrete action tokenizers often suffer from high reconstruction loss, failing to preserve the fine-grained dynamics required for precise control. This "discretization bottleneck" significantly limits the performance ceiling of downstream
Vision-
Language-Action (VLA) models. To address this, we propose ${M}^2$Tok, a Multi-head Multi-codebook Action Tokenizer designed to minimize reconstruction error and enhance policy performance. Our approach introduces two key structural innovations: (1) we decompose the latent action features into multiple heads, enabling the model to implicitly align specific heads with distinct action dimensions; (2) we assign independent codebooks to each head for quantization. By leveraging the combinatorial nature of multiple codebooks, we significantly expand the representational expressivity of the tokenizer, leading to substantially lower reconstruction loss compared to previous methods. We evaluate the ${M}^2$Tok-based VLA on the RoboTwin, Simpler-Env, and 3 zero-shot real-world tasks. Experimental results demonstrate our method not only achieves superior reconstruction fidelity but also significantly boosts the success rate of VLA models. Comprehensive ablation studies further confirm the effectiveness of the multi-head and multi-codebook mechanisms. Code is available at https://github.com/cpaaax/M2Tok.
Read the full abstract ▾
All authors Chunpu Xu, Zhixuan Liang, Yuhao Zhang, Chi-Min Chan, Jessie Wang, Yang Xiao, Mengkang Hu, Xiaokang Yang, Yao Mu
Computer Vision
Evolving Error States: Failure-Aware Progressive Repair for Ultrasound Lesion Segmentation
Ziliang Wang, XuJiang Tang, Lu Yuting et al. (+4 more)
Reliability under sparse and heterogeneous failures remains a fundamental challenge for medical image
segmentation. High average accuracy can conceal a small set of structurally distinct and clinically consequential errors. Existing post-hoc correction methods alleviate this problem, but typically estimate false-positive and false-negative corrections from the same fixed prediction. This ignores the dynamic
evolution of error states and limits the correction of complex cases. Inspired by iterative error feedback in structured prediction, we propose Failure-Aware Progressive Repair (FAPR). FAPR represents the current
segmentation mask as a dynamic failure state and models each repair operation as a state-transition operator. Each accepted correction forms a new prediction state for subsequent error diagnosis and repair, enabling later operations to adapt to preceding changes. Conditional routing selectively activates necessary state transitions, while failure replay exposes the model to rare error states. By keeping the base segmentor frozen, FAPR preserves its established
segmentation capability while improving difficult cases. Across three public ultrasound lesion
segmentation benchmarks, FAPR improves mean DSC by 1.52%. On the very-hard subsets of BUSI and TN3K, the average gain reaches 13.77%.
Read the full abstract ▾
All authors Ziliang Wang, XuJiang Tang, Lu Yuting, Weixin Xu, Yongqiang Zhao, Ying Fu, Kehua Guo
Computer Vision
Unified Response Geometry for Structured Pruning
Kaixiang Shu
Structured
pruning is commonly formulated as ranking individual channels, although channel responses can be complementary or cancel through downstream mixing. Motivated by these response interactions, we formulate
pruning as the selection of a subset with large joint response capacity, followed by a separate functional realization step. Our unified response geometry maps each candidate set to \(M(D,R)=D^{1/2}RD^{1/2}\) and uses its determinant together with Schur-greedy residuals to select non-redundant coordinates. The same construction yields two information-conditioned instances: an unlabeled instance based on activation covariance, and a task-conditioned instance that combines activation and gradient variance for response scale with gradient correlation for complementarity. To convert the selected subset into an executable network, we fold predictable removed responses into successor weights through ridge compensation and recalibrate batch-normalization statistics, without fine-tuning the network. On ImageNet ResNet-50, the unlabeled instance reaches \(65.4\%\) and \(53.9\%\) Top-1 accuracy at 30% and 40% deletion, versus \(59.8\%\) and \(43.1\%\) for strength-only selection; the task-conditioned instance reaches \(67.7\%\) and \(56.3\%\) under the same protocol. A six-family screen shows architecture-dependent behavior, with positive relative contrasts in several convolutional and expansion-layer settings and clear boundary cases in windowed
attention. These results support response geometry as a conditional principle for structured
pruning, with its benefit determined jointly by the observed response and the architecture in which that response is realized.
Read the full abstract ▾
Computer Vision
WISE: A Lightweight, Weakly-Supervised Model for Onboard Fire Smoke Detection and Localization
Sha Lu, Yu Sun, Liang Zhao et al. (+6 more)
Wildfire smoke detection from satellite imagery is critical for early warning and rapid response. For onboard satellite deployment, detection systems must operate under strict
memory and latency constraints while providing spatially informative outputs for downstream decision-making. Existing tile-level classification methods are computationally efficient but lack spatial localization, whereas pixel-level
segmentation approaches provide detailed masks yet are typically too computationally demanding for real-time onboard execution. To address this gap, we propose WISE (Weakly-supervised Inference-efficient Smoke Extraction), a deployment-oriented framework for onboard fire smoke detection and localization. WISE leverages only tile-level annotations through a teacher-student
distillation strategy, where an offline teacher provides soft spatial supervision to a lightweight WISE-Student optimized for efficient onboard inference. The student jointly predicts tile-level smoke presence and smoke probability maps within a single forward pass, enabling spatially informative detection under strict computational constraints. WISE was evaluated through in-orbit execution aboard the ISS-mounted IMAGIN-e payload. Three model variants achieve average inference times of 0.10 s, 0.14 s, and 0.26 s per tile, indicating near-real-time per-tile inference within onboard resource limits. Ground-based experiments on Landsat 5 and Landsat 8 imagery further indicate effective detection and spatially informative localization. The best-performing variant achieves a mean tile-level F1 score of 0.964 and a mean pixel-level F1 score of 0.750 across 10 runs, while containing only 0.12M parameters and requiring approximately 3 GFLOPs. Together, these results indicate that WISE is a practical candidate for low-latency wildfire smoke monitoring from space under onboard resource constraints.
Read the full abstract ▾
All authors Sha Lu, Yu Sun, Liang Zhao, Jixue Liu, Lin Liu, Jiuyong Li, A. K. Qin, Alejandro Mousist, Stefan Peters
Computer Vision · Cross-listed from cs.LG
A Lightweight CNN Integrated Compact Convolutional Transformer for Multi-Scale Feature Learning and reducing computational complexity for breast cancer mammography image detection and classification
Md Taimur Ahad and Ainuddin Ahmed
Over the years, Convolutional Neural Networks (CNNs) have demonstrated strong capability in cancer detection and classification using medical images. However, CNN-based models often struggle to capture long-range contextual dependencies. In such scenarios, integrating Compact Convolutional
Transformer (CCT) architectures after the CCT layer allows CNN-extracted features to reshape into compact patch
tokens using a CCT tokenizer, followed by the addition of positional
embeddings to preserve spatial structure. Using 5-fold cross-validation, the model was tested on 3 sets of breast cancer mammography. With only 250,435 parameters, the model achieved 99%-100% accuracy across 3
datasets, indicating robust generalization. Explainable AI (XAI) was integrated into the model to explain the breast cancer classification process to enhance clinical trust. The results indicate that the proposed framework is suitable for computer-aided diagnosis systems, particularly in resource-constrained clinical environments. The novelty of the proposed CNN-integrated CCT overcomes the limitation of CNN's gradient degradation in the last layers by integrating convolutional tokenization with
transformer-based learning. Lighter than ViT, which is effective in capturing long-range dependencies, the model has also proven efficient in breast cancer classification by capturing long-range dependencies among breast tissue regions.
Read the full abstract ▾
Computer Vision
Understanding Dynamic Scenes at Gigapixel Scale: Wide-Area Spatio-Temporal Perception from UAVs
Yuhang Zhu, Meiyi Zhu, Yunkai Dang et al. (+4 more)
UAV-borne imaging has advanced from megapixel to gigapixel sensors, shifting aerial perception from recognizing individual targets to understanding entire dynamic scenes. We characterize this demand as Wide-area Spatio-temporal Scene Understanding (WSTU), which requires wide-area coverage, per-target resolution, and temporal continuity at once, a combination existing
datasets lack. To fill this gap, we introduce an ultra-High-resolution (12768x9564) Airborne Remote-sensing
Dataset (HARD) annotated at three levels for object detection, multi-object tracking, and scene-level visual question answering. Ultra-high-resolution imagery raises per-frame processing time to seconds. At that scale latency can no longer be ignored in evaluation. Thus, we propose a latency-aware metric for multi-object tracking called streaming-HOTA (s-HOTA). Extensive baseline experiments show how ultra-high-resolution processing reshapes each task. For detection, the end-to-end pipeline affects accuracy and speed as much as the detector itself does. For tracking, high latency charges the association axis far more unevenly than the detection axis, and association is where pipelines diverge. As a result, the pipeline that performs best offline can lose its lead under s-HOTA. For VQA,
vision-
language models remain weak at cross-frame identity binding and cannot transfer their single-frame gains to it. Together these findings show that the baselines we evaluate fall short of WSTU. HARD provides the data and the systematic baselines to advance it.
Read the full abstract ▾
All authors Yuhang Zhu, Meiyi Zhu, Yunkai Dang, Zhangnan Li, Yuxuan Wang, Wenbin Li, Hongbing Pan
Computer Vision
CapMap-MS-TTA: 3rd Place Solution for the MUMU Track of the 8th LSVOS Challenge at ECCV 2026
Chengfeng Qiu and Kaifeng Wei
The MUMU track of the 8th Large-scale
Video Object
Segmentation (LSVOS) Challenge requires a single unified
multimodal model to jointly solve image tagging (Task A), open-vocabulary object detection (Task B), and English captioning (Task C) under strict resource constraints (<=0.5B parameters and <=8 GB peak GPU
memory). We present CapMap-MS-TTA, a training-free submission built on Microsoft Florence-2-base (~231M parameters), combining caption keyword mapping with multi-scale flip test-time augmentation. Task C uses the native <DETAILED_CAPTION> pathway with length/
token sanitization. Task A maps the same detailed caption into the official quality/scene/event vocabularies via an expanded keyword lexicon with whole-word matching and a lightweight expand-hints stage. Task B runs Florence-2 open detection (<OD>) with multi-scale and horizontal-flip test-time augmentation (TTA), followed by label-aware non-maximum suppression (NMS). Without fine-tuning, the system improves our reproduced Florence-2 baseline from 15.16 to a best public score of 16.4815, and ranks 3rd on the final MUMU leaderboard.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Energy-Regularized Imitation Learning for Force- and Work-Aware Robotic Manipulation
Toshiki Otani, Hiromu Taketsugu and Norimichi Ukita
This paper studies energy-aware manipulation as a physically grounded learning problem. We define a joint-space mechanical-work proxy from joint torque and angular displacement, and train a differentiable energy predictor that estimates this work from
robot states and actions. The predictor converts a non-differentiable simulator-side physical quantity into a differentiable regularizer for fine-tuning a pretrained manipulation policy. We instantiate the framework with RVT-2 on RLBench and evaluate 12 manipulation tasks involving object contact, articulated motion, placement, pushing, and sweeping. The proposed fine-tuning reduces the average mechanical work from 208.8J to 204.4J (i.e., 2.1% reduction), while the mean task success rate also increases slightly from 86.2% to 86.9%. These results show that work-aware policy optimization can suppress physically inefficient motion without requiring an explicit differentiable dynamics model.
Read the full abstract ▾
Computer Vision
Multi-View Mixture-of-Experts with Vision-Language Reranking for Cross-View Object Geo-Localization
Xuyu Fan, Qi Ming, Zhu Han et al. (+6 more)
Cross-view object geo-localization (CVOGL) locates a target in satellite imagery using drone or street-view queries. Existing methods train separate detectors for each viewpoint, leading to parameter redundancy and impeding cross-view knowledge sharing. Moreover, top-ranked satellite candidates are often visually similar, so visual appearance and categorical labels alone are insufficient to resolve such ambiguity. To address these, we propose MVLGeo, an efficient framework designed to unify multiple viewpoints and reduce model redundancy. First, we introduce environmental contextual text from the query view as cues to distinguish visually similar candidates via
Vision-
Language Reranking (VL-Rerank). Second, we design a multi-view Mixture-of-Experts architecture (MV-MoE) with a shared encoder and view-specific experts to reduce redundancy and promote knowledge sharing, while cross-view contrastive learning aligns their representations for consistency. Third, we introduce an adaptive elliptical prior (ESAM-Prior) as auxiliary positional encoding for anisotropic
geometric perception. Extensive experiments on the CVOGL
benchmarks confirm that MVLGeo, as a unified model for multiple query viewpoints, achieves state-of-the-art performance, demonstrating robustness to input degradation and generalization across viewpoints. Code and models will be available on GitHub to facilitate future work.
Read the full abstract ▾
All authors Xuyu Fan, Qi Ming, Zhu Han, Liuqian Wang, Siyuan Cao, Xiaohan Zhang, Xudong Zhao, Mingjing Zhao, Yuhan Zhang
Computer Vision
Stealthy in Semantics, Antagonistic in Space: Attacking Visible-Infrared Object Detectors via Object-Level Misalignment
Yueqi Zhu, Qi Ming, Guo Cheng et al. (+6 more)
Visible-infrared object detectors are used for robust perception under challenging illumination and weather conditions. Current physical attacks apply conspicuous patches to spatially aligned target regions, which are noticeable to human observers. Meanwhile, most of these methods only perturb the appearance within the aligned region, without explicitly targeting the correspondence between modalities or the fusion process. In this paper, we propose CamoShift, an
adversarial framework for visible-infrared object detection. By combining visual camouflage with object-level infrared shifting, CamoShift breaks cross-modal spatial
alignment and disrupts fusion. Specifically, the Semantic Camouflage Module (SCM) generates a stealthy camouflaged patch that can be attached to the host object and maintains its effectiveness in the infrared branch through an RGB-IR adapter. The Object-level Spatial Decoupling Module (OSDM) shifts the infrared target evidence in a scale-aware manner, so as to break object-level correspondence and disrupt cross-modal fusion. Then, the Harmonic
Adversarial loss (HarAdv loss) further balances attack strength and visual stealth during optimization. To the best of our knowledge, we are the first to target both visual stealthiness and attack success in visible-infrared object detection. Extensive experimental results show that CamoShift achieves a superior balance between attack effectiveness and visual stealth. Code and models will be available on GitHub.
Read the full abstract ▾
All authors Yueqi Zhu, Qi Ming, Guo Cheng, Yongkang Zhang, Feiran Liu, Juan Fang, Jiahuan Zhou, Jiangmeng Li, Yuhan Zhang
Computer Vision
MCLC-NET: Multimodal Continual Learning for Leaf Counting
Ruchi Bhatt, Pratibha Kumari, Shreya Bansal et al. (+3 more)
Leaf counting is an important task in plant phenotyping for monitoring plant growth and estimating crop yield. Most existing methods rely on RGB images, but their performance is often affected by occlusion, lighting variations, and other real-world challenges. Additional modalities, such as depth and thermal images, can provide useful complementary information. However,
multimodal leaf counting remains underexplored. Also, many existing methods assume that all training data are available simultaneously, which is impractical in real agricultural settings, where data is collected over time from multiple sources. To address these challenges, we propose MCLC-NET, a
multimodal continual learning framework for leaf counting. It learns tasks sequentially using a
memory-based strategy with a
memory buffer to retain important samples from previous tasks. We also introduce MMLC, a real-world
multimodal leaf-counting
dataset designed for a domain incremental scenario (DIS) in CL. It contains RGB, depth, and thermal images collected across different crop types under varying environmental conditions, arranged in three orderings: crop-wise, time-wise, and mixed. Experimental results, averaged over three random seeds, demonstrate that MCLC-NET consistently outperforms existing methods across all three task orderings, achieving the lowest AMSE of 0.675$\pm$0.027, 0.542$\pm$0.069, and 0.745$\pm$0.057, respectively.
Read the full abstract ▾
All authors Ruchi Bhatt, Pratibha Kumari, Shreya Bansal, Vedant Agnihotri, Dwarikanath Mahapatra, Mukesh Saini
Computer Vision
PRISM: Predictive Representation of Interaction Style and Motion for Social Robot Navigation
Bo-Han Chen, Hiromu Taketsugu and Norimichi Ukita
Humans often observe others before interacting and adjust their behavior accordingly.
Robot navigation in crowds, however, often represents pedestrians mainly by observed
geometric states, leaving individual differences in interaction tendencies implicit. We propose PRISM (Predictive Representation of Interaction Style and Motion), a framework that infers interaction traits from passive observations of human-human interactions. PRISM encodes human trajectories into a continuous ordinal latent space with a
transformer encoder trained by Rank-N-Contrast loss, and pairs each inferred trait with a temporal-stability score supplied to the
navigation policy. In randomized crowd simulations, PRISM reduces collision rates over the geometry-only baseline and yields small improvements in
navigation-time and path-length metrics. These results suggest the utility of passive latent-trait inference for social
navigation in dynamic crowds.
Read the full abstract ▾
Computer Vision
Aligned Consensus Teaching for Label-Efficient Oriented Object Detection in Weakly-Aligned Visible-Infrared Imagery
Qi Ming, Xiaxin Yuan, Jiahuan Zhou et al. (+6 more)
Visible-infrared object detection (VIOD) detects objects with oriented bounding boxes from paired visible and infrared images. Existing methods depend on costly dual-modality annotations. Semi-supervised learning can reduce this burden, but extending it from single-modal detection to VIOD is challenging. In the practical image-pair-level setting considered here, only a few pairs are labeled in both modalities, while the rest are completely unlabeled. This limited supervision creates three challenges: (i) too few labeled boxes for robust cross-modal
alignment; (ii) pseudo-label errors caused by branch-wise misses accumulate during self-training; and (iii) tail-class annotations become critically scarce as the labeling budget decreases. We propose Aligned Consensus Teacher (ACT) for label-efficient VIOD in this setting. Its Cycle-Consistent Region
Alignment (CRA) combines cycle consistency and sparse anchors with reliability-weighted regional matching. Cross-Modal Consensus Mean-Teacher (CMC-MT) forms consensus pseudo labels under pair-preserving views to recover branch-wise misses and supervise unlabeled pairs. Text-Guided Cross-Modal Instance Augmentation (TG-CMIA) uses a
vision-
language scene prior to compose tail-class instance pairs while preserving RGB–IR offsets. To the best of our knowledge, ACT is the first framework to study semi-supervised VIOD under this image-pair-level setting. Experiments on DroneVehicle and VEDAI show consistent gains across annotation ratios. With 10% labeled pairs on DroneVehicle, ACT reaches 94.3% of the mAP obtained by the same detector under full supervision. Code and models will be available on GitHub to facilitate future work.
Read the full abstract ▾
All authors Qi Ming, Xiaxin Yuan, Jiahuan Zhou, Jiangmeng Li, Xudong Zhao, Zhanchao Huang, Juan Fang, Shaoguang Huang, Aleksandra Pizurica
Computer Vision · Cross-listed from cs.RO
A Comprehensive Review of Generative Physical Artificial Intelligence
Satyam Gaba, Krutiksinh Rana, Siva Sai et al. (+2 more)
The integration of large-scale foundation models with physical embodiments has led to significant advancements in robotics known as Generative Physical Artificial Intelligence (GPAI). These agentic AI systems autonomously perceive, reason, and act in complex real-world situations. This survey comprehensively analyzes GPAI systems, focusing on their architectural foundations, current applications, and key limitations. We introduce a taxonomy of five distinct approaches:
Robot Foundation Models (RFMs) for cross-platform skill transfer;
Vision-
Language Action (VLA) models for end-to-end multi-modal perception and control; Large Behavior Models (LBMs) for human-like movement generation;
Diffusion Policy Models (DPMs) for
diffusion model-based temporally coherent action generation; and World Foundation Models (WFMs) for physics-compliant simulation and data generation. We examine how these approaches complement each other: WFMs generate training data for VLAs and DPMs, RFMs enable cross-platform deployment of learned policies, while LBMs provide motion priors for natural behavior. Through examples across autonomous vehicles, industrial automation, healthcare robotics, and
humanoid systems, we identify significant performance improvements and summarize promising research directions in data-efficient learning, sim-to-real transfer, edge-compatible architectures, and safety frameworks. These insights advance
embodied AI for IoT-connected environments where intelligent
agents interact with networked sensors, actuators, and edge devices.
Read the full abstract ▾
All authors Satyam Gaba, Krutiksinh Rana, Siva Sai, Vinay Chamola, Dusit Niyato
Computer Vision · Cross-listed from cs.RO
Beyond Pixel Similarity: Task-Aware Evaluation of GAN-Based Synthetic Sonar Data for Robotic Perception
Hannan Ejaz Keen, Muhammad Moazam Fraz and Karsten Berns
Synthetic data can reduce the cost of collecting and annotating training data for robotic perception, but generating sensor observations that preserve the characteristics relevant to downstream perception remains challenging, particularly for sonar imagery. In this work, we investigate whether conventional image-fidelity metrics adequately reflect the downstream perception performance of GAN-generated synthetic sonar data. We employ a Pix2Pix conditional generative
adversarial network with four discriminator configurations characterized by different receptive fields: PixelGAN, PatchGAN-16, PatchGAN-70, and ImageGAN. The models are trained using sonar imagery from two
datasets and evaluated using conventional image-fidelity metrics, including Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Mean Squared Error (MSE). To complement these pixel-level measures with task-oriented evaluation, YOLOX-S, YOLOX-L, and Faster R-CNN detectors are trained exclusively on real sonar imagery and subsequently evaluated on the GAN-generated images using identical test samples and annotations across all discriminator configurations. The results reveal a discrepancy between image-fidelity and downstream object-detection performance: the configuration achieving the best SSIM, PSNR, and MSE does not consistently yield the best detection performance. In particular, PatchGAN configurations achieve strong downstream detection results despite not achieving the highest pixel-level similarity scores. These findings suggest, for the
datasets and models considered, pixel-level image-fidelity metrics alone may not consistently capture the task-relevant realism of synthetic sonar observations and motivate the use of task-aware evaluation for synthetic sensor data intended for robotic perception.
Read the full abstract ▾
Computer Vision
Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration
Zhixin Cheng, Jiacheng Deng, Xiaotian Yin et al. (+3 more)
Detection-free methods for image-to-point cloud registration are prone to erroneous correspondences caused by domain and modality discrepancies, limited sensitivity of feature extractors, and the presence of non-overlapping regions. The Masked Autoencoder (MAE) has shown strong performance in visual representation for images and point clouds. It may be helpful to apply this approach to image-to-point cloud registration, a task that requires unified feature extraction and accurate cross-modal correspondences. Standard MAE's random masking may overlook key regions due to limited camera views, reducing registration effectiveness. To address this, we propose the Intermodal Dual-MAE Framework (ID-MAE) with a Similarity-based RL Masking Strategy (SRLM), which adaptively masks informative positions by leveraging cross-modal similarity and reinforcement learning, thus narrowing the modality gap. Our method enhances cross-modal representation learning by enforcing representation consistency during feature extraction, thereby enabling more reliable 2D-3D correspondence estimation. Experiments on RGB-D Scenes v2 and 7-Scenes
benchmarks show that our method achieves state-of-the-art performance in image-to-point cloud registration.
Read the full abstract ▾
All authors Zhixin Cheng, Jiacheng Deng, Xiaotian Yin, Baoqun Yin, Richang Hong, Tianzhu Zhang
Computer Vision · Cross-listed from cs.RO
Not All Layers Need Tuning: Diagnosing and Directing Adaptation in Vision-Language-Action Models
Shahram Najam Syed, Arthur Jakobsson, Prayuj Sachdev et al. (+1 more)
Fine-tuning a
Vision-
Language-Action (VLA) model for a new deployment environment is expensive, yet most methods apply uniform-capacity adapters to every network region as if every region requires equal adjustment. This paper tests that assumption on five architecturally diverse VLAs (OpenVLA-OFT, $\pi_0$, SmolVLA, DTP, Octo; 93M-7B parameters). Measuring per-region adaptation cost as normalized parameter displacement under region-isolated fine-tuning reveals an adaptation spectrum in which appearance shifts concentrate cost in the
vision encoder, instruction shifts in the
language backbone, and novel-object shifts in the
vision encoder together with the action head, across all five architectures. To exploit this structure, we introduce a pipeline that observes, diagnoses, allocates, and adapts. From ten unlabeled target observations and without fine-tuning, the diagnostic estimates per-region cost by combining reference-free gradient and Monte Carlo Dropout signals with a Centered Kernel
Alignment score against a cached source reference; the allocator converts the estimates into variable-rank LoRA adapters under a parameter budget and freezes well-calibrated regions; and standard LoRA fine-tuning trains the resulting adapters. The diagnostic ranks regions within each deployment at a median Spearman of 0.91, and the allocation matches or exceeds uniform LoRA at every budget we tested on LIBERO and CALVIN. On a physical xArm-7, the pipeline matches full fine-tuning under an instruction-wording shift with 0.04% of its trainable parameters, and on five held-out scenes evaluated without retraining it leads every baseline, with 11-23 successes of 30 rollouts against 8-18 for the strongest parameter-efficient baseline at equal or larger budgets and 2-11 for full fine-tuning. These results suggest that adaptation cost in VLAs is structured enough to measure before fine-tuning begins.
Read the full abstract ▾
All authors Shahram Najam Syed, Arthur Jakobsson, Prayuj Sachdev, Jeffrey Ichnowski
Computer Vision
vidax: A Unified JAX Framework for Video Generative Models on Accelerator Meshes
Congyue Deng
Open-source
video generative models ship almost exclusively as PyTorch/CUDA reference implementations. This leaves Cloud TPU pods without a production-ready inference path, despite offering large, cost-effective accelerator
memory pools ideal for long-sequence spatiotemporal
attention. We present vidax, an open-source JAX/Flax inference engine and zero-copy PyTorch-to-JAX weight translator for modern
video generation architectures. vidax covers a diverse set of spatiotemporal models — including
Diffusion Transformers, omnimodal Mixture-of-
Transformers, 3D VAEs, text encoders, and native samplers — with zero PyTorch dependency in the execution path. The framework unifies 1D tensor parallelism with DeepSpeed-Ulysses sequence parallelism on a single JAX sharding mesh, integrates TPU flash-
attention kernels, and implements per-layer weight offloading to support reference resolutions that exceed single-device
memory. We
benchmark compile times, latency, and peak
memory utilization on TPU v4-8 hardware, and document real-world numerical bugs surfaced during checkpoint translation. vidax is released open-source as a baseline for JAX and TPU
video generation research.
Read the full abstract ▾
Computer Vision
GeoCueFormer: Geometry-Guided Wavelet Representation and Prediction-Cued Dual-Stage Decoder for Underwater Semantic Segmentation
Xian Wu, Xinjin Li, Yiliu Xu et al. (+2 more)
Underwater semantic
segmentation is essential for marine ecosystem monitoring, yet remains challenging due to severe visual degradation. Light absorption and scattering often lead to color shifts, low contrast, and blurred boundaries, making shallow detail features unreliable. Existing
underwater segmentation methods improve RGB feature aggregation or boundary prediction, but still lack an explicit mechanism to distinguish structure-related details from degradation-induced responses. To address this limitation, we propose GeoCueFormer, a lightweight framework that combines geometry-constrained frequency enhancement with prediction-cued refinement. GeoCueFormer performs stage-specific wavelet enhancement on hierarchical encoder features to complement shallow boundary details while preserving deep structural semantics. A depth-derived spatial gate constrains shallow frequency enhancement toward geometry-consistent regions, and a prediction-cued dual-stage decoder further refines ambiguous high-resolution features. GeoCueFormer obtains 82.23% and 73.04% mIoU on SUIM and DUT, respectively. Under comparable model complexity and standard
benchmark settings on SUIM and DUT, it achieves SOTA performance while maintaining a favorable accuracy-complexity trade-off. These results show that distinguishing structural details from degradation-induced interference is more effective for
underwater segmentation.
Read the full abstract ▾
All authors Xian Wu, Xinjin Li, Yiliu Xu, Yining Liu, Yong Jiang
Computer Vision
Finder: Agentic Closed-Loop Object Finding for Embodied Grounding
Shixiong Xu, Zhiyuan Chen, Song Ding et al. (+4 more)
Finding the object referred to by
language in a partially observed 3D scene is a core capability for
embodied agents. Existing approaches either couple object search with online exploration, which can be costly when relevant observations have already been captured, or query pre-built open-vocabulary maps and scene graphs in a static, one-shot fashion. We present Finder, an agentic closed-loop object-finding primitive for
embodied grounding. Instead of treating grounding as passive
retrieval from a fixed scene representation, Finder maintains a typed loop state that links query-conditioned planning, scoped evidence gathering, candidate verification, and accept/continue/abort control. When evidence is incomplete or ambiguous, the loop can redirect subsequent perception and comparison rather than simply returning the top retrieved object. On open-vocabulary
embodied Object
Retrieval in Habitat/HM3D and real-world RGB-D scenes, Finder improves the averaged 1m success rate by 15.75 points over strong baselines. The same primitive also transfers to sequential object grounding and
embodied object-centric question answering, improving spatial and temporal localization without changing the inner grounding protocol. Project page: https://finder-vln.github.io.
Read the full abstract ▾
All authors Shixiong Xu, Zhiyuan Chen, Song Ding, Rui Luo, Xiaowei Liang, Dongxu Miao, Zhiying Du
Computer Vision
Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformers
Zheng Liu, Xin Gao, Jinchao Zhu et al. (+1 more)
Parameter-efficient fine-tuning (PEFT) has recently emerged as a pivotal research direction for adapting pre-trained point cloud
transformers to diverse downstream tasks. Although existing methods achieve excellent fine-tuning performance with high parameter efficiency, they ignore inference efficiency. To tackle this problem, a novel PEFT method termed position anchor tuning (PAT) is proposed in this paper. As multi-head
attention (MHA) and feed-forward network (FFN) are computation-heavy blocks in pre-trained
transformers, PAT decreases their computational cost through
token aggregation-expansion pairs. Each pair comprises a
token aggregation module (TAM) and a
token expansion module (TEM). For MHA and FFN blocks, TAMs extract representative
tokens from their input
tokens based on position anchors in 3D space. These extracted
tokens, rather than the original input
tokens, are processed by the blocks, thereby reducing the number of
tokens involved in computation. Then, TEMs propagate the learned representations back to the original input
tokens. Since TAMs are solely responsible for capturing task-specific representations, base-sharing low-rank adaptation (BSLoRA) is further introduced to enable them to learn such representations effectively with only a small number of trainable parameters. Extensive experiments on widely used
benchmarks demonstrate that PAT performs comparably to state-of-the-art methods while incurring significantly lower computational overhead and fewer trainable parameters.
Read the full abstract ▾
All authors Zheng Liu, Xin Gao, Jinchao Zhu, Gao Huang
Computer Vision
CoAtNet-DeepMoE: A Convolution-Attention Hybrid with DeepSeek Mixture-of-Experts for Parameter-Efficient Tomato Disease Classification
Md Nadim Mahamood, Md Arif Shahriar, Md Shafi Ud Doula et al. (+1 more)
The world population is growing rapidly, and technology is improving in parallel. Meeting the huge demand for food for these 7 billion people not only depends on increasing food production but also on reducing food loss. Crop losses due to disease affect both the food supply and the financial and economic stability of a country. Tomatoes are among the top food-producing crops globally, and a significant portion of this production is lost due to disease. People have used Machine Learning techniques for feature extraction and early diagnosis of tomato diseases, and nowadays, Deep Learning-based models are widely used for disease recognition. However, most existing models are highly parameter-intensive, which increases the time required for training and inference. As a result, while lightweight models are more suitable for user-friendly applications, they often show a reduction in performance. To balance performance and model size, we propose CoAtNet-DeepMoE, a Convolution-
Attention hybrid architecture for rich feature extraction, further enhanced with a DeepSeek Mixture of Experts to substantially reduce the number of parameters without sacrificing accuracy. We evaluate our model on both balanced and imbalanced
datasets from Kaggle and PlantVillage, demonstrating robustness and achieving 99.80% accuracy, 99.80% precision, 99.80% recall, and 99.80% F1-score on Kaggle, and 99.83% accuracy, 99.85% precision, 99.76% recall, and 99.80% F1-score on PlantVillage, representing state-of-the-art performance with only 2.47M parameters. The source code will be available at https://github.com/nadimbrur/CoAt-MoE.
Read the full abstract ▾
All authors Md Nadim Mahamood, Md Arif Shahriar, Md Shafi Ud Doula, Kamrul Hasan
Computer Vision
SetPlanner: A Lightweight Plug-in Point-Set Planner for Frozen SAM
Dawei Yan, Yuezhe Yang, Menglan Ruan et al. (+2 more)
Segment Anything Models provide reusable priors, yet they require user
prompts and cannot support fully automatic instrument
segmentation. Automatic prompting is difficult for thin, articulated, reflective, and partly occluded tools, where several configurations can be valid. We formulate automatic prompting as lightweight point-set planning and isolate the point source under a frozen pathway. To this end, we present SetPlanner, a 1.52M-parameter plug-in point-set planner for frozen SAM. The plug-in preserves SAM's point-
prompt interface and enables reuse across backbones. SetPlanner plans complete unordered K-point sets from geometry-aware targets with a permutation-aware conditional flow. SAM decodes eight candidates; their consensus readout yields a ground-truth-free prediction. Across three endoscopic
datasets, SetPlanner wins all six transfer routes over a LoRA-adapted system. Under our frozen-pathway protocol, SetPlanner reaches 0.934 Dice on Kvasir-Instrument and recovers 96% of a 44.4-point localization gap, while candidate disagreement ranks low-Dice cases at AUROC 0.969.
Read the full abstract ▾
All authors Dawei Yan, Yuezhe Yang, Menglan Ruan, Chunfeng Yang, Yudong Zhang
Computer Vision
IRIS: Implicit Rendering Matters for Pose-Free Novel View Synthesis
Wenyu Li, Sidun Liu, Peng Qiao et al. (+2 more)
Novel view synthesis from unposed multi-view images remains challenging, as the model must jointly learn scene representations and camera parameters without pose supervision. Existing approaches largely fall into two extremes: implicit latent-space rendering is flexible and easy to optimize, but often yields weakly grounded camera estimation; explicit 3D representations provide stronger
geometric grounding, but introduce heavier parameterization and more fragile optimization. In this paper, we present IRIS, a fully self-supervised framework that provides a practical middle ground between these two paradigms. Instead of decoding free latent
tokens or reconstructing fully explicit 3D primitives, IRIS represents the scene as a latent neural field and renders novel views by querying this field under self-predicted cameras. Specifically, projected features from reference views are aggregated at sampled 3D points to form point-wise latent features, which are then composed along target rays for rendering. This design preserves the flexibility and optimization stability of implicit modeling, while introducing stronger
geometric structure than unconstrained latent rendering. Extensive experiments show that IRIS achieves strong novel view synthesis quality with competitive pose accuracy under fully self-supervised learning. Our project page: https://leo-frank.github.io/IRIS
Read the full abstract ▾
All authors Wenyu Li, Sidun Liu, Peng Qiao, Yong Dou, Tongrui Hu