Computer Vision
Sphere-Depth: A Benchmark for Depth Estimation Methods with Varying Spherical Camera Orientations
Soulayma Gazzeh, Giuseppe Mazzola, Liliana Lo Presti et al. (+1 more)
Reliable depth estimation from spherical images is crucial for 360{\deg}
vision in robotic
navigation and immersive scene understanding. However, the onboard spherical camera can experience unintentional pose variations in real-world robotic platforms that, along with the
geometric distortions inherent in equirectangular projections, significantly impact the effectiveness of depth estimation. To study this issue, a novel public
benchmark, called Sphere-Depth, is introduced to systematically evaluate the robustness of
monocular depth estimation models from equirectangular images in a reproducible way. Camera pose perturbations are simulated and used to assess the performance of a popular perspective-based model, Depth Anything, and of spherical-aware models such as Depth Anywhere, ACDNet, Bifuse++, and SliceNet. Furthermore, to ensure meaningful evaluation across models, a depth calibration-based error protocol is proposed to convert predicted relative depth values into metric depth values using supervised learned scaling factors for each model. Experiments show that even models explicitly designed to process spherical images exhibit substantial performance degradation when variations in the camera pose are observed with respect to the canonical pose. The full
benchmark, evaluation protocol, and
dataset splits are made publicly available at: https://github.com/sgazzeh/Sphere_depth
Read the full abstract ▾
All authors Soulayma Gazzeh, Giuseppe Mazzola, Liliana Lo Presti, Marco La Cascia
Computer Vision
Enhanced Privacy and Communication Efficiency in Non-IID Federated Learning with Adaptive Quantization and Differential Privacy
Emre Ard{\i}ç and Yakup Genç
Federated learning (FL) is a distributed machine learning method where multiple devices collaboratively train a model under the management of a central server without sharing underlying data. One of the key challenges of FL is the communication bottleneck caused by variations in connection speed and bandwidth across devices. Therefore, it is essential to reduce the size of transmitted data during training. Additionally, there is a potential risk of exposing sensitive information through the model or gradient analysis during training. To address both
privacy and communication efficiency, we combine differential
privacy (DP) and adaptive quantization methods. We use
Laplacian-based DP to preserve
privacy, which is relatively underexplored in FL and offers tighter
privacy guarantees than Gaussian-based DP. We propose a simple and efficient global bit-length scheduler using round-based cosine annealing, along with a client-based scheduler that dynamically adapts based on client contribution estimated through
dataset entropy analysis. We evaluate our approach through extensive experiments on CIFAR10, MNIST, and medical imaging
datasets, using non-IID data distributions across varying client counts, bit-length schedulers, and
privacy budgets. The results show that our adaptive quantization methods reduce total communicated data by up to 52.64% for MNIST, 45.06% for CIFAR10, and 31% to 37% for medical imaging
datasets compared to 32-bit float training while maintaining competitive model accuracy and ensuring robust
privacy through differential
privacy.
Read the full abstract ▾
Computer Vision
A Heterogeneous Two-Stream Framework for Video Action Recognition with Comparative Fusion Analysis
Md. Afzalur Rahaman and Tahmid Rahman
Most two-stream action recognition networks apply the same convolutional backbone to both RGB and optical flow streams, ignoring the fact that the two modalities have fundamentally different structural properties. Optical flow captures fine-grained motion patterns, while RGB frames carry rich appearance and scene context - treating them identically discards this distinction. We propose DualStreamHybrid, a heterogeneous two-stream architecture that assigns each stream a backbone suited to its input: a pretrained ViT-Tiny/16 for RGB frames, and a MobileNetV2 trained from scratch on a 20-channel stacked optical flow representation. A learned projection layer maps the two differently-sized feature vectors to a common dimensionality before fusion, enabling the two streams to interact without forcing architectural
symmetry. We design five fusion strategies within a unified framework - late fusion, concatenation, cross-
attention, weighted fusion, and gated fusion - and evaluate them on UCF11 (1,600
videos, 11 classes) and UCF50 (6,681
videos, 50 classes) to study how fusion behaviour scales with
dataset size. On UCF11, cross-
attention achieves 98.12% test accuracy, outperforming the RGB-only ViT-Tiny baseline of 95.94%, which suggests that explicit inter-modal
attention is particularly effective on smaller, less complex
datasets. On UCF50, weighted fusion reaches 96.86% and proves the most consistent strategy across both
benchmarks. The learned stream weights reveal an interesting pattern: UCF11 sees near-equal modality contribution (RGB: 0.507, flow: 0.493), while UCF50 favours the RGB stream slightly more (RGB: 0.554, flow: 0.446) - arguably reflecting the larger and more visually diverse action space. Taken together, these results suggest that even a lightweight motion stream meaningfully complements a strong appearance encoder, and that the optimal fusion strategy depends on
dataset scale.
Read the full abstract ▾
Computer Vision
PushupBench: Your VLM is not good at counting pushups
Shengzhi Li, Jiarun Chen, Karun Sharma et al. (+2 more)
Large
vision-
language models (VLMs) can recognize what happens in
video but fail to count how many times. We introduce PushupBench, 446 long-form clips (avg. 36.7s) for evaluating repetition counting. The best frontier model achieves 42.1% exact accuracy; open-source 4B models score $\sim$6%, matching supervised baselines. We show that accuracy alone misleads – weaker models exploit the modal count rather than reason temporally. Fine-tuning on counting with 1k samples transfers to general
video understanding: MVBench (+2.15), PerceptionTest (+1.88), TVBench (+4.54), suggesting counting is a proxy for broader temporal
reasoning.PushupBench incorporated in lmms-eval (https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/1262) and hosted on (pushupbench.com/)
Read the full abstract ▾
All authors Shengzhi Li, Jiarun Chen, Karun Sharma, Jiaqi Su, Shichao Pei
Computer Vision
Learn&Drop: Fast Learning of CNNs based on Layer Dropping
Giorgio Cruciata, Luca Cruciata, Liliana Lo Presti et al. (+2 more)
This paper proposes a new method to improve the training efficiency of deep convolutional neural networks. During training, the method evaluates scores to measure how much each layer's parameters change and whether the layer will continue learning or not. Based on these scores, the network is scaled down such that the number of parameters to be learned is reduced, yielding a speed up in training. Unlike state-of-the-art methods that try to compress the network to be used in the inference phase or to limit the number of operations performed in the backpropagation phase, the proposed method is novel in that it focuses on reducing the number of operations performed by the network in the forward propagation during training. The proposed training strategy has been validated on two widely used architecture families: VGG and ResNet. Experiments on MNIST, CIFAR-10 and Imagenette show that, with the proposed method, the training time of the models is more than halved without significantly impacting accuracy. The FLOPs reduction in the forward propagation during training ranges from 17.83% for VGG-11 to 83.74% for ResNet-152. These results demonstrate the effectiveness of the proposed technique in speeding up learning of CNNs. The technique will be especially useful in applications where fine-tuning or online training of convolutional models is required, for instance because data arrive sequentially.
Read the full abstract ▾
All authors Giorgio Cruciata, Luca Cruciata, Liliana Lo Presti, Jan Van Gemert, Marco La Cascia
Computer Vision
Breaking the Resource Wall: Geometry-Guided Sequence Modeling for Efficient Semantic Segmentation
Sheng-Wei Chan, Hsin-Jui Pan, Chun-Po Shen et al. (+3 more)
High-performance semantic
segmentation has achieved significant progress in recent years, often driven by increasingly large backbones and higher computational budgets. While effective, such approaches introduce substantial computational overhead and limit accessibility under constrained hardware settings. In this paper, we propose DGM-Net (Directional
Geometric Mamba Network), an efficient architecture that improves modeling capability through structural design rather than increasing model capacity. We introduce Directional
Geometric Mamba (G-Mamba), a linear-complexity O(N) operator as an alternative to conventional context modeling modules such as ASPP and PPM. To further enhance structural awareness in state space model (SSM)-based modeling, we design the DGM-Module, which extracts centripetal flow fields and topological skeletons to guide the scanning process and improve boundary preservation. Without relying on large-scale pretraining or heavy backbone scaling, DGM-Net achieves 80.8% mIoU within 28k iterations, 82.3% mIoU on Cityscapes test set, and 45.24% mIoU on ADE20K. In addition, the model maintains stable performance under constrained hardware settings (e.g., batch size of 2 on 8GB VRAM), highlighting its efficiency and practicality. These results demonstrate that incorporating
geometric guidance into SSM-based architectures provides an effective and resource-efficient direction for semantic
segmentation.
Read the full abstract ▾
All authors Sheng-Wei Chan, Hsin-Jui Pan, Chun-Po Shen, Chia-Min Lin, Yung-Che Wang, Jen-Shiun Chiang
Computer Vision
Keypoint-based Dynamic Object 6-DoF Pose Tracking via Event Camera
Zhe Wang, Qijin Song, Zihao Li et al. (+2 more)
Accurate 6-DoF pose estimation of objects is critical for
robots to perform precise manipulation tasks. However, for dynamic object pose estimation, conventional camera-based approaches face several major challenges, such as motion blur, sensor noise, and low-light limitation. To address these issues, we employ event cameras, whose high dynamic range and low latency offer a promising solution. Furthermore, we propose a keypoint-based detection and tracking approach for dynamic object pose estimation. Firstly, a keypoint detection network is constructed to extract keypoints from the time surface generated by the event stream. Subsequently, the polarity and spatial coordinates of the events are leveraged, and the event density in the vicinity of each keypoint is utilized to achieve continuous keypoint tracking. Finally, a hash mapping is established between the 2D keypoints and the 3D model keypoints, and the EPnP
algorithm is employed to estimate the 6-DoF pose. Experimental results demonstrate that, whether in simulated or real event environments, the proposed method outperforms the event-based state-of-the-art methods in terms of both accuracy and robustness.
Read the full abstract ▾
All authors Zhe Wang, Qijin Song, Zihao Li, Jingyu Xiao, Weibang Bai
Computer Vision · Cross-listed from cs.LG
V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think
Bingda Tang, Yuhui Zhang, Xiaohan Wang et al. (+3 more)
Aligning denoising generative models with human preferences or verifiable rewards remains a key challenge. While policy-gradient online reinforcement learning (RL) offers a principled post-training framework, its direct application is hindered by the intractable likelihoods of these models. Prior work therefore either optimizes an induced Markov decision process (MDP) over sampling trajectories, which is stable but inefficient, or uses likelihood surrogates based on the
diffusion evidence lower bound (ELBO), which have so far underperformed on visual generation. Our key insight is that the ELBO-based approach can, in fact, be made both stable and efficient. By reducing surrogate variance and controlling gradient steps, we show that this approach can beat MDP-based methods. To this end, we introduce Variational GRPO (V-GRPO), a method that integrates ELBO-based surrogates with the Group Relative Policy Optimization (GRPO)
algorithm, alongside a set of simple yet essential techniques. Our method is easy to implement, aligns with pretraining objectives, and avoids the limitations of MDP-based methods. V-GRPO achieves state-of-the-art performance in text-to-image synthesis, while delivering a $2\times$ speedup over MixGRPO and a $3\times$ speedup over DiffusionNFT.
Read the full abstract ▾
All authors Bingda Tang, Yuhui Zhang, Xiaohan Wang, Jiayuan Mao, Ludwig Schmidt, Serena Yeung-Levy
Computer Vision
Hierarchical Spatio-Channel Clustering for Efficient Model Compression in Medical Image Analysis
Sisipho Hamlomo, Marcellin Atemkeng, Habte Tadesse Likassa et al. (+5 more)
Convolutional neural networks (CNNs) have become increasingly difficult to deploy in resource-constrained environments due to their large
memory and computational requirements. Although low-rank compression methods can reduce this burden, most existing approaches compress spatial and channel redundancy independently and therefore do not fully exploit the localised structure within convolutional feature maps. This paper proposes a hierarchical spatio-channel low-rank compression framework for CNNs that exploits redundancy across spatial regions and channel activations. Unlike conventional methods, which apply a uniform decomposition across an entire layer, the proposed approach first partitions feature maps into spatial regions, then groups channels according to their co-activation patterns within each region, and finally applies rank-adaptive SVD to each resulting spatio-channel cluster. The method is evaluated on an AlexNet-based
brain tumour MRI classification model and compared with Global SVD and Tucker decomposition under \(3\times\) and \(6\times\) compression budgets. Our method outperforms both baselines, reducing FLOPs from \(8.21\,\mathrm{G}\) to \(1.55\,\mathrm{G}\) (\(81.1\%\) reduction), achieving a \(1.38\times\) inference speed-up, and increasing classification accuracy from \(87.76\%\) to \(89.80\%\). The method also improves the macro \(F_1\)-score and performance on challenging classes such as meningioma. A hyper-parameter trade-off analysis demonstrates that the framework provides Pareto-optimal configurations, enabling control over the balance between compression and predictive performance. Moderate clustering with adaptive rank selection yields strong results. Bootstrap standard errors are reported for all classification metrics.
Read the full abstract ▾
All authors Sisipho Hamlomo, Marcellin Atemkeng, Habte Tadesse Likassa, Blaise Ravelo, Thierry Bouwmans, Sébastien Lalléchère, Antoine Vacavant, Ding-Geng Chen
Computer Vision · Cross-listed from physics.flu-dyn
Physics-Informed Temporal U-Net for High-Fidelity Fluid Interpolation
Eshwar R. A., Nevin Mathew Thomas, Nehal G et al. (+1 more)
Reconstructing high-fidelity fluid dynamics from sparse temporal observations is quite challenging, mainly due to the chaotic and non-linear nature of fluid transport. Standard deep learning-based interpolation methods often tend to regress to the mean, which results in spatial blurring and temporal strobing, especially noticeable around the observed anchor frames where transitions become discontinuous. In this work, we propose a novel Temporal U-Net architecture that integrates a VGG-based perceptual loss along with a Physics-Informed Bridge to overcome these issues. By introducing time-weighted feature blending and enforcing a
parabolic boundary condition defined by t(1 - t), the model ensures smooth transitions while also maintaining perfect consistency at the endpoints. Experimental results on multi-channel RGB fluid data show that our method clearly outperforms standard models, both in terms of structural fidelity and texture preservation. In particular, the model achieves a Mean Absolute Error of 0.015, compared to 0.085 for a standard L1 baseline. Further Spatial Power Spectral Density (PSD) analysis reveals that the model is able to retain high-frequency turbulent details that are usually lost in deterministic reconstructions.
Read the full abstract ▾
All authors Eshwar R. A., Nevin Mathew Thomas, Nehal G, Farida M. Begam
Computer Vision
He Hu, Tengjin Weng, Zebang Cheng et al. (+5 more)
Recent
multimodal large
language models (MLLMs) have shown strong capabilities in perception,
reasoning, and generation, and are increasingly used in applications such as social
robots and human-computer interaction, where understanding human
emotions is essential. However, existing
benchmarks mainly formulate
emotion understanding as a static recognition problem, leaving it largely unclear whether current MLLMs can understand
emotion as a dynamic process that evolves, shifts between states, and unfolds across diverse social contexts. To bridge this gap, we present EmoTrans, a
benchmark for evaluating
emotion dynamics understanding in
multimodal videos. EmoTrans contains 1,000 carefully collected and manually annotated
video clips, covering 12 real-world scenarios, and further provides over 3,000 task-specific question-answer (QA) pairs for fine-grained evaluation. The
benchmark introduces four tasks, namely
Emotion Change Detection (ECD),
Emotion State Identification (ESI),
Emotion Transition
Reasoning (ETR), and Next
Emotion Prediction (NEP), forming a progressive evaluation framework from coarse-grained detection to deeper
reasoning and prediction. We conduct a comprehensive evaluation of 18 state-of-the-art MLLMs on EmoTrans and obtain two main findings. First, although current MLLMs show relatively stronger performance on coarse-grained
emotion change detection, they still struggle with fine-grained
emotion dynamics modeling. Second, socially complex settings, especially multi-person scenarios, remain substantially challenging, while
reasoning-oriented variants do not consistently yield clear improvements. To facilitate future research, we publicly release the
benchmark, evaluation protocol, and code at https://github.com/Emo-gml/EmoTrans.
Read the full abstract ▾
All authors He Hu, Tengjin Weng, Zebang Cheng, Yu Wang, Jiachen Luo, Björn Schuller, Zheng Lian, Laizhong Cui
Computer Vision
Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detection
Sanghoon Lee, Geon Lee, Hyekang Park et al. (+1 more)
Conventional object detectors typically operate under a closed-set assumption, limiting recognition to a predefined set of base classes seen during training. Open-vocabulary object detection (OVD) addresses this limitation by leveraging
vision-
language models (VLMs) to generate pseudo labels for novel object classes. However, existing OVD methods suffer from two critical drawbacks: (1) inaccurate class label assignments, as VLMs are optimized for image-level predictions rather than the region-level predictions required for pseudo labeling, and (2) unreliable objectness scores from region proposal networks (RPNs) trained exclusively on base object classes. To address these issues, we propose a novel pseudo labeling framework for OVD. Our approach introduces a hierarchical confidence calibration (HCC) technique, which ensures reliable class label estimation by assessing consistency across hierarchical semantic levels (class, super- and sub-category). We also present LoCLIP, a parameter-efficient adaptation of CLIP that incorporates an objectness
token to mitigate base class bias problem of RPNs and provide reliable objectness estimations for novel object classes. Extensive experiments on standard OVD
benchmarks, including COCO and LVIS, demonstrate that our approach clearly sets a new state of the art, validating the effectiveness of our approach. Project site: https://cvlab.yonsei.ac.kr/projects/HCC
Read the full abstract ▾
All authors Sanghoon Lee, Geon Lee, Hyekang Park, Bumsub Ham
Computer Vision
H-SemiS: Hierarchical Fusion of Semi and Self-Supervised Learning for Knee Osteoarthritis Severity Grading
Chandravardhan Singh Raghaw, Anushka Parwal, Shahid Shafi Dar et al. (+2 more)
Knee osteoarthritis (KOA) is a degenerative joint disease that can lead to chronic pain, reduced mobility, and long-term disability. Automated severity grading from knee radiographs can support early assessment, but current methods heavily depend on large labeled
datasets and remain sensitive to class imbalance, noisy samples, and variability in clinical annotations. To alleviate these limitations, we propose a Hierarchical fusion of Semi-Supervised framework with Self-Supervision (H-SemiS) for KOA severity grading in knee X-ray samples using limited annotated data. Rather than treating severity grading as a flat multi-class problem, H-SemiS decomposes the task into a sequence of binary sub-tasks within a semi-supervised teacher-student architecture, directly mitigating the impact of class imbalance. To further enhance feature learning from unlabeled data, the framework integrates an
adversarial self-supervised reconstruction module that encourages the network to capture robust anatomical structures. In parallel, a teacher-student design with
quantum-inspired feature mixing improves discrimination boundaries between adjacent grades when pseudo-labels are noisy. We comprehensively evaluate H-SemiS on two challenging multi-class
datasets and assess its generalizability on two binary-class
datasets. Our experimental results demonstrate the superiority of the proposed H-SemiS framework across multiple evaluation metrics, consistently outperforming several competing baselines and state-of-the-art methods. The code is publicly available at https://github.com/chandravardhan-singh-raghaw/H-SemiS.
Read the full abstract ▾
All authors Chandravardhan Singh Raghaw, Anushka Parwal, Shahid Shafi Dar, Prajakta Darade, Nagendra Kumar
Computer Vision
EAD-Net: Emotion-Aware Talking Head Generation with Spatial Refinement and Temporal Coherence
Yahui Li, Yinfeng Yu, Liejun Wang et al. (+1 more)
Emotionally talking head
video generation aims to generate expressive portrait
videos with accurate lip synchronization and emotional facial expressions. Current methods rely on simple emotional labels, leading to insufficient semantic information. While introducing high-level semantics enhances expressiveness, it easily causes lip-sync degradation. Furthermore, mainstream generation methods struggle to balance computational efficiency and global motion awareness in long
videos and suffer from poor temporal coherence. Therefore, we propose an
Emotion-Aware Diffusion model-based Network, called EAD-Net. We introduce SyncNet supervision and Temporal Representation
Alignment (TREPA) to mitigate lip-sync degradation caused by multi-modal fusion. To model complex spatio-temporal dependencies in long
video sequences, we propose a Spatio-Temporal Directional
Attention (STDA) mechanism that captures global motion patterns through strip
attention. Additionally, we design a Temporal Frame graph
Reasoning Module (TFRM) to explicitly model temporal coherence between
video frames through graph structure learning. To enhance emotional semantic control, a large
language model is employed to extract textual descriptions from real
videos, serving as high-level semantic guidance. Experiments on the HDTF and MEAD
datasets demonstrate that our method outperforms existing methods in terms of lip-sync accuracy, temporal consistency, and emotional accuracy.
Read the full abstract ▾
All authors Yahui Li, Yinfeng Yu, Liejun Wang, Shengjie Shen
Computer Vision
KAConvNet: Kolmogorov-Arnold Convolutional Networks for Vision Recognition
Zhaoxiang Liu, Zhicheng Ma, Kaikai Zhao et al. (+2 more)
The Convolutional Neural Networks (CNNs) have been the dominant and effective approach for general computer
vision tasks. Recently, Kolmogorov-Arnold neural networks (KANs), based on the Kolmogorov-Arnold representation
theorem, have shown potential to replace Multi-Layer Perceptrons (MLPs) in deep learning. KANs, which use learnable nonlinear activations on edges and simple summation on nodes, offer fewer parameters and greater explainability compared to MLPs. However, there has been limited exploration of integrating the Kolmogorov-Arnold representation
theorem with convolutional methods for computer
vision tasks. Existing attempts have merely replaced learnable activation functions with weights, undermining KANs' theoretical foundation and limiting their potential effectiveness. Additionally, the B-spline curves used in KANs suffer from computational inefficiency and a tendency to overfit. In this paper, we propose a novel Kolmogorov-Arnold Convolutional Layer that deeply integrates the Kolmogorov-Arnold representation
theorem with convolution. This layer provides stronger method interpretability because it is based on established mathematical
theorems and its design has theoretical
alignment. Building on the Kolmogorov-Arnold Convolutional Layer, we design an efficient network architecture called KAConvNet, which outperforms existing methods combining KAN and convolution, and achieves competitive performance compared to mainstream ViTs and CNNs. We believe that our work offers valuable insight into the field of artificial intelligence and will inspire the development of more innovative CNNs in the 2020s. The code is publicly available at https://github.com/UnicomAI/KAConvNet.
Read the full abstract ▾
All authors Zhaoxiang Liu, Zhicheng Ma, Kaikai Zhao, Kai Wang, Shiguo Lian
Computer Vision
Jingxuan Kang, Ziqi Zhang, Shaoming Zheng et al. (+9 more)
Segmentation is central to clinical diagnosis and monitoring, yet the reliability of modern foundation models in medical imaging still depends on the availability of precise
prompts. The Segment Anything Model (SAM) offers powerful zero-shot capabilities, although it collapses under the weak, generic, and noisy
prompts that dominate real clinical workflows. In practice, annotations such as centerline points are coarse and ambiguous, often drifting across neighboring anatomy and misguiding SAM toward inconsistent or incomplete masks. We introduce SPD, a Saliency-Guided
Prompt Distillation framework that converts these unreliable cues into robust guidance. SPD first learns data-driven anatomical priors through a lightweight saliency head to obtain confident localization maps. These priors then drive Contextual
Prompt Distillation, which validates and enriches noisy
prompts using cues from anatomically adjacent slices, producing a consensus
prompt set that matches the behavior of expert
reasoning. A Pairwise Slice Consistency objective further enforces local anatomical coherence during
segmentation. Experiments on four challenging MRI and CT
benchmarks demonstrate that SPD consistently outperforms existing SAM adaptations and supervised baselines, delivering large gains in both region-based and boundary-based metrics. SPD provides a practical and principled path toward reliable foundation model deployment in clinical environments where only imperfect
prompts are available.
Read the full abstract ▾
All authors Jingxuan Kang, Ziqi Zhang, Shaoming Zheng, Shuang Li, Uday Bharat Patel, Alexander Harry Fitzhugh, Phillip Lung, Yusuf Kiberu, Nikesh Jathanna, Shahnaz Jamil-Copley, Bernhard Kainz, Chen Qin
Computer Vision
STAND: Semantic Anchoring Constraint with Dual-Granularity Disambiguation for Remote Sensing Image Change Captioning
Yanpei Gong, Beichen Zhang, Hao Wang et al. (+5 more)
Remote sensing image change captioning (RSICC) aims to describe the difference between two remote sensing images. While recent methods have explored
video modeling, they largely overlook the inherent ambiguities in viewpoint, scale, and prior knowledge, lacking effective constraints on the encoder. In this paper, we present STAND, a Semantic Anchoring Constraint with Dual-Granularity Disambiguation for RSICC, to progressively resolve these ambiguities. Specifically, to establish a reliable feature foundation, we first introduce an interpretable constraint to regularize temporal representations. Operating on these purified features, a dual-granularity disambiguation module resolves spatial uncertainties by coupling macro-level global context aggregation for viewpoint confusion with micro-level frequency-refocused
attention for small-object scale enhancement. Ultimately, to translate these visually disambiguated features into precise text, a semantic concept anchoring module leverages
language categorical priors to tackle knowledge ambiguity during decoding. Extensive experiments verify the superiority of STAND and its effectiveness in addressing ambiguities.
Read the full abstract ▾
All authors Yanpei Gong, Beichen Zhang, Hao Wang, Xuhang Fu, Zhaobo Qi, Xinyan Liu, Yuanrong Xu, Weigang Zhang
Computer Vision
MetaErr: Towards Predicting Error Patterns in Deep Neural Networks
Varun Totakura and Shayok Chakraborty
Due to the unprecedented success of deep learning, it has become an integral component in several multimedia computing applications in todays world. Unfortunately, deep learning systems are not perfect and can fail, sometimes abruptly, without prior warning or explanation. While reducing the error rate of deep neural networks has been the primary focus of the multimedia community, the problem of predicting when a deep learning system is going to fail has received significantly less research
attention. In this paper, we propose a simple yet effective framework, MetaErr, to address this under-explored problem in deep learning research. We train a meta-model whose goal is to predict whether a base deep neural network will succeed or fail in predicting a particular data sample, by observing the base models performance on a given learning task. The meta-model is completely agnostic of the architecture and training parameters of the base model. Such an error prediction system can be immensely useful in a variety of smart multimedia applications. Our empirical studies corroborate the promise and potential of our framework against competing baselines. We further demonstrate the usefulness of our framework to improve the performance of pseudo-labeling-based semi-supervised learning, and show that MetaErr outperforms several strong baselines on three
benchmark computer
vision datasets.
Read the full abstract ▾
Computer Vision
Bridging the Pose-Semantic Gap: A Cascade Framework for Text-Based Person Anomaly Search
Zequn Xie, Guijin Luo, Chuxin Wang et al. (+4 more)
Text-based person anomaly search retrieves specific behavioral events from surveillance archives using natural-
language queries. Although recent pose-aware methods align
geometric structures well, they face a fundamental Pose-Semantic Gap: semantically different actions can share similar skeletal geometries. While
Multimodal Large
Language Models (MLLMs) can reduce this ambiguity, using them for large-scale
retrieval is computationally prohibitive. We propose the Structure-Semantic Decoupled Cascade (SSDC) framework, which decouples
retrieval into two stages: (1) Structure-Aware Coarse
Retrieval, where a lightweight model quickly filters candidates by skeletal similarity ; and (2) Detective Squad Interaction, a multi-
agent semantic verification module. The squad consists of a Detective for fast binary filtering, an Analyst for evidence extraction, and a Writer for semantic synthesis. Finally, we re-rank candidates by fusing the synthesized captions with structural priors. Experiments on the PAB
benchmark show that SSDC achieves state-of-the-art performance by balancing efficiency and semantic
reasoning.
Read the full abstract ▾
All authors Zequn Xie, Guijin Luo, Chuxin Wang, Sihang Cai, Tao Jin, Zhou Zhao, Yixuan Tang
Computer Vision · Cross-listed from cs.LG
Contrastive Learning for Multimodal Human Activity Recognition with Limited Labeled Data
Long Jing, Zhixiong Yang, Yajun Zhang et al. (+1 more)
Human activity recognition serves as the foundation for various emerging applications. In recent years, researchers have used collaborative sensing of multi-source sensors to capture complex and dynamic human activities. However,
multimodal human activity sensing typically encounters highly heterogeneous data across modalities and label scarcity, resulting in an application gap between existing solutions and real-world needs. In this paper, we propose CLMM, a general contrastive learning framework for human activity recognition that achieves effective
multimodal recognition with limited labeled data. CLMM employs a novel two-stage training strategy. In the first stage, CLMM employs a CNN-DiffTransformer encoder to capture cross-modal shared information by extracting local and global features. Meanwhile, a hard-positive samples weighting
algorithm enhances gradient propagation to reinforce shared learning. In the second stage, a dual-branch architecture combining quality-guided
attention and bidirectional gated units captures modality-specific information, while a primary-auxiliary collaborative training strategy fuses both shared and modality-specific information. Experimental results on three public
datasets demonstrate that CLMM significantly improves state-of-the-art baselines in both recognition accuracy and convergence performance.
Read the full abstract ▾
All authors Long Jing, Zhixiong Yang, Yajun Zhang, Xinlong Feng
Computer Vision
Lightweight and Production-Ready PDF Visual Element Parsing
Meizhu Liu, Yassi Abbasi, Matthew Rowe et al. (+2 more)
PDF documents contain critical visual elements such as figures, tables, and forms whose accurate extraction is essential for document understanding and
multimodal retrieval-augmented generation (RAG). Existing PDF parsers often miss complex visuals, extract non-informative artifacts (e.g., watermarks, logos), produce fragmented elements, and fail to reliably associate captions with their corresponding elements, which degrades downstream
retrieval and question answering. We present a lightweight and production level PDF parsing framework that can accurately detect visual elements and associates captions using a combination of spatial heuristics, layout analysis, and semantic similarity. On popular
benchmark datasets and internal product data, the proposed solution achieves $\geq96\%$ visual element detection accuracy and $93\%$ caption association accuracy. When used as a preprocessing step for
multimodal RAG, it significantly outperforms state-of-the-art parsers and large
vision-
language models on both internal data and the MMDocRAG
benchmark, while reducing latency by over $2\times$. We have deployed the proposed system in challenging production environment.
Read the full abstract ▾
All authors Meizhu Liu, Yassi Abbasi, Matthew Rowe, Michael Avendi, Paul Li
Computer Vision
SemiGDA: Generative Dual-distribution Alignment for Semi-Supervised Medical Image Segmentation
Kaiwen Huang, Yi Zhou, Yizhe Zhang et al. (+2 more)
Semi-supervised learning addresses label scarcity and high annotation costs in medical image
segmentation by exploiting the latent information in unlabeled data to enhance model performance. Traditional discriminative
segmentation relies on
segmentation masks, neglecting feature-level distribution constraints. This limits robust semantic representation learning and adaptive modeling of unlabeled data in scenarios with few labels. To address these limitations, we propose SemiGDA, a novel Generative Dual-distribution
Alignment framework for semi-supervised medical image
segmentation. Our SemiGDA overcomes the reliance of discriminative methods on large labeled
datasets by aligning feature and semantic distributions to boost semantic learning and scene adaptability. Specifically, we propose a Dual-distribution
Alignment Module (DAM), which employs two structurally distinct encoders to model image and mask feature distributions. It enforces their
alignment in the latent space via distributional constraints, establishing structured feature consistency. Moreover, we design a Consistency-Driven Skip Adapter (CDSA) strategy, which introduces dual skip adapters (Image and Mask) to fuse multi-scale features via skip connections. Using a consistency loss, CDSA enhances cross-branch semantic
alignment and reinforces fine-grained semantic consistency. Experimental results on diverse medical
datasets show that our method outperforms other state-of-the-art semi-supervised
segmentation methods. Code is released at: https://github.com/taozh2017/SemiGDA.
Read the full abstract ▾
All authors Kaiwen Huang, Yi Zhou, Yizhe Zhang, Jingxiong Li, Tao Zhou
Computer Vision
A Hierarchical Ensemble Inference Pipeline for Robust White Blood Cell Classification Under Domain Shifts
Ruyi Dai, Tingkwong Ng and Hao Chen
Automated white blood
cell (WBC) classification is essential for scalable leukaemia screening. However, real-world deployment is challenged by domain shifts caused by staining protocols, scanner characteristics, and inter-laboratory variability, which often degrade model performance. The White Blood
Cell Classification Challenge (WBCBench) at ISBI 2026 aims to advance robust WBC recognition, with a focus on accurately identifying blast
cells and other clinically critical rare subtypes. We propose a
memory-augmented, hierarchical ensemble pipeline for WBC classification under domain shifts, leveraging a feature bank and a DinoBloom backbone fine-tuned with LoRA. Our three-stage inference hierarchy combines k-nearest neighbors (kNN)
retrieval at each level, reducing over-reliance on any single decision. Evaluated on the WBCBench
dataset, our method ranks within the top ten by macro F1-score in the final testing phase.
Read the full abstract ▾
Computer Vision
LatentBurst: A Fast and Efficient Multi Frame Super-Resolution for Hexadeca-Bayer Pattern CIS images
Sangwook Baek, Vin Van Duong, Karam Park et al. (+1 more)
This paper introduces a novel multi frame super-resolution network (MFSR) for burst hexadeca Bayer pattern Contact Image Sensor (CIS) images, which includes demosaicing, denoising, multi-frame fusion, and super-resolution. Designing a high-quality reconstruction network poses several challenges as follows: 1) Unlike the Bayer color filter array (CFA) pattern, it is hard to interpolate hexadeca-Bayer pattern since the pixel distance between the same color groups increases; 2) Due to large object motion and camera movements, the final fusion result usually suffers the misalignment resulting a blurry image or ghosting artifacts; 3) The proposed network should be fast and efficient enough to operate in real-time on mobile devices. To overcome these challenges, we propose a novel network, called LatentBurst, which contains: 1) a pyramid align and fusion approach in latent feature to deal with large motion scenario; 2) an efficient UNet-based structure which can run efficiently on mobile device; 3) fine-tuned optical flow estimation and two-step knowledge
distillation to reduce domain-gap more effectively. Experimental results in various scenarios demonstrate the effectiveness of our proposed method compared with other state-of-the-art methods.
Read the full abstract ▾
All authors Sangwook Baek, Vin Van Duong, Karam Park, Pilkyu Park
Computer Vision
MotionHiFlow: Text-to-motion via hierarchical flow matching
Heng Li, Xiaotong Lin, Ling-An Zeng et al. (+3 more)
Text-to-motion generation aims to generate 3D human motions that are tightly aligned with the input text while remaining physically plausible and rich in fine-grained detail. Although recent approaches can produce complex and natural movements, they usually operate at only one temporal scale, which limits both semantic
alignment and temporal coherence. Inspired by the fact that complex motions are conceptualized hierarchically rather than at a single temporal scale in the human cognitive system, we propose MotionHiFlow, a hierarchical flow matching framework to generate motion progressively by constructing flow path from low to high temporal scales. The flows at lower scales capture high-level semantics and coarse motion structures, while flows at higher scales refine temporal details. To link the flows across scales, we introduce a novel cross-scale transition process, ensuring continuity and preserving noise consistency. Furthermore, by integrating a Text-Motion
Diffusion Transformer and a
topology-aware Motion VAE, MotionHiFlow explicitly models structural dependencies among joints via joint-aware positional encoding and skeletal
topology, enabling precise semantic
alignment alongside fine-grained motion details. Extensive experiments on HumanML3D and KIT-ML
benchmarks demonstrate state-of-the-art performance, with ablation studies confirming the effectiveness of the hierarchical design and key components. Code is available at https://github.com/ai-lh/MotionHiFlow.
Read the full abstract ▾
All authors Heng Li, Xiaotong Lin, Ling-An Zeng, Yulei Kang, Shuai Li, Jian-Fang Hu
Computer Vision
Micro-Expression-Aware Avatar Fingerprinting via Inter-Frame Feature Differencing
Masoumeh Chapariniya, Jean-Marc Odobez, Volker Dellwo et al. (+1 more)
Avatar fingerprinting, i.e., verifying who drives a synthetic talking-head
video rather than whether it is real, is a critical safeguard for authorized use of face-reenactment technology. Existing methods rely on a fixed, non-differentiable landmark extraction stage that prevents the fingerprinting model from being optimized end-to-end from raw pixels. We propose a preprocessing-free system built on a micro-expression-aware backbone operating on raw
video frames, with inter-frame feature differencing as the core design principle: consecutive feature maps are subtracted in the learned deep feature space, so that temporally stable appearance dimensions contribute zero to the output while driver-specific motion dynamics are preserved. A controlled ablation on NVFAIR confirms that temporal motion accounts for the large majority of discriminative performance, and that raw appearance features actively degrade identity separation. Both the choice of backbone and the differencing principle are essential: differencing alone is insufficient when applied to a generic encoder, as appearance-dominated features collapse to near-identical representations across adjacent frames, while the micro-expression-aware F5C backbone retains measurable motion variation that the differencing operation can exploit. Without any external preprocessing, our model achieves an overall AUC of 0.877 on NVFAIR and matches or exceeds the landmark-based baseline on the majority of cross-generator pairs.
Read the full abstract ▾
All authors Masoumeh Chapariniya, Jean-Marc Odobez, Volker Dellwo, Teodora Vuković
Computer Vision
AnalogRetriever: Learning Cross-Modal Representations for Analog Circuit Retrieval
Yihan Wang, Lei Li, Yao Lai et al. (+2 more)
Analog
circuit design relies heavily on reusing existing intellectual property (IP), yet searching across heterogeneous representations such as SPICE netlists, schematics, and functional descriptions remains challenging. Existing methods are largely limited to exact matching within a single modality, failing to capture cross-modal semantic relationships. To bridge this gap, we present AnalogRetriever, a unified tri-modal
retrieval framework for analog
circuit search. We first build a high-quality
dataset on top of Masala-CHAI through a two-stage repair pipeline that raises the netlist compile rate from 22% to 100%. Built on this foundation, AnalogRetriever encodes schematics and descriptions with a
vision-
language model and netlists with a port-aware relational graph convolutional network, mapping all three modalities into a shared
embedding space via curriculum contrastive learning. Experiments show that AnalogRetriever achieves an average Recall@1 of 75.2% across all six cross-modal
retrieval directions, significantly outperforming existing baselines. When integrated into the AnalogCoder agentic framework as a
retrieval-augmented generation module, it consistently improves functional pass rates and enables previously unsolved tasks to be completed. Our code and
dataset will be released.
Read the full abstract ▾
All authors Yihan Wang, Lei Li, Yao Lai, Jing Wang, Yan Lu
Computer Vision
DyABD: The Abdominal Muscle Segmentation in Dynamic MRI Benchmark
Niamh Belton, Victoria Joppin, Aonghus Lawlor et al. (+4 more)
This work introduces DyABD, a novel and complex
benchmark dataset of dynamic abdominal MRIs from patients with abdominal hernias and associated high quality abdominal muscle annotations. DyABD is the first-of-its-kind in four key ways; (1) it proposes the first abdominal muscle
segmentation task, (2) the dynamic MRIs are acquired whilst the patients perform various exercises, introducing extreme anatomical variability, making it one of the most challenging
segmentation datasets to date, (3) it includes both pre and post corrective MRIs and (4) DyABD promotes clinical research into the high recurrence rates of abdominal hernias. Beyond
dataset introduction, this work provides a comprehensive evaluation of the generalisation capabilities of existing
segmentation models across Supervised, Few Shot and Zero Shot paradigms on the unseen DyABD
dataset. This work reveals that there is still room for substantial improvement in the field of medical image
segmentation, with the majority of techniques achieving a Dice Coefficient of 0.82. This work therefore sheds light on the true progress of the field and redefines the
benchmark for progress in medical image
segmentation.
Read the full abstract ▾
All authors Niamh Belton, Victoria Joppin, Aonghus Lawlor, Catherine Masson, Thierry Bege, David Bendahan, Kathleen M. Curran
Computer Vision
One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognition
Balaji Darur, Amanmeet Garg and Makarand Tapaswi
Video Situation Recognition (VidSitu) addresses the challenging problem of "who did what to whom, with what, how, and where" in a
video. It tests thorough
video understanding by requiring identification of salient actions and associated short descriptions for event roles across multiple events. Grounding with VidSitu requires spatio-temporal localization of key entities across shots and varied appearances. We posit that coherent
video understanding requires consistent identification of entities that play different roles. We propose
Multimodal Entity Coreference (MEC) to unite entity descriptions in text with grounding across the
video. Towards this, we introduce CineMEC, a multi-stage approach that unites event role mention groups with visual clusters of entities, without explicit grounding supervision during training. Our approach is designed to exploit the synergy between visual grounding and captioning, where improving one influences the other and vice versa. For evaluation, we extend the VidSitu
dataset with grounding annotations. While previous work focuses primarily on descriptions, CineMEC improves consistency across both: captioning (+2.5% CIDEr, +7% LEA) and visual grounding (+18% HOTA).
Read the full abstract ▾
Computer Vision
A Topology fixated Shape Gradient Framework for Non Simple Boundary Extraction for CIE Lab color images with Repulsive Energy
Shafeequdheen Palengara, Jyotiranjan Nayak and Vijayakrishna Rowthu
A levelset free but a hybrid image
segmentation approach based on a modified version of the piece wise constant shape gradient of an Mumford Shah shape functional and a repulsive function is considered. The
segmentation is performed a non-local shape based through an
evolution of discrete curves driven by a non local shape based energy to segment images containing disjoint regions and multiple boundaries. This formulation has a novel additional component as a multivariable function dependent on a few sampled points of the curves that handles the occurrence of self intersection during boundary curves
evolution. The method is applied to a few gray scale and color images, including images with nested structures and astronomical objects. The results indicate effective
segmentation in complex scenarios with absolute control on the
topology of the segments and self-intersections of the boundaries
Read the full abstract ▾