Computer Vision
A Two-Stage Dual-Modality Model for Facial Emotional Expression Recognition
Jiajun Sun and Zhe Gao
This paper addresses the expression (EXPR) recognition challenge in the 10th Affective Behavior Analysis in-the-Wild (ABAW) workshop and competition, which requires frame-level classification of eight facial emotional expressions from unconstrained
videos. This task is challenging due to inaccurate face localization, large pose and scale variations, motion blur, temporal instability, and other confounding factors across adjacent frames. We propose a two-stage dual-modal (audio-visual) model to address these difficulties. Stage I focuses on robust visual feature extraction with a pretrained DINOv2-based encoder. Specifically, DINOv2 ViT-L/14 is used as the backbone, a padding-aware augmentation (PadAug) strategy is employed for image padding and data preprocessing from raw
videos, and a mixture-of-experts (MoE) training head is introduced to enhance classifier diversity. Stage II addresses modality fusion and temporal consistency. For the visual modality, faces are re-cropped from raw
videos at multiple scales, and the extracted visual features are averaged to form a robust frame-level representation. Concurrently, frame-aligned Wav2Vec 2.0 audio features are derived from short audio windows to provide complementary acoustic cues. These dual-modal features are integrated via a lightweight gated fusion module, followed by inference-time temporal smoothing. Experiments on the ABAW
dataset demonstrate the effectiveness of the proposed method. The two-stage model achieves a Macro-F1 score of 0.5368 on the official validation set and 0.5122 +/- 0.0277 under 5-fold cross-validation, outperforming the official baselines.
Read the full abstract ▾
Computer Vision
Real-World Point Tracking with Verifier-Guided Pseudo-Labeling
Görkay Aydemir, Fatma Güney and Weidi Xie
Models for long-term point tracking are typically trained on large synthetic
datasets. The performance of these models degrades in real-world
videos due to different characteristics and the absence of dense ground-truth annotations. Self-training on unlabeled
videos has been explored as a practical solution, but the quality of pseudo-labels strongly depends on the reliability of teacher models, which vary across frames and scenes. In this paper, we address the problem of real-world fine-tuning and introduce verifier, a meta-model that learns to assess the reliability of tracker predictions and guide pseudo-label generation. Given candidate trajectories from multiple pretrained trackers, the verifier evaluates them per frame and selects the most trustworthy predictions, resulting in high-quality pseudo-label trajectories. When applied for fine-tuning, verifier-guided pseudo-labeling substantially improves the quality of supervision and enables data-efficient adaptation to unlabeled
videos. Extensive experiments on four real-world
benchmarks demonstrate that our approach achieves state-of-the-art results while requiring less data than prior self-training methods. Project page: https://kuis-ai.github.io/track_on_r
Read the full abstract ▾
Computer Vision
RDNet: Region Proportion-Aware Dynamic Adaptive Salient Object Detection Network in Optical Remote Sensing Images
Bin Wan, Runmin Cong, Xiaofei Zhou et al. (+3 more)
Salient object detection (SOD) in remote sensing images faces significant challenges due to large variations in object sizes, the computational cost of self-
attention mechanisms, and the limitations of CNN-based extractors in capturing global context and long-range dependencies. Existing methods that rely on fixed convolution kernels often struggle to adapt to diverse object scales, leading to detail loss or irrelevant feature aggregation. To address these issues, this work aims to enhance robustness to scale variations and achieve precise object localization. We propose the Region Proportion-Aware Dynamic Adaptive Salient Object Detection Network (RDNet), which replaces the CNN backbone with the SwinTransformer for global context modeling and introduces three key modules: (1) the Dynamic Adaptive Detail-aware (DAD) module, which applies varied convolution kernels guided by object region proportions; (2) the Frequency-matching Context Enhancement (FCE) module, which enriches contextual information through wavelet interactions and
attention; and (3) the Region Proportion-aware Localization (RPL) module, which employs cross-
attention to highlight semantic details and integrates a Proportion Guidance (PG) block to assist the DAD module. By combining these modules, RDNet achieves robustness against scale variations and accurate localization, delivering superior detection performance compared with state-of-the-art methods.
Read the full abstract ▾
All authors Bin Wan, Runmin Cong, Xiaofei Zhou, Hao Fang, Yaoqi Sun, Sam Kwong
Computer Vision
ForensicZip: More Tokens are Better but Not Necessary in Forensic Vision-Language Models
Yingxin Lai, Zitong Yu, Jun Wang et al. (+3 more)
Multimodal Large
Language Models (MLLMs) enable interpretable multimedia forensics by generating textual rationales for forgery detection. However, processing dense visual sequences incurs high computational costs, particularly for high-resolution images and
videos. Visual
token pruning is a practical acceleration strategy, yet existing methods are largely semantic-driven, retaining salient objects while discarding background regions where manipulation traces such as high-frequency anomalies and temporal jitters often reside. To address this issue, we introduce ForensicZip, a training-free framework that reformulates
token compression from a forgery-driven perspective. ForensicZip models temporal
token evolution as a Birth-Death Optimal Transport problem with a slack dummy node, quantifying physical discontinuities indicating transient generative artifacts. The forensic scoring further integrates transport-based novelty with high-frequency priors to separate forensic evidence from semantic content under large-ratio compression. Experiments on deepfake and AIGC
benchmarks show that at 10%
token retention, ForensicZip achieves $2.97\times$ speedup and over 90% FLOPs reduction while maintaining state-of-the-art detection performance.
Read the full abstract ▾
All authors Yingxin Lai, Zitong Yu, Jun Wang, Linlin Shen, Yong Xu, Xiaochun Cao
Computer Vision · Cross-listed from cs.RO
SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Robotics
Mengzhen Liu, Enshen Zhou, Cheng Chi et al. (+6 more)
Active perception and manipulation are crucial for
robots to interact with complex scenes. Existing methods struggle to unify semantic-driven active perception with robust, viewpoint-invariant execution. We propose SaPaVe, an end-to-end framework that jointly learns these capabilities in a data-efficient manner. Our approach decouples camera and manipulation actions rather than placing them in a shared action space, and follows a bottom-up training strategy: we first train semantic camera control on a large-scale
dataset, then jointly optimize both action types using hybrid data. To support this framework, we introduce ActiveViewPose-200K, a
dataset of 200k image-
language-camera movement pairs for semantic camera movement learning, and a 3D geometry-aware module that improves execution robustness under dynamic viewpoints. We also present ActiveManip-Bench, the first
benchmark for evaluating active manipulation beyond fixed-view settings. Extensive experiments in both simulation and real-world environments show that SaPaVe outperforms recent
vision-
language-action models such as GR00T N1 and \(\pi_0\), achieving up to 31.25% higher success rates in real-world tasks. These results show that tightly coupled perception and execution, when trained with decoupled yet coordinated strategies, enable efficient and generalizable active manipulation. Project page: https://lmzpai.github.io/SaPaVe
Read the full abstract ▾
All authors Mengzhen Liu, Enshen Zhou, Cheng Chi, Yi Han, Shanyu Rong, Liming Chen, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang
Computer Vision
BehaviorVLM: Unified Finetuning-Free Behavioral Understanding with Vision-Language Reasoning
Jingyang Ke, Weihan Li, Amartya Pradhan et al. (+2 more)
Understanding freely moving animal behavior is central to neuroscience, where pose estimation and behavioral understanding form the foundation for linking neural activity to natural actions. Yet both tasks still depend heavily on human annotation or unstable unsupervised pipelines, limiting scalability and reproducibility. We present BehaviorVLM, a unified
vision-
language framework for pose estimation and behavioral understanding that requires no task-specific finetuning and minimal human labeling by guiding pretrained
Vision-
Language Models (VLMs) through detailed, explicit, and verifiable
reasoning steps. For pose estimation, we leverage
quantum-dot-grounded behavioral data and propose a multi-stage pipeline that integrates temporal, spatial, and cross-view
reasoning. This design greatly reduces human annotation effort, exposes low-confidence labels through
geometric checks such as reprojection error, and produces labels that can later be filtered, corrected, or used to fine-tune downstream pose models. For behavioral understanding, we propose a pipeline that integrates deep embedded clustering for over-segmented behavior discovery, VLM-based per-clip
video captioning, and LLM-based
reasoning to merge and semantically label behavioral segments. The behavioral pipeline can operate directly from visual information and does not require keypoints to segment behavior. Together, these components enable scalable, interpretable, and label-light analysis of multi-animal behavior.
Read the full abstract ▾
All authors Jingyang Ke, Weihan Li, Amartya Pradhan, Jeffrey Markowitz, Anqi Wu
Computer Vision
LatentGeo: Learnable Auxiliary Constructions in Latent Space for Multimodal Geometric Reasoning
Haiying Xu, Zihan Wang, Song Dai et al. (+3 more)
Despite recent advances in
multimodal reasoning, representing auxiliary
geometric constructions remains a fundamental challenge for
multimodal large
language models (MLLMs). Such constructions are absent from the original diagram and must be introduced before
theorems apply. Existing approaches predominantly rely on explicit construction paradigms, including text-based
geometric specification, visual-
token interleaving during
reasoning, and tool-augmented
geometric execution. However, these methods either fail to faithfully represent complex spatial relationships, incur representation mismatch between discrete symbols and continuous
geometric structures, or rely on external capabilities that hinder end-to-end optimization. To address these limitations, we propose LatentGeo, a framework that learns continuous latent visual representations to internalize auxiliary
geometric constructions without pixel-level rendering or external executors. We design a three-stage curriculum that progressively aligns and internalizes these latent representations through auxiliary visual supervision, followed by LaGDPO, a latent-aware reinforcement learning procedure that stabilizes latent representations during policy optimization while improving end-task correctness. To systematically evaluate construction-centric representation quality, we introduce GeoAux, a new
benchmark targeting visually dependent geometry problems, and conduct experiments on GeoAux and MathVerse. Results show that LatentGeo achieves substantial gains on
geometric reasoning tasks, particularly those requiring auxiliary constructions. Extensive analyses and ablation studies further validate the effectiveness of each component in our framework.
Read the full abstract ▾
All authors Haiying Xu, Zihan Wang, Song Dai, Zhengxuan Zhang, Kairan Dou, Xuming Hu
Computer Vision
GlyphBanana: Advancing Precise Text Rendering Through Agentic Workflows
Zexuan Yan, Jiarui Jin, Yue Ma et al. (+5 more)
Despite recent advances in generative models driving significant progress in text rendering, accurately generating complex text and mathematical formulas remains a formidable challenge. This difficulty primarily stems from the limited instruction-following capabilities of current models when encountering out-of-distribution
prompts. To address this, we introduce GlyphBanana, alongside a corresponding
benchmark specifically designed for rendering complex characters and formulas. GlyphBanana employs an agentic workflow that integrates auxiliary tools to inject glyph templates into both the latent space and
attention maps, facilitating the iterative refinement of generated images. Notably, our training-free approach can be seamlessly applied to various Text-to-Image (T2I) models, achieving superior precision compared to existing baselines. Extensive experiments demonstrate the effectiveness of our proposed workflow. Associated code is publicly available at https://github.com/yuriYanZeXuan/GlyphBanana.
Read the full abstract ▾
All authors Zexuan Yan, Jiarui Jin, Yue Ma, Shijian Wang, Jiahui Hu, Wenxiang Jiao, Yuan Lu, Linfeng Zhang
Computer Vision
Linking Perception, Confidence and Accuracy in MLLMs
Yuetian Du, Yucheng Wang, Rongyu Zhang et al. (+5 more)
Recent advances in Multi-modal Large
Language Models (MLLMs) have predominantly focused on enhancing visual perception to improve accuracy. However, a critical question remains unexplored: Do models know when they do not know? Through a probing experiment, we reveal a severe confidence miscalibration problem in MLLMs. To address this, we propose Confidence-Driven Reinforcement Learning (CDRL), which uses original-noise image pairs and a novel confidence-based reward to enhance perceptual sensitivity and robustly calibrate the model's confidence. Beyond training benefits, calibrated confidence enables more effective test-time scaling as a free lunch. We further propose Confidence-Aware Test-Time Scaling (CA-TTS), which dynamically coordinates Self-Consistency, Self-Reflection, and Visual Self-Check modules guided by confidence signals. An Expert Model acts in multiple roles (e.g., Planner, Critic, Voter) to schedule these modules and provide external verification. Our integrated framework establishes new state-of-the-art results with consistent 8.8% gains across four
benchmarks. More ablation studies demonstrate the effectiveness of each module and scaling superiority.
Read the full abstract ▾
All authors Yuetian Du, Yucheng Wang, Rongyu Zhang, Zhijie Xu, Boyu Yang, Ming Kong, Jie Liu, Qiang Zhu
Computer Vision
EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Next
Ye Pan, Chi Kit Wong, Yuanhuiyi Lyu et al. (+8 more)
Egocentric video provides a natural modality for studying human behavior, but conventional visual understanding captures mainly observable scenes, objects, and actions rather than the latent goals that organize them. Existing intent
benchmarks typically focus on coarse event-level goals and overlook how intent evolves across procedural steps. We introduce EgoIntent, a step-level intent-understanding
benchmark comprising 3,014 pre-outcome micro-steps from 32
egocentric videos across 15 indoor and outdoor daily-life scenarios. Each step is manually annotated along three complementary dimensions: Local Intent (What), the immediate goal; Procedural Intent (Why), the role of the step in the broader procedure; and Next-Plan (Next), the action most likely to follow. Multiple rounds of human review refine temporal boundaries and annotation quality. We evaluate 15
multimodal large
language models using reference-based scores and complementary reference-free diagnostics. Controlled studies on four representative models show that only one model gains significantly from correct temporal order, while a single boundary frame outperforms the full ordered clip for three models. Step-only input performs best for all four models, and adding 15 seconds of history significantly degrades three. Revealing the current outcome improves Local Intent by 7.81 points, while revealing the following step improves Next-Plan by 13.17 points. These findings indicate that current models can achieve strong intent-prediction scores through static boundary cues without robustly exploiting temporal order or procedural history.
Read the full abstract ▾
All authors Ye Pan, Chi Kit Wong, Yuanhuiyi Lyu, Hanqian Li, Chenfei Liao, Jiahao Huo, Lutao Jiang, Zixin Zhang, Jiacheng Chen, Yuqian Fu, Xu Zheng
Computer Vision
FlashMotion: Few-Step Controllable Video Generation with Trajectory Guidance
Quanhao Li, Zhen Xing, Rui Wang et al. (+4 more)
Recent advances in trajectory-controllable
video generation have achieved remarkable progress. Previous methods mainly use adapter-based architectures for precise motion control along predefined trajectories. However, all these methods rely on a multi-step denoising process, leading to substantial time redundancy and computational overhead. While existing
video distillation methods successfully distill multi-step generators into few-step, directly applying these approaches to trajectory-controllable
video generation results in noticeable degradation in both
video quality and trajectory accuracy. To bridge this gap, we introduce FlashMotion, a novel training framework designed for few-step trajectory-controllable
video generation. We first train a trajectory adapter on a multi-step
video generator for precise trajectory control. Then, we distill the generator into a few-step version to accelerate
video generation. Finally, we finetune the adapter using a hybrid strategy that combines
diffusion and
adversarial objectives, aligning it with the few-step generator to produce high-quality, trajectory-accurate
videos. For evaluation, we introduce FlashBench, a
benchmark for long-sequence trajectory-controllable
video generation that measures both
video quality and trajectory accuracy across varying numbers of foreground objects. Experiments on two adapter architectures show that FlashMotion surpasses existing
video distillation methods and previous multi-step models in both visual quality and trajectory consistency.
Read the full abstract ▾
All authors Quanhao Li, Zhen Xing, Rui Wang, Haidong Cao, Qi Dai, Daoguo Dong, Zuxuan Wu
Computer Vision
O3N: Omnidirectional Open-Vocabulary Occupancy Prediction for Embodied Intelligent Robotics
Mengfei Duan, Hao Shi, Fei Teng et al. (+4 more)
The rapid
evolution of consumer electronics toward
embodied intelligence has accelerated the emergence of Consumer
Embodied Intelligent Robotics (CEIRs), where intelligent devices are expected to perceive, understand, and interact with complex real-world environments. Understanding and reconstructing the 3D world through omnidirectional perception is therefore becoming increasingly important for CEIRs operating in complex and dynamic environments. However, existing
vision-based 3D occupancy prediction methods are constrained by limited perspective inputs and a predefined training distribution, making them difficult to support
embodied intelligent systems that require comprehensive and safe perception of scenes in open-world exploration. To address this, we present O3N, the first framework for open-vocabulary occupancy prediction from a single omnidirectional RGB image. O3N embeds omnidirectional voxels in a polar-spiral
topology via the Polar-spiral Mamba (PsM) module, enabling continuous spatial representation and long-range context modeling across 360{\deg}. The Occupancy Cost Aggregation (OCA) module introduces a principled mechanism for unifying
geometric and semantic supervision within the voxel space, ensuring consistency between reconstructed geometry and underlying semantic structure. Moreover, Natural Modality
Alignment (NMA) establishes a gradient-free
alignment pathway that harmonizes visual features, voxel
embeddings, and text semantics, forming a consistent pixel-voxel-text representation triad for open-world perception. Extensive experiments on multiple models demonstrate that our method not only achieves state-of-the-art performance on QuadOcc and Human360Occ
benchmarks but also exhibits remarkable cross-scene generalization and semantic scalability. The source code will be made publicly available at https://github.com/MengfeiD/O3N.
Read the full abstract ▾
All authors Mengfei Duan, Hao Shi, Fei Teng, Guoqiang Zhao, Yuheng Zhang, Zhiyong Li, Kailun Yang
Computer Vision
HATS: Hardness-Aware Trajectory Synthesis for GUI Agents
Rui Shao, Ruize Gao, Bin Xie et al. (+5 more)
Graphical user interface (GUI)
agents powered by large
vision-
language models (VLMs) have shown remarkable potential in automating digital tasks, highlighting the need for high-quality trajectory data to support effective
agent training. Yet existing trajectory synthesis pipelines often yield
agents that fail to generalize beyond simple interactions. We identify this limitation as stemming from the neglect of semantically ambiguous actions, whose meanings are context-dependent, sequentially dependent, or visually ambiguous. Such actions are crucial for real-world robustness but are under-represented and poorly processed in current
datasets, leading to semantic misalignment between task instructions and execution. To address these issues, we propose HATS, a Hardness-Aware Trajectory Synthesis framework designed to mitigate the impact of semantic ambiguity. We define hardness as the degree of semantic ambiguity associated with an action and develop two complementary modules: (1) hardness-driven exploration, which guides data collection toward ambiguous yet informative interactions, and (2)
alignment-guided refinement, which iteratively validates and repairs instruction-execution
alignment. The two modules operate in a closed loop: exploration supplies refinement with challenging trajectories, while refinement feedback updates the hardness signal to guide future exploration. Extensive experiments show that
agents trained with HATS consistently outperform state-of-the-art baselines across
benchmark GUI environments.
Read the full abstract ▾
All authors Rui Shao, Ruize Gao, Bin Xie, Yixing Li, Kaiwen Zhou, Shuai Wang, Weili Guan, Gongwei Chen
Computer Vision
Hoi3DGen: Generating High-Quality Human-Object-Interactions in 3D
Agniv Sharma, Xianghui Xie, Tom Fischer et al. (+2 more)
Modeling and generating 3D human-object interactions from text is crucial for applications in AR, XR, and gaming. Existing approaches often rely on score
distillation from text-to-image models, but their results suffer from the Janus problem and do not follow text
prompts faithfully due to the scarcity of high-quality interaction data. We introduce Hoi3DGen, a framework that generates high-quality textured meshes of human-object interaction that follow the input interaction descriptions precisely. We first curate realistic and high-quality interaction data leveraging
multimodal large
language models, and then create a full text-to-3D pipeline, which achieves orders-of-magnitude improvements in interaction fidelity. Our method surpasses baselines by 4-15x in text consistency and 3-7x in 3D model quality, exhibiting strong generalization to diverse categories and interaction types, while maintaining high-quality 3D generation.
Read the full abstract ▾
All authors Agniv Sharma, Xianghui Xie, Tom Fischer, Eddy Ilg, Gerard Pons-Moll
Computer Vision · Cross-listed from cs.RO
CRAFT: A Tendon-Driven Hand with Hybrid Hard-Soft Compliance
Leo Lin, Shivansh Patel, Jay Moon et al. (+2 more)
We introduce CRAFT hand, a tendon-driven anthropomorphic hand with hybrid hard-soft compliance for contact-rich manipulation. The design is based on a simple idea: contact is not uniform across the hand. Impacts concentrate at joints, while links carry most of the load. CRAFT places soft material at joints and keeps links rigid, and uses rollingcontact joint surfaces to keep flexion on repeatable motion paths. Fifteen motors mounted on the fingers drive the hand through tendons, keeping the form factor compact and the fingers light. In structural tests, CRAFT improves strength and endurance while maintaining comparable repeatability. In teleoperation, CRAFT improves handling of fragile and low-friction items, and the hand covers 33/33 grasps in the Feix taxonomy. The full design costs under $600 and will be released open-source with visionbased teleoperation and simulation integration. Project page: http://craft-hand.github.io/
Read the full abstract ▾
All authors Leo Lin, Shivansh Patel, Jay Moon, Svetlana Lazebnik, Unnat Jain
Computer Vision
EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generation
Yan Li, Ning Liao, Xiangyu Zhao et al. (+5 more)
The development of unified
multimodal large
language models (MLLMs) is fundamentally challenged by the granularity gap between visual understanding and generation: understanding requires high-level semantic abstractions, while image generation demands fine-grained pixel-level representations. Existing approaches usually enforce the two supervision on the same set of representation or decouple these two supervision on separate feature spaces, leading to interference and inconsistency, respectively. In this work, we propose EvoTok, a unified image tokenizer that reconciles these requirements through a residual
evolution process within a shared latent space. Instead of maintaining separate
token spaces for pixels and semantics, EvoTok encodes an image into a cascaded sequence of residual
tokens via residual vector quantization. This residual sequence forms an
evolution trajectory where earlier stages capture low-level details and deeper stages progressively transition toward high-level semantic representations. Despite being trained on a relatively modest
dataset of 13M images, far smaller than the billion-scale
datasets used by many previous unified tokenizers, EvoTok achieves a strong reconstruction quality of 0.43 rFID on ImageNet-1K at 256x256 resolution. When integrated with a large
language model, EvoTok shows promising performance across 7 out of 9 visual understanding
benchmarks, and remarkable results on image generation
benchmarks such as GenEval and GenAI-Bench. These results demonstrate that modeling visual representations as an evolving trajectory provides an effective and principled solution for unifying visual understanding and generation.
Read the full abstract ▾
All authors Yan Li, Ning Liao, Xiangyu Zhao, Shaofeng Zhang, Xiaoxing Wang, Yifan Yang, Junchi Yan, Xue Yang
Computer Vision
Towards Universal Computational Aberration Correction in Photographic Cameras: A Comprehensive Benchmark Analysis
Xiaolong Qian, Qi Jiang, Yao Gao et al. (+5 more)
Prevalent Computational Aberration Correction (CAC) methods are typically tailored to specific optical systems, leading to poor generalization and labor-intensive re-training for new lenses. Developing CAC paradigms capable of generalizing across diverse photographic lenses offers a promising solution to these challenges. However, efforts to achieve such cross-lens universality within consumer photography are still in their early stages due to the lack of a comprehensive
benchmark that encompasses a sufficiently wide range of optical aberrations. Furthermore, it remains unclear which specific factors influence existing CAC methods and how these factors affect their performance. In this paper, we present comprehensive experiments and evaluations involving 24 image restoration and CAC
algorithms, utilizing our newly proposed UniCAC, a large-scale
benchmark for photographic cameras constructed via automatic optical design. The Optical Degradation Evaluator (ODE) is introduced as a novel framework to objectively assess the difficulty of CAC tasks, offering credible quantification of optical aberrations and enabling reliable evaluation. Drawing on our comparative analysis, we identify three key factors – prior utilization, network architecture, and training strategy – that most significantly influence CAC performance, and further investigate their respective effects. We believe that our
benchmark,
dataset, and observations contribute foundational insights to related areas and lay the groundwork for future investigations.
Benchmarks, codes, and Zemax files will be available at https://github.com/XiaolongQian/UniCAC.
Read the full abstract ▾
All authors Xiaolong Qian, Qi Jiang, Yao Gao, Lei Sun, Zhonghua Yi, Kailun Yang, Luc Van Gool, Kaiwei Wang
Computer Vision
Node-RF: Learning Generalized Continuous Space-Time Scene Dynamics with Neural ODE-based NeRFs
Hiran Sarkar, Liming Kuang, Yordanka Velikova et al. (+1 more)
Predicting scene dynamics from visual observations is challenging. Existing methods capture dynamics only within observed boundaries failing to extrapolate far beyond the training sequence. Node-RF (Neural ODE-based NeRF) overcomes this limitation by integrating Neural Ordinary Differential Equations (NODEs) with dynamic Neural Radiance Fields (NeRFs), enabling a continuous-time, spatiotemporal representation that generalizes beyond observed trajectories at constant
memory cost. From visual input, Node-RF learns an implicit scene state that evolves over time via an ODE solver, propagating feature
embeddings via differential calculus. A NeRF-based renderer interprets calculated
embeddings to synthesize arbitrary views for long-range extrapolation. Training on multiple motion sequences with shared dynamics allows for generalization to unseen conditions. Our experiments demonstrate that Node-RF can characterize abstract system behavior without explicit model to identify critical points for future predictions.
Read the full abstract ▾
All authors Hiran Sarkar, Liming Kuang, Yordanka Velikova, Benjamin Busam
Computer Vision
LoV3D: Grounding Cognitive Prognosis Reasoning in Longitudinal 3D Brain MRI via Regional Volume Assessments
Zhaoyang Jiang, Zhizhong Fu, David McAllister et al. (+2 more)
Longitudinal
brain MRI is essential for characterizing the progression of neurological diseases such as Alzheimer's disease assessment. However, current deep-learning tools fragment this process: classifiers reduce a scan to a label, volumetric pipelines produce uninterpreted measurements, and
vision-
language models (VLMs) may generate fluent but potentially hallucinated conclusions. We present LoV3D, a pipeline for training 3D
vision-
language models, which reads longitudinal T1-weighted
brain MRI, produces a region-level anatomical assessment, conducts longitudinal comparison with the prior scan, and finally outputs a three-class diagnosis (Cognitively Normal, Mild Cognitive Impairment, or Dementia) along with a synthesized diagnostic summary. The stepped pipeline grounds the final diagnosis by enforcing label consistency, longitudinal coherence, and biological plausibility, thereby reducing the risks of
hallucinations. The training process introduces a clinically-weighted Verifier that scores candidate outputs automatically against normative references derived from standardized volume metrics, driving Direct Preference Optimization without a single human annotation. On a subject-level held-out ADNI test set (479 scans, 258 subjects), LoV3D achieves 93.7% three-class diagnostic accuracy (+34.8% over the no-grounding baseline), 97.2% on two-class diagnosis accuracy (+4% over the SOTA) and 82.6% region-level anatomical classification accuracy (+33.1% over VLM baselines). Zero-shot transfer yields 95.4% on MIRIAD (100% Dementia recall) and 82.9% three-class accuracy on AIBL, confirming high generalizability across sites, scanners, and populations. Code is available at https://github.com/Anonymous-TEVC/LoV-3D.
Read the full abstract ▾
All authors Zhaoyang Jiang, Zhizhong Fu, David McAllister, Yunsoo Kim, Honghan Wu
Computer Vision
Beyond Convolution: A Taxonomy of Structured Operators for Learning-Based Image Processing
Simone Cammarasana
The convolution operator is the fundamental building block of modern convolutional neural networks (CNNs), owing to its simplicity, translational equivariance, and efficient implementation. However, its structure as a fixed, linear, locally-averaging operator limits its ability to capture structured signal properties such as low-rank decompositions, adaptive basis representations, and non-uniform spatial dependencies. This paper presents a systematic taxonomy of operators that extend or replace the standard convolution in learning-based image processing pipelines. We organise the landscape of alternative operators into five families: (i) decomposition-based operators, which separate structural and noise components through singular value or tensor decompositions; (ii) adaptive weighted operators, which modulate kernel contributions as a function of spatial position or signal content; (iii) basis-adaptive operators, which optimise the analysis bases together with the network weights; (iv) integral and kernel operators, which generalise the convolution to position-dependent and non-linear kernels; and (v)
attention-based operators, which relax the locality assumption entirely. For each family, we provide a formal definition, a discussion of its structural properties with respect to the convolution, and a critical analysis of the tasks for which the operator is most appropriate. We further provide a comparative analysis of all families across relevant dimensions – linearity, locality, equivariance, computational cost, and suitability for image-to-image and image-to-label tasks – and outline the open challenges and future directions of this research area.
Read the full abstract ▾
Computer Vision
Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos
Shuo Sun, Unal Artan, Malcolm Mielle et al. (+2 more)
We address the challenging problem of dense dynamic scene reconstruction and camera pose estimation from multiple freely moving cameras – a setting that arises naturally when multiple observers capture a shared event. Prior approaches either handle only single-camera input or require rigidly mounted, pre-calibrated camera rigs, limiting their practical applicability. We propose a two-stage optimization framework that decouples the task into robust camera tracking and dense depth refinement. In the first stage, we extend single-camera visual SLAM to the multi-camera setting by constructing a spatiotemporal connection graph that exploits both intra-camera temporal continuity and inter-camera spatial overlap, enabling consistent scale and robust tracking. To ensure robustness under limited overlap, we introduce a wide-baseline initialization strategy using feed-forward reconstruction models. In the second stage, we refine depth and camera poses by optimizing dense inter- and intra-camera consistency using wide-baseline optical flow. Additionally, we introduce MultiCamRobolab, a new real-world
dataset with ground-truth poses from a motion capture system. Finally, we demonstrate that our method significantly outperforms state-of-the-art feed-forward models on both synthetic and real-world
benchmarks, while requiring less
memory.
Read the full abstract ▾
All authors Shuo Sun, Unal Artan, Malcolm Mielle, Achim J. Lilienthal, Martin Magnusson
Computer Vision
NBAvatar: Neural Billboards Avatars with Realistic Hand-Face Interaction
David Svitov, Mahtab Dahaghin, Pietro Morerio et al. (+1 more)
We present NBAvatar - a method for realistic rendering of head avatars handling non-rigid deformations caused by hand-face interaction. To this end, we introduce a novel hybrid implicit-explicit representation for animated avatars by combining the training of explicit oriented planar primitives with implicit neural rendering. Such a combination of representations in the end-to-end pipeline enables NBAvatar to handle temporally and pose-consistent geometry, along with fine-grained appearance details provided by the neural rendering technique. To enable joint optimization of different representations we propose a geometry-aware training scheme that allows our hybrid representation to surpass existing approaches in terms of novel-view and novel-pose rendering quality. Specifically, NBAvatar achieves up to 53% LPIPS reduction compared to Gaussian-based avatar methods, while also improving PSNR and SSIM, and achieves higher structural similarity compared to the state-of-the-art hand-face interaction method InteractAvatar.
Read the full abstract ▾
All authors David Svitov, Mahtab Dahaghin, Pietro Morerio, Alessio Del Bue
Computer Vision
Adaptive Weighted h-Transform Sampling for Coarse-Guided Visual Generation
Yanghao Wang, Ziqi Jiang, Zhen Wang et al. (+1 more)
Coarse-guided visual generation, which synthesizes fine visual samples from degraded or low-fidelity coarse references, is essential for various real-world applications. While training-based approaches are effective, they are inherently limited by high training costs and restricted generalization due to paired data collection. Accordingly, recent training-free works propose to leverage pretrained
diffusion models and incorporate guidance during the sampling process. However, these training-free methods either require knowing the forward (fine-to-coarse) transformation operator, e.g., bicubic downsampling, or are difficult to balance between guidance and synthetic quality. To address these challenges, we propose a novel guided method by using the h-transform, a tool that can constrain
stochastic processes (e.g., sampling process) under desired conditions. Specifically, we modify the transition probability at each sampling timestep by adding to the original differential equation with a drift function $h$, which approximately steers the generation toward the ideal fine sample. To address unavoidable approximation errors, we introduce an adaptive weight scheduler that combines a noise-level-aware initialization with a correction based on cross-timestep consistency, balancing guidance adherence and synthesis quality. Extensive experiments across diverse image and
video generation tasks demonstrate its effectiveness and generalization.
Read the full abstract ▾
All authors Yanghao Wang, Ziqi Jiang, Zhen Wang, Long Chen
Computer Vision
Continual Learning with Vision-Language Models via Semantic-Geometry Preservation
Chiyuan He, Zihuan Qiu, Fanman Meng et al. (+4 more)
Continual learning of pretrained
vision-
language models (VLMs) is prone to catastrophic forgetting, yet current approaches adapt to new tasks without explicitly preserving the cross-modal semantic geometry inherited from pretraining and previous stages, allowing new-task supervision to induce
geometric distortion. We observe that the most pronounced drift tends to concentrate in vulnerable neighborhoods near the old-new semantic interface, where shared visual patterns are easily re-explained by new textual semantics. To address this under an exemplar-free constraint, we propose Semantic Geometry Preservation for Continual Learning (SeGP-CL). SeGP-CL first probes the drift-prone region by constructing a compact set of
adversarial anchors with dual-targeted projected gradient descent (DPGD), which drives selected new-task seeds toward old-class semantics while remaining faithful in raw visual space. During training, we preserve cross-modal structure by anchor-guided cross-modal geometry
distillation (ACGD), and stabilize the textual reference frame across tasks via a lightweight text semantic-geometry regularization (TSGR). After training, we estimate anchor-induced raw-space drift to transfer old visual prototypes and perform dual-path inference by fusing cross-modal and visual cues. Extensive experiments on five continual learning
benchmarks demonstrate that SeGP-CL consistently improves stability and forward transfer, achieving state-of-the-art performance while better preserving semantic geometry of VLMs. Code is available at: https://github.com/chiyuan-IVIPLab/SeGP-CL.
Read the full abstract ▾
All authors Chiyuan He, Zihuan Qiu, Fanman Meng, Runtong Zhang, Linfeng Xu, Qingbo Wu, Hongliang Li
Computer Vision · Cross-listed from eess.AS
Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition
Umberto Cappellazzo, Stavros Petridis and Maja Pantic
Audio-Visual
Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework using Shapley values to analyze modality contributions in AVSR. Through experiments on six models across two
benchmarks and varying SNR levels, we introduce three analyses: Global SHAP for overall modality balance, Generative SHAP for contribution dynamics during decoding, and Temporal
Alignment SHAP for input-output correspondence. Our findings reveal that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation. Modality balance evolves during generation, temporal
alignment holds under noise, and SNR is the dominant factor driving modality weighting. These findings expose a persistent audio bias, motivating ad-hoc modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.
Read the full abstract ▾
Computer Vision
Single Pixel Image Classification using an Ultrafast Digital Light Projector
Aisha Kanwal, Graeme E. Johnstone, Fahimeh Dehkhoda et al. (+5 more)
Pattern recognition and image classification are essential tasks in machine
vision. Autonomous vehicles, for example, require being able to collect the complex information contained in a changing environment and classify it in real time. Here, we experimentally demonstrate image classification at multi-kHz frame rates combining the technique of single pixel imaging (SPI) with a low complexity machine learning model. The use of a microLED-on-CMOS digital light projector for SPI enables ultrafast pattern generation for sub-ms image encoding. We investigate the classification accuracy of our experimental system against the broadly accepted benchmarking task of the MNIST digits classification. We compare the classification performance of two machine learning models: An extreme learning machine (ELM) and a backpropagation trained deep neural network. The complexity of both models is kept low so the overhead added to the inference time is comparable to the image generation time. Crucially, our single pixel image classification approach is based on a spatiotemporal transformation of the information, entirely bypassing the need for image reconstruction. By exploring the performance of our SPI based ELM as binary classifier we demonstrate its potential for efficient anomaly detection in ultrafast imaging scenarios.
Read the full abstract ▾
All authors Aisha Kanwal, Graeme E. Johnstone, Fahimeh Dehkhoda, Johannes H. Herrnsdorf, Robert K. Henderson, Martin D. Dawson, Xavier Porte, Michael J. Strain
Computer Vision
Nyxus: A Next Generation Image Feature Extraction Library for the Big Data and AI Era
Nicholas Schaub, Andriy Kharchenko, Hamdah Abbasi et al. (+3 more)
Modern imaging instruments can produce terabytes to petabytes of data for a single experiment. The biggest barrier to processing big image
datasets has been computational, where image analysis
algorithms often lack the efficiency needed to process such large
datasets or make tradeoffs in robustness and accuracy. Deep learning
algorithms have vastly improved the accuracy of the first step in an analysis workflow (region
segmentation), but the expansion of domain specific feature extraction libraries across scientific disciplines has made it difficult to compare the performance and accuracy of extracted features. To address these needs, we developed a novel feature extraction library called Nyxus. Nyxus is designed from the ground up for scalable out-of-core feature extraction for 2D and 3D image data and rigorously tested against established standards. The comprehensive feature set of Nyxus covers multiple biomedical domains including radiomics and cellular analysis, and is designed for computational scalability across CPUs and GPUs. Nyxus has been packaged to be accessible to users of various skill sets and needs: as a Python package for code developers, a command line tool, as a Napari plugin for low to no-code users or users that want to visualize results, and as an Open Container Initiative (OCI) compliant container that can be used in cloud or super-computing workflows aimed at processing large data sets. Further, Nyxus enables a new methodological approach to feature extraction allowing for programmatic tuning of many features sets for optimal computational efficiency or coverage for use in novel machine learning and deep learning applications.
Read the full abstract ▾
All authors Nicholas Schaub, Andriy Kharchenko, Hamdah Abbasi, Sameeul Samee, Hythem Sidky, Nathan Hotaling
Computer Vision
Pano360: Perspective to Panoramic Vision with Geometric Consistency
Zhengdong Zhu, Weiyi Xue, Zuyuan Yang et al. (+2 more)
Prior panorama stitching approaches heavily rely on pairwise feature correspondences and are unable to leverage
geometric consistency across multiple views. This leads to severe distortion and misalignment, especially in challenging scenes with weak textures, large parallax, and repetitive patterns. Given that multi-view
geometric correspondences can be directly constructed in 3D space, making them more accurate and globally consistent, we extend the 2D
alignment task to the 3D photogrammetric space. We adopt a novel
transformer-based architecture to achieve 3D awareness and aggregate global information across all views. It directly utilizes camera poses to guide image warping for global
alignment in 3D space and employs a multi-feature joint optimization strategy to compute the seams. Additionally, to establish an evaluation
benchmark and train our network, we constructed a large-scale
dataset of real-world scenes. Extensive experiments show that our method significantly outperforms existing alternatives in
alignment accuracy and perceptual quality.
Read the full abstract ▾
All authors Zhengdong Zhu, Weiyi Xue, Zuyuan Yang, Wenlve Zhou, Zhiheng Zhou
Computer Vision
CrossEarth-SAR: A SAR-Centric and Billion-Scale Geospatial Foundation Model for Domain Generalizable Semantic Segmentation
Ziqi Ye, Ziyang Gong, Ning Liao et al. (+10 more)
Synthetic Aperture Radar (SAR) enables global, all-weather earth observation. However, owing to diverse imaging mechanisms, domain shifts across sensors and regions severely hinder its semantic generalization. To address this, we present CrossEarth-SAR, the first billion-scale SAR
vision foundation model built upon a novel physics-guided sparse mixture-of-experts (MoE) architecture incorporating physical descriptors, explicitly designed for cross-domain semantic
segmentation. To facilitate large-scale pre-training, we develop CrossEarth-SAR-200K, a weakly and fully supervised
dataset that unifies public and private SAR imagery. We also introduce a
benchmark suite comprising 22 sub-
benchmarks across 8 distinct domain gaps, establishing the first unified standard for domain generalization semantic
segmentation on SAR imagery. Extensive experiments demonstrate that CrossEarth-SAR achieves state-of-the-art results on 20
benchmarks, surpassing previous methods by over 10% mIoU on some
benchmarks under multi-gap transfer. All code,
benchmark and
datasets will be publicly available.
Read the full abstract ▾
All authors Ziqi Ye, Ziyang Gong, Ning Liao, Xiaoxing Hu, Di Wang, Hongruixuan Chen, Chen Huang, Yiguo He, Yuru Jia, Xiaoxing Wang, Haipeng Wang, Xue Yang, Junchi Yan
Computer Vision
Ada3Drift: Adaptive Training-Time Drifting for One-Step 3D Visuomotor Robotic Manipulation
Chongyang Xu, Yixian Zou, Tianyu Yang et al. (+4 more)
Diffusion-based visuomotor policies model complex action distributions through iterative denoising, but repeated inference adds latency to robotic control. One-step generators reduce this cost, motivating training objectives that retain useful action structure with few demonstrations. We present Ada3Drift, a point-cloud-conditioned policy that builds on Drifting Models to perform distribution refinement during training and generate action chunks in one forward pass. Our central design is a regression-to-drifting curriculum: paired action regression first emphasizes observation–action correspondence, while a sigmoid schedule progressively increases a batch-level action-distribution regularizer. The drifting term combines attraction to demonstrated actions and repulsion among generated samples using inherited multi-temperature aggregation. A timestep-free generator preserves single-step inference throughout. On Adroit, Meta-World, RoboTwin, and five real-world tasks, Ada3Drift achieves the highest reported average success rates among the evaluated baselines with 1 NFE, compared with 10 NFE for the
diffusion baselines. Controlled ablations favor sigmoid scheduling at matched demonstration budgets. We will release our code and pretrained model weights.
Read the full abstract ▾
All authors Chongyang Xu, Yixian Zou, Tianyu Yang, Fanman Meng, Ziliang Feng, Li Lu, Shuaicheng Liu