Computer Vision
Hybrid Machine Learning Model for Forest Height Estimation from TanDEM-X and Landsat Data
Islam Mansour, Ronny Haensch, Irena Hajnsek et al. (+1 more)
Integrating machine learning (ML) with physical models (PM) has emerged as a promising way of retrieving geophysical parameters from remote sensing data. In this context, a ML model for estimating forest height from TanDEM-X interferometric coherence measurements has recently been proposed, that constrains the learning process through a PM. While the features used for training and inversion where selected to ensure the physical consistency of the solutions, they could not resolve all height / structure and baseline / terrain slope ambiguities in the data. To improve this, the extension of the feature space with optical Landsat data is proposed able to provide complementary information on forest type or structure. The extended model is applied and validated on several TanDEM-X acquisitions over the Gabonese Lopé national park site and assessed against airborne
LiDAR measurements. Results show a 13.5% reduction in RMSE and a 16.6% reduction in MAE compared to the original hybrid model, confirming the added value of multispectral inputs.
Read the full abstract ▾
All authors Islam Mansour, Ronny Haensch, Irena Hajnsek, Konstantinos Papathanassiou
Computer Vision
Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction
Hao Xu, Yilin Liu, Yinqiao Wang et al. (+2 more)
We ask whether everyday open-world
monocular videos can be turned into reusable 4D interaction primitives: articulated hand motion, object shape with 6D pose over time, and the when/where of contact. Such a capability would enable scalable mining of real interactions and, beyond reconstruction, support scene-aware synthesis and planning. However, reconstructing hand-object interaction (HOI) from challenging
monocular videos remains difficult: methods often assume known objects or curated scenes, and separately estimated hands and objects easily become misaligned under clutter, occlusion, and unseen object geometries. Targeting this setting, we present Open-CHOIR, an Open-world Contact-aware HOI Reconstruction framework for a
monocular camera, using contact as an explicit coupling signal between hands and objects. Open-CHOIR first initializes a coarse, contact-agnostic 4D HOI sequence from open-world visual priors. It then introduces a generative HOI spatial rectification module to predict ray-depth corrections and rectify hand-object relative placement, then derive initial per-frame contact correspondences on the rectified geometry. Last, a contact-aware joint optimization with dynamically updated contact constraints enforces
geometric, temporal, and contact consistency. Experiments on controlled and challenging
videos show that Open-CHOIR improves object reconstruction, physical plausibility, and temporal consistency over state-of-the-art methods. Code, data, and pretrained weights are available at https://github.com/hxwork/CHOIR.
Read the full abstract ▾
All authors Hao Xu, Yilin Liu, Yinqiao Wang, Chi-Wing Fu, Niloy J. Mitra
Computer Vision
Towards Integrated Rock Support Visualisation in 3D Point Cloud of Underground Mines
Dibyayan Patra, Simit Raval, Pasindu Ranasinghe et al. (+2 more)
The effectiveness of rock support in underground mines depends on the interaction between installed rock bolts and the structural fabric of the surrounding rock mass. However, discontinuity characterisation and rock bolt identification are commonly treated as separate tasks, limiting their value for integrated support assessment. This study presents an automated framework for integrated rock support visualisation using 3D point clouds of underground mine excavations. The framework integrates structure mapping, rock bolt identification, discontinuity plane fitting, and bolt orientation estimation into a unified workflow optimised for accuracy and computational efficiency. The outputs are used to generate an integrated 3D visualisation of fitted discontinuity planes and bolt vectors, enabling direct assessment of their spatial intersections and
geometric relationships. A complementary stereographic analysis of discontinuity poles and bolt orientations is also performed to evaluate overall bolting
geometric effectiveness relative to the mapped structural fabric. Additionally, bolt-level quality metrics, including exposed protrusion length and deviation from the local roof normal, are visualised to support assessment of installation quality. The proposed framework is demonstrated on real underground metal mine scans, producing accurate structure mapping and rock bolt identification results in medium-scale point clouds. Overall, the study provides a practical step towards automated, integrated geotechnical assessment of rock support effectiveness without requiring manual measurements or additional in-situ data acquisition.
Read the full abstract ▾
All authors Dibyayan Patra, Simit Raval, Pasindu Ranasinghe, Bikram Banerjee, Ismet Canbulat
Computer Vision
Comparative Evaluation of Deep Learning Models for Fake Image Detection
Akhitha Pakala, Mohammed Mahir Rahman, Shahzad Memon et al. (+1 more)
The growing sophistication of GAN-based image manipulation presents significant challenges for digital forensics. This study compares the performance of four pretrained CNN architectures including VGG16, ResNet50, EfficientNetB0, and XceptionNet for fake image detection using a unified preprocessing and training pipeline. A
dataset of real and manipulated images was processed through resizing, normalization, and augmentation to address class imbalance and improve generalization. Models were evaluated using Accuracy, Precision, Recall, F1-score, and ROC-AUC. VGG16 achieved the highest accuracy at 91%, with XceptionNet, ResNet50, and EfficientNetB0 each reaching 90%. EfficientNetB0 showed stronger sensitivity to fake images but reduced reliability on real samples, reflecting imbalance-driven bias. Limitations include
dataset imbalance, overfitting, and limited interpretability, which affect cross-domain robustness. The study provides a reproducible baseline and underscores the need for balanced
datasets, advanced augmentation, and
fairness-aware training to develop reliable fake image detection systems.
Read the full abstract ▾
All authors Akhitha Pakala, Mohammed Mahir Rahman, Shahzad Memon, Tauseef Ahmed
Computer Vision
Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
Yutong Xie, Zhenglin Hua, Ran Wang et al. (+3 more)
Large
Vision-
Language Models (LVLMs) have shown remarkable performance on a wide range of
vision-
language tasks. Despite this progress, they are still prone to
hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient
attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel
hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual
Attention Discrepancy (ILVAD). Specifically, we obtain the
attention weights from early generated
tokens to visual
tokens across layers and identify the
tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance
attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain
attention scores of generated text to visual evidence, in order to select and emphasize text
tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple
benchmark evaluations conducted on five recently released models show that our method can consistently mitigate
hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.
Read the full abstract ▾
All authors Yutong Xie, Zhenglin Hua, Ran Wang, Wing W. Y. Ng, Xizhao Wang, Yuheng Jia
Computer Vision
Towards UAV Detection in the Real World: A New Multispectral Dataset UAVNet-MS and a New Method
Yihang Luo, Jun Chen, Chao Xiao et al. (+13 more)
The proliferation of unmanned aerial vehicles (
UAVs) has created urgent demand for precise
UAV monitoring. Existing RGB-based systems rely on spatial cues that degrade at small scales, particularly with high inter-type similarity, target-clutter ambiguity, and low contrast. Multispectral imaging (MSI) encodes material-aware spectral signatures, yet MSI-based fine-grained small-
UAV detection remains underexplored due to lack of dedicated
datasets. We introduce UAVNet-MS, the first multispectral
dataset for fine-grained small-
UAV detection, comprising 15,618 temporally synchronized RGB-MSI data cubes (1440x1080) with bounding box annotations. The
dataset features challenging small objects (93.7% <= 32^2 pixels, average 18^2 pixels, ~0.02% image area) under low contrast. We propose MFDNet, a dual-stream baseline addressing array-induced parallax and spatial-spectral fusion. Extensive evaluation under RGB-only, MSI-only, and RGB+MSI protocols against 20 detectors shows MFDNet achieves +6.2% AP50 improvement over best RGB-only methods, demonstrating spectral cues provide complementary material evidence beyond spatial cues. This work provides foundational
dataset, strong baseline, and
benchmark for multispectral
UAV monitoring research.
Read the full abstract ▾
All authors Yihang Luo, Jun Chen, Chao Xiao, Yingqian Wang, Zhaoxu Li, Qiang Ling, Xu He, Nuo Chen, Gaowei Guo, Hongge Li, Miao Li, Longguang Wang, Yulan Guo, Li Liu, Wei An, Zhijie Chen
Computer Vision
Preserve, Reveal, Expand: Towards Faithful 4D Video Editing with Region-Aware Conditioning and Benchmarking
Zhangchi Hu, Wenzhang Sun, Xiangchen Yin et al. (+5 more)
While 4D-driven
video diffusion models can generate visually plausible
videos, faithful 4D editing poses a stricter requirement: source-observed content should be preserved, while disoccluded and out-of-view regions must be synthesized naturally. We identify a key failure mode in existing methods, which we term Evidence-Role Mismatch: source-backed observations, uncertain rendered cues, and unsupported regions are entangled within a single conditioning signal, which can lead to preservation drift, ghosting, and unstable extrapolation. To mitigate this issue, we propose PREX (Preserve, Reveal, Expand), a region-aware framework that decomposes the target spatiotemporal volume into three distinct roles based on observation support and scene extent. PREX traces edited 4D points back to source frames and retrieves observed colors and pairs these appearance cues with geometry-aware confidence and region roles, and injects them into a frozen
video diffusion backbone through an adapter trained with self-supervised proxy editing tasks. We further introduce PREBench, a diagnostic
benchmark specifically designed for 4D
video editing, to expose and quantify such region-specific failures. PREBench separately evaluates Preserve fidelity, Reveal quality, and Expand plausibility of
videos generated by 4D driven
video diffusion models. Experiments show that PREX substantially reduces region-specific failure modes while maintaining strong visual quality and 4D edit control compared with existing 4D-based
video diffusion methods. Code will be publicly released. Project Page: https://ricepastem.github.io/PREX-Open
Read the full abstract ▾
All authors Zhangchi Hu, Wenzhang Sun, Xiangchen Yin, Jiahui Yuan, Chunfeng Wang, Hao Li, Kun Zhan, Xiaoyan Sun
Computer Vision
DrawMotion: Generating 3D Human Motions by Freehand Drawing
Tao Wang, Lei Jin, Zhihua Wu et al. (+7 more)
Text-to-motion generation, which translates textual descriptions into human motions, faces the challenge that users often struggle to precisely convey their intended motions through text alone. To address this issue, this paper introduces DrawMotion, an efficient
diffusion-based framework designed for multi-condition scenarios. DrawMotion generates motions based on both a conventional text condition and a novel hand-drawing condition, which provide semantic and spatial control over the generated motions, respectively. Specifically, we tackle the fine-grained motion generation task from three perspectives: 1) freehand drawing condition. To accurately capture users' intended motions without requiring tedious textual input, we develop an
algorithm to automatically generate hand-drawn stickman sketches across different
dataset formats; 2) multi-condition fusion. We propose a Multi-Condition Module (MCM) that is integrated into the
diffusion process, enabling the model to exploit all possible condition combinations while reducing computational complexity compared to conventional approaches; and 3) training-free guidance. Notably, the MCM in DrawMotion ensures that its intermediate features lie in a continuous space, allowing classifier-guidance gradients to update the features and thereby aligning the generated motions with user intentions while preserving fidelity. Quantitative experiments and user studies demonstrate that the freehand drawing approach reduces user time by approximately 46.7% when generating motions aligned with their imagination. The code, demos, and relevant data are publicly available at https://github.com/InvertedForest/DrawMotion.
Read the full abstract ▾
All authors Tao Wang, Lei Jin, Zhihua Wu, Qiaozhi He, Jiaming Chu, Yu Cheng, Junliang Xing, Jian Zhao, Shuicheng Yan, Li Wang
Computer Vision
Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models
Yulin Zhao and Zheng Zhang
Vision-
Language Models (VLMs) face a bottleneck of prohibitive computational costs arising from massive visual
token sequences during inference. Existing
vision token reduction methods alleviate this burden, but they unintentionally preserve the isolated visual subject strictly aligned with the user's query, which fails to substantially explore salient subjects and their contextual relationships. In this paper, we propose SPpruner, a subject-centric progressive reduction paradigm that emulates the Focus-then-Context mechanism of the human visual perception system. Specifically, we first construct a focus identification module to explicitly model the interplay between visual saliency and semantic relevance. Herein, it can excavate the comprehensive visual subject spectrum to ensure a high-fidelity representation of visual input. Subsequently, a context-aware structural scanning module is developed to aggregate contextual cues from neighboring regions. As such, it can effectively restore global relational dependencies to uphold the structural integrity of the preserved subjects. Extensive experiments demonstrate that our paradigm consistently outperforms SOTA methods, achieving up to 2.53 times speedup with only 22.2% of visual
tokens retained in Qwen2.5-VL and a 67% FLOPs reduction on LLaVA with a negligible 0.6% accuracy drop.
Read the full abstract ▾
Computer Vision
Bridging Structure and Language: Graph-Based Visual Reasoning for Autonomous Road Understanding
Lena Wild, Katie Z Luo and Marco Pavone
Structured road understanding of lane geometry,
topology, and traffic element relationships is foundational to safe autonomous driving. While
vision-
language models (VLMs) offer promising semantic flexibility, they lack the
geometric and relational grounding required for precise road
reasoning. Conversely, traditional modular systems, e.g., HD maps and topological road graphs, provide structural precision but remain semantically rigid. To bridge this gap, we introduce the Combined Road Substrate (CRS), a graph-grounded framework that makes
geometric road structure and open-vocabulary semantics jointly executable in a single representation. CRS enables the automatic generation of compositionally complex and linguistically varied question-answer pairs via recursive graph queries, augmented with a "grounding for free" mechanism that ensures logical traceability to specific map elements, and procedurally extracted chain-of-thought supervision traces. We demonstrate that state-of-the-art VLMs - including large, closed-source models - struggle significantly with structured road
reasoning, yet training a small 2- or 4-billion-parameter model with as few as 20 to 80 CRS-enriched scenes yields stable gains in compositional
reasoning tasks of varying depth. Analysis of model behavior via verifiable
reasoning traces reveals a systematic shift in failure modes: whereas baseline models fail at relational scene understanding, CRS-trained models reduce failures to attribute recognition, suggesting that the primary bottleneck in road understanding is not model scale, but the absence of structured supervision.
Read the full abstract ▾
Computer Vision
PaintCopilot: Modeling Painting as Autonomous Artistic Continuation
Yunge Wen, Yaluo Wang, Yuancheng Shen et al. (+2 more)
Existing neural painting methods are target-driven: given a reference image, strokes are optimized to reconstruct it, fixing the outcome before painting begins. We instead ask whether a model can predict plausible painting actions when the final image is unknown, a problem we call autonomous artistic continuation. PaintCopilot realizes this through three lightweight, task-specific models: a Target Predictor that infers a provisional visual target directly from the evolving canvas, a Stroke Predictor that formulates brushstroke generation as flow matching to capture multiple plausible continuations, and a Region Sampler that generates stroke sequences within artist-specified regions via a conditional VAE, trained on a constructed
dataset of 3000 portraits with stroke-level supervision. This lets AI autonomously act on an artwork whose direction is still open, through interruptible, revisable delegation mechanisms spanning individual strokes, extended sequences, bounded regions, and prior stroke history. A two-week deployment with ten artists suggests that creative agency was shaped not simply by how much work artists delegated, but by their ability to continually negotiate the direction, scope, and persistence of AI participation.
Read the full abstract ▾
All authors Yunge Wen, Yaluo Wang, Yuancheng Shen, Robert Krueger, Paul Pu Liang
Computer Vision
3D Reconstruction and Knowledge Distillation to Improve Multi-View Image Models to Explore Spike Volume Estimation in Wheat
Olivia Zumsteg, Jannis Widmer, Yann Bourdé et al. (+4 more)
Accurate estimation of wheat spike volume is important for yield component analysis and stress resilience assessment, yet field-based measurement remains challenging. Active 3D sensing methods such as Light Detection and Ranging (
LiDAR) or time-of-flight (ToF) are sensitive to plant motion or poorly suited to outdoor conditions, while 3D reconstructions are computationally expensive. Direct 2D image processing would offer computational advantages, but image-based models lack explicit
geometric information. We therefore propose a hybrid 2D-3D approach with knowledge
distillation during training while enabling efficient image-only inference. First, we train a rigid-invariant point cloud network using distance-based histogram features to obtain pose-robust
geometric representations. We then combine the 3D model with a proposed multi-view image-based regulated
Transformer (RT) in an ensemble architecture. Finally, we distill the ensemble knowledge into a purely image-based student model using either feature-based or label-based
distillation. The two distilled RTs reduce the mean absolute error (MAE) from 654.31 mm$^3$ of the non-distilled RT to 639.93 mm$^3$ and 644.62 mm$^3$, and increase correlation from 0.76 to 0.77 and 0.82, respectively. At the same time, inference time is reduced from 160 ms to 1.4 ms per spike.
Distillation further mitigates volume-dependent bias and reshapes the latent representation of the image model toward a geometry-aware shape. Our results demonstrate that 3D-informed training of a 2D
Transformer allows for scalable and efficient spike volume estimation for high-throughput field phenotyping.
Read the full abstract ▾
All authors Olivia Zumsteg, Jannis Widmer, Yann Bourdé, Norbert Kirchgessner, Andreas Hund, Lukas Roth, Paraskevi Nousi
Computer Vision · Cross-listed from cs.LG
Winfree Oscillatory Neural Network
Jiawen Dai and Yue Song
Oscillations and synchronization are widely believed to play a fundamental role in representation and computation. However, existing machine learning approaches based on synchronization dynamics have largely been confined to specialized settings such as object discovery, with limited evidence of scalability to standard
vision benchmarks or logic
reasoning tasks. We propose the Winfree Oscillatory Neural Network (WONN), a dynamical neural architecture based on generalized Winfree dynamics. WONN evolves representations on the
torus $(S^1)^d$ through structured oscillatory interactions, combining phase-based inductive biases with flexible and hierarchical interaction mechanisms instantiated as either fixed trigonometric mappings or learnable neural networks. We evaluate WONN on image recognition and complex
reasoning tasks, including CIFAR, ImageNet, Maze-hard, and Sudoku. Across these domains, WONN achieves competitive or superior performance with strong parameter efficiency. In particular, WONN is, to our knowledge, the first synchronization-based oscillatory architecture to scale competitively to ImageNet-1K. Furthermore, on Maze-hard, WONN achieves 80.1% accuracy using only 1% of the parameters of prior state-of-the-art models. These results suggest that structured oscillatory dynamics provide a scalable and parameter-efficient alternative to conventional neural architectures.
Read the full abstract ▾
Computer Vision
RISE: Reliable Improvement in Self-Evolving Vision-Language Models
Chaoran Xu, Yingmao Miao, Pengfei Zhang et al. (+3 more)
Vision-
language models (VLMs) have achieved strong
multimodal reasoning capabilities, but further improving them still relies heavily on large-scale human-constructed supervision for post-training. Such supervision is costly to obtain, especially for
reasoning-intensive
multimodal tasks where questions, answers, and feedback signals must be carefully designed. This motivates self-evolving learning, where a model improves itself through a dual-role closed loop: a questioner autonomously poses questions and a solver learns to solve them. However, we observe that current VLM self-evolving methods still face three major challenges: coarse-grained role alternation delays the interaction between question generation and solver adaptation; generated questions can progressively degrade in quality; and question types may collapse toward a narrow distribution. These issues limit the efficiency and reliability of self-
evolution. Thus, we propose RISE, a reliable self-evolving framework for
vision-
language models. RISE is built on three complementary designs: fine-grained role alternation, which shortens the feedback loop between the questioner and the solver to improve efficiency; a quality supervisor, which improves question validity and pseudo-label reliability; and skill-aware dynamic balancing, which mitigates mode collapse and maintains broad skill coverage during
evolution. Together, these components enable more reliable and effective self-
evolution from unlabeled images. Experiments on two VLM backbones across seven
benchmarks show that RISE consistently improves the base models, yielding broad and sustained gains. Our code is publicly available at https://github.com/AMAP-ML/RISE.
Read the full abstract ▾
All authors Chaoran Xu, Yingmao Miao, Pengfei Zhang, Hao Dou, Lei Sun, Xiangxiang Chu
Computer Vision
FlowLong: Inference-time Long Video Generation via Manifold-constrained Tweedie Matching
Jangho Park, Geon Yeong Park, Gihyun Kwon et al. (+1 more)
Extending the generation horizon of
video diffusion models to long sequences remains a long-standing and important challenge. Existing training-free approaches fall into two categories: extensions of bidirectional models, which are tightly coupled to specific architectures and suffer from quality degradation over long horizons, and autoregressive models, which accumulate drift errors due to exposure bias and tend to produce repetitive motion patterns. To address these issues, we propose a novel but simple inference-time approach for long
video generation that is architecture-agnostic and requires no additional training. Our method generates long
videos via overlapping sliding windows, where predicted clean samples from adjacent windows are blended via Tweedie matching to enforce both
manifold constraint and temporal consistency across overlap regions.
Stochastic early-phase sampling then synchronizes per-window trajectories by injecting fresh noise after each Tweedie matching correction in the high-noise phase, before transitioning to deterministic ODE sampling to preserve fine-grained visual fidelity. Applied to various
video generation models, our method generates
videos several times longer than the native window length while outperforming both training-free and autoregressive baselines in temporal consistency and visual quality, and further extends to audio-
video joint generation and text-to-3DGS without any fine-tuning.
Read the full abstract ▾
All authors Jangho Park, Geon Yeong Park, Gihyun Kwon, Jong Chul Ye
Computer Vision
SynCB: A Synergy Concept-Based Model with Dynamic Routing Between Concepts and Complementary Neural Branches
Tores Julie, Sun Rémy, Sassatelli Lucile et al. (+3 more)
Concept-based (CB) models provide interpretability and support test-time human intervention, while standard neural networks (NN) offer strong task performance but little transparency. Prior work has explored hybrid formulations that integrate concepts and additional representations to improve accuracy, often at the cost of human interventions. We introduce the Synergy Concept-Based Model (SynCB) framework, that combines a CB branch with a complementary neural branch, and a trainable routing module that dynamically selects which branch to use for each input. Unlike prior models, which fuse residual and concept-based predictions, SynCB keeps the two branches distinct and coordinates them through the routing module. Moreover, both branches are learned jointly, allowing information sharing between the complementary neural branch and CB branches through their common backbone. To improve responsiveness to interventions, we further introduce a test-time intervention policy and a corresponding loss. Across five
datasets and CB
benchmarks, SynCB consistently achieves higher task accuracy while remaining more responsive to human interventions, surpassing the full neural baseline by up to 3.9 percentage points and exceeding the strongest competitor in intervention performance by up to 6.43 percentage points.
Read the full abstract ▾
All authors Tores Julie, Sun Rémy, Sassatelli Lucile, Ancarani Elisa, Wu Hui-Yin, Precioso Frédéric
Computer Vision
JFAA: Technical Report for the EPIC-KITCHENS-100 Action Anticipation Challenge at EgoVis 2026
Qiaohui Chu, Haoyu Zhang, Yisen Feng et al. (+4 more)
We propose JFAA, a JEPA-based Future Action Anticipation method for the EPIC-KITCHENS-100 (EK-100) Action Anticipation task. Inspired by the representation learning and future prediction ability of V-JEPA 2.1, JFAA uses a frozen encoder and predictor to extract observed context features and near-future latent
tokens. A lightweight attentive probe is then trained to predict verb, noun, and action logits with separate task queries. To improve robustness, we further build a field-aware ensemble over selected epoch-level predictions, allowing each output field to benefit from its most reliable candidates. Experimental results on the official challenge server show that JFAA achieves first place in the EgoVis 2026 EK-100 Action Anticipation Challenge. Our code will be released at https://github.com/CorrineQiu/JFAA.
Read the full abstract ▾
All authors Qiaohui Chu, Haoyu Zhang, Yisen Feng, Meng Liu, Weili Guan, Dongmei Jiang, Liqiang Nie
Computer Vision
VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026
Qiaohui Chu, Haoyu Zhang, Yisen Feng et al. (+4 more)
We propose VISTA, a V-JEPA Integrated StillFast Temporal Anticipator for the Ego4D Short-Term Object Interaction Anticipation (STA) Challenge at EgoVis 2026. Given an
egocentric video timestamp, the task requires anticipating the next human-object interaction, including the future active object's bounding box, noun category, verb category, time-to-contact, and confidence score. VISTA follows a StillFast-style design that combines object-centric spatial detection with short-horizon temporal context. Specifically, a COCO-pretrained Faster R-CNN ResNet-50 FPN detector generates object proposals from the last observed high-resolution frame, while a frozen V-JEPA 2.1 temporal branch extracts clip-level
egocentric context from the observed
video. The temporal representation is injected into the detection pathway through feature modulation and ROI-level context fusion. The fused proposal features are then passed to multi-head STA predictors for box refinement, noun classification, verb classification, time-to-contact regression, and interaction confidence estimation. For the final submission, we further ensemble complementary predictions to improve robustness. Experimental results on the official challenge server show that VISTA achieves first place in the EgoVis 2026 Ego4D STA Challenge. Our code will be released at https://github.com/CorrineQiu/VISTA.
Read the full abstract ▾
All authors Qiaohui Chu, Haoyu Zhang, Yisen Feng, Meng Liu, Weili Guan, Dongmei Jiang, Liqiang Nie
Computer Vision
FruitEnsemble: MLLM-Guided Arbitration for Heterogeneous ensemble in Fine-Grained Fruit Recognition
Enhui Yu, Junhui Li, Ruitong Lu et al. (+2 more)
Fine-grained fruit classification is a critical yet challenging task in agricultural computer
vision, primarily hindered by a severe shortage of high-quality
datasets and the high visual similarity between classes. To address these challenges, we first constructed a comprehensive
dataset comprising 306 fruit categories with 116,233 samples. Moreover, we propose FruitEnsemble, a practical two-stage dynamic inference framework designed to overcome the generalization limitations of static single-model architectures. In the first stage, FruitEnsemble employs a validation-calibrated weighted ensemble of heterogeneous backbones to generate a robust Top-3 candidate pool. To tackle difficult samples, we introduce an expert arbitration mechanism: when ensemble confidence falls below 0.6, a
multimodal large
language model (MLLM) is triggered to perform rigorous visual verification by integrating external botanical descriptions using Chain-of-Thought (CoT)
reasoning. Furthermore, we optimized the training pipeline with a hard sample-aware joint loss. Extensive experiments demonstrate that FruitEnsemble achieves a classification accuracy of 70.49% and outperforms existing state-of-the-art models. Our framework provides an efficient, deployment-oriented solution for real-world agricultural visual sorting and quality inspection tasks.
Read the full abstract ▾
All authors Enhui Yu, Junhui Li, Ruitong Lu, Jialu Li, Youshan Zhang
Computer Vision
HDMoE: A Hierarchical Decoupling-Fusion Mixture-of-Experts Framework for Multimodal Cancer Survival Prediction
Huayi Wang, Haochao Ying, Yuyang Xu et al. (+5 more)
Multimodal survival prediction, a crucial yet challenging task, demands the integration of
multimodal medical data (\eg Whole Slide Images (WSIs) and Genomic Profiles) to achieve accurate prognostic modeling. Given the inherent heterogeneity across modalities, the feature decoupling-fusion paradigm has emerged as a dominant approach. However, these methods have the following shortcomings: (1) fail to reduce the redundant information of modality features before decoupling, which negatively affects the feature decoupling and fusion effect;(2) lack the ability to model the fine-grained relationships of the features and capture the local information interactions between intra- and inter-modality features. To address these issues, we propose a \underline{H}ierarchical \underline{D}ecoupling-Fusion \underline{M}ixture-\underline{o}f-\underline{E}xperts (HDMoE) framework with two levels of MoE and \underline{R}andom \underline{F}eature \underline{R}eorganization (RFR) modules.In the first-level MoE, shared experts and routed experts are employed to remove redundant information and extract fine-grained specific features within each modality, while the second-level MoE facilitates fine-grained inter-modality feature decoupling. Besides, we design two RFR modules following each level of MoE to finely fuse intra- and inter-modality features, which can help the model capture more fine-grained relationships between modalities. Extensive experimental results on our private Liver Cancer (LC) and three TCGA public
datasets confirm the effectiveness of our proposed method. Codes are available at https://github.com/ZJUMAI/HDMoE.
Read the full abstract ▾
All authors Huayi Wang, Haochao Ying, Yuyang Xu, Qiyao Zheng, jun wang, Cheng Zhang, Ying Sun, Jian Wu
Computer Vision
Map-Mono-Ego: Map-Grounded Global Human Pose Estimation from Monocular Egocentric Video
Hiroyuki Deguchi, Ryosuke Hori, Kotaro Amaya et al. (+3 more)
Monocular egocentric human pose estimation is essential for ubiquitous activity monitoring. However, understanding the user's absolute location within the environment remains a challenge. Existing methods primarily focus on relative motion from an initial position, and tend not to account for the wearer's absolute location within an environment. Furthermore, inherent scale ambiguity in
monocular vision leads to severe translational drift, limiting long-term tracking without specialized multi-sensor hardware. To address this, we propose MapMonoEgo, a novel framework achieving globally consistent human pose estimation solely from a
monocular camera by leveraging a pre-scanned 3D point cloud. We also introduce AIST-Living
dataset, a new
dataset pairing
egocentric video with ground-truth motion in a scanned environment. Experiments demonstrate that our approach significantly outperforms the state-of-the-art baseline, proving its utility for practical monitoring tasks without specialized hardware.
Read the full abstract ▾
All authors Hiroyuki Deguchi, Ryosuke Hori, Kotaro Amaya, Tsubasa Maruyama, Mitsunori Tada, Hideo Saito
Computer Vision · Cross-listed from cs.MA
ProCrit: Self-Elicited Multi-Perspective Reasoning with Critic-Guided Revision for Multimodal Sarcasm Detection
Yingjia Xu, Jiulong Wu, Bowen Zhang et al. (+3 more)
Multimodal sarcasm detection requires
reasoning over cross-modal incongruities between literal expression and intended meaning, yet the specific analytical perspectives needed vary across samples due to the diversity of sarcastic mechanisms. While recent methods make this analytical process explicit, they still rely on fixed, predefined perspectives that operate independently under hand-crafted routing rules. We argue that
multimodal sarcasm detection instead calls for self-elicited multi-perspective
reasoning, where a model autonomously generates the perspectives needed for each sample and progressively integrates them into a coherent analysis. To realize this goal, we propose ProCrit, a Proposal-Critic two-
agent framework with a proposal
agent for multi-perspective
reasoning and a critic
agent for external evaluation and targeted revision guidance. First, to overcome the lack of process-level supervision in existing sarcasm
datasets, ProCrit synthesizes process-level
reasoning annotations through a dynamic-role agentic rollout: a strong
vision-
language model sequentially spawns analytical roles within a shared context, and the resulting multi-role trajectories are flattened into sequences that preserve cross-perspective dependencies while enabling efficient autoregressive generation. Second, to improve
reasoning reliability, ProCrit adopts a draft-critique-revise paradigm in which an independent critic identifies
reasoning deficiencies and provides targeted natural-
language feedback for directed revision. Finally, we develop a mutual-refinement training framework that jointly optimizes proposal drafting and feedback-guided revision via dual-stage reinforcement learning, while refining the critic
agent according to the actual effectiveness of its feedback. Experiments on three widely used
benchmarks demonstrate the effectiveness of ProCrit.
Read the full abstract ▾
All authors Yingjia Xu, Jiulong Wu, Bowen Zhang, Baokui Guo, Siyuan Chai, Min Cao
Computer Vision
Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models
Jeffrey Wang, Jonathan Gregory and Grigorios G. Chrysos
Modern
vision backbones treat pointwise activations (e.g., ReLU, GELU) and exponential softmax as essential sources of nonlinearity, but we demonstrate they are not required within MetaFormer-style
vision backbones. We design activation-free
polynomial alternatives for three core primitives (MLPs, convolutions, and
attention), where Hadamard products replace standard nonlinearities to yield
polynomial functions of the input. These modules integrate seamlessly into existing architectures: instantiated within MetaFormer, a modular framework for
vision backbones, our PolyNeXt models match or exceed activation-based counterparts across model scales on ImageNet classification, ADE20K semantic
segmentation, and out-of-distribution robustness. We also substantially outperform prior
polynomial networks at reduced computational cost, showing that
polynomial variants of standard modules beat complex custom architectures.
Read the full abstract ▾
Computer Vision
USV: Towards Understanding the User-generated Short-form Videos
Haoyue Cheng, Su Xu, Liwei Jin et al. (+3 more)
Several large-scale
video datasets have been published these years and have advanced the area of
video understanding. However, the newly emerged user-generated short-form
videos have rarely been studied. This paper presents USV, the User-generated Short-form
Video dataset for high-level semantic
video understanding. The
dataset contains around 224K
videos collected from UGC platforms by label queries without extra manual verification and trimming. Although
video understanding has achieved plausible improvement these years, most works focus on instance-level recognition, which is not sufficient for learning the representation of the high-level semantic information of
videos. Therefore, we further establish two tasks: topic recognition and
video-text
retrieval on USV. We propose two unified and effective baseline methods Multi-Modality Fusion Network (MMF-Net) and
Video-Text Contrastive Learning (VTCL), to tackle the topic recognition task and
video-text
retrieval respectively, and carry out comprehensive
benchmarks to facilitate future research. Our project page is https://usvdataset.github.io.
Read the full abstract ▾
All authors Haoyue Cheng, Su Xu, Liwei Jin, Wayne Wu, Chen Qian, Limin Wang
Computer Vision
ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models
Qirui Shen, Wenda Wang, Jiachen Lu et al. (+5 more)
Architectural spatial intelligence, the ability to recognize and infer architectural space, is fundamental to tasks such as
robot navigation,
embodied interaction, and 3D scene understanding and generation. Although extensive research has evaluated the basic spatial skills of
Vision-
Language Models (VLMs) such as relative orientation, distance comparison, and object counting, these tasks cover only the most elementary levels of spatial cognition and largely overlook higher-level cognition of architectural space, including layout understanding, circulation patterns, and functional zoning. In this work, we present ArchSIBench, a
Benchmark for Architectural Spatial Intelligence based on the perspectives from architecture, cognitive science, and psychology. ArchSIBench covers five core dimensions: perception,
reasoning,
navigation, transformation, and configuration, comprising 17 fine-grained subtasks. Through careful manual annotation by experts with architectural backgrounds, we construct 3,000 question-answer pairs to enable comprehensive evaluation of architectural spatial intelligence. Based on ArchSIBench, we evaluate various VLMs and find that the architectural spatial intelligence of most models shows significant differences from human baselines; additionally, models exhibit substantial variability across capability dimensions. Some state-of-the-art models can approach the level of human evaluators without architectural training. However, a clear gap remains compared to human evaluators with architectural training, particularly in spatial transformation and configuration
reasoning. We believe that ArchSIBench will provide important insights and systematic resources for measuring and advancing the architectural spatial intelligence of VLMs. The
dataset and code are available at https://huggingface.co/
datasets/ArchSIBench/ArchSIBench.
Read the full abstract ▾
All authors Qirui Shen, Wenda Wang, Jiachen Lu, Zilong Huang, Jin Bai, Lei He, Hongxuan Chen, Weixin Huang
Computer Vision
HyDAR-Pano3D: A Hybrid Disentangled Anatomical Recovery Framework for Panoramic-to-3D Reconstruction
Yaoyao Yue, Jérôme Schmid, Xiaoshuang Li et al. (+2 more)
Panoramic radiograph (PR) is fundamentally used in routine dental care, but it inherently provides only a two-dimensional (2D) projection of complex three-dimensional (3D) craniofacial anatomy. Most existing learning-based methods attempt to computationally recover this 3D information by directly regressing native cone-beam computed
tomography (CBCT) volumes from PR. However, this direct mapping requires the model to simultaneously learn common anatomical structures and patient-specific morphological variations. This entangled formulation makes the ill-posed 2D-to-3D inverse problem highly ambiguous, often producing over-smoothed reconstructions with blurred anatomical boundaries. To address this, we propose HyDAR-Pano3D, a two-stage framework that reformulates PR-to-CBCT reconstruction as a disentangled anatomical recovery problem. In Stage 1, a dual-encoder network integrates radiographic features with SAM-derived semantic priors to reconstruct an arch-normalized canonical volume. In Stage 2, an Anatomical Restoration Network predicts a prior-constrained structured deformation field to map this canonical volume back to the native space, restoring individual morphological variations. Experiments on three large-scale
datasets show that HyDAR-Pano3D significantly outperforms baseline methods ($p < 0.05$), achieving a 25.76 dB PSNR, 85.70% SSIM, and an 83.83% overall anatomical Dice score. The synthesized volumes successfully support downstream
segmentation of whole teeth (82.4% Dice) and the inferior alveolar canal (72.2% Dice), demonstrating that our disentangled approach preserves clinically relevant structures to enable robust anatomy-aware assessment when CBCT data is unavailable.
Read the full abstract ▾
All authors Yaoyao Yue, Jérôme Schmid, Xiaoshuang Li, Eduardo Delamare, Jinman Kim
Computer Vision
RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnesses
Minh Anh Nguyen, Quang Huy Tran, Bao Ngoc Le et al. (+2 more)
Open-vocabulary 3D scene graph generation seeks to describe object instances and their relations with flexible natural-
language predicates. The central difficulty is not only vocabulary expansion, but supervision reliability: relation annotations in 3D scene graph
datasets are selective, and many valid object-pair relations are unannotated. We propose RelWitness, a framework for open-vocabulary 3D scene graph generation from posed RGB-D sequences under incomplete relation supervision. The key concept is a relation witness: a concrete visual-
geometric cue that makes a relation observable in the captured scene. Support relations require contact and vertical ordering; containment requires enclosure; proximity requires metric closeness; orientation requires facing direction; and stable relations should persist across views where both objects are visible. RelWitness constructs relation witness records from RGB views, depth maps, reconstructed 3D geometry, role-sensitive text, object-prior null views, and multi-view consistency. A visual-
geometric witness verifier assigns unannotated relation candidates to verified missing positives, reliable negatives, or uncertain unlabeled cases. A witness-guided positive-unlabeled objective then learns from incomplete annotations without turning every missing label into a negative. We further introduce witness-consistent decoding and an RGB-D missing-relation audit protocol. Simulated manuscript-planning experiments on 3DSSG/3RScan and ScanNet-derived open-vocabulary splits show the intended behavior: improved unseen-relation recognition, higher witness precision, lower
hallucination, and reduced redundant relation phrases. All numerical results are planning values and must be replaced by reproduced measurements before submission
Read the full abstract ▾
All authors Minh Anh Nguyen, Quang Huy Tran, Bao Ngoc Le, Tuan Kiet Pham, Sui Yang Guang
Computer Vision
TERDNet: Transformer Encoder-Recurrent Decoder Network for Scene Change Detection
Jiae Yoon and Ue-Hwan Kim
In this work, we address the challenge of Scene Change Detection (SCD), where the goal is to identify variations between two images of the same location captured at different times. Existing SCD models often overlook the varying importance of features across layers, employ single-step decoders that confine refinement, and provide limited insight into encoder pretraining strategies. We propose TERDNet, a
Transformer Encoder-Recurrent Decoder Network designed to overcome these limitations. TERDNet consists of a
transformer-based encoder that extracts multi-level representations, a feature fusion module that integrates correlation volumes with these features, a recurrent 3-gate-GRU decoder that performs iterative refinement, and a combined convolution-interpolation upsampler that restores fine-grained resolution. Extensive experiments on four public
benchmarks show that TERDNet consistently outperforms prior approaches and produces more accurate and detailed change masks. Ablation studies confirm the benefit of
segmentation-based pretraining and the effectiveness of our fusion design. In addition, robustness tests under viewpoint misalignment confirm TERDNet's potential for deployment in real-world robotic systems, where reliable perception is critical. Our code is available at https://github.com/AutoCompSysLab/TERDNet.
Read the full abstract ▾
Computer Vision
VSCD: Video-based Scene Change Detection in Unaligned Scenes
Jiae Yoon and Ue-Hwan Kim
Detecting what has changed in an environment is essential for long-term autonomy, yet most change detection settings assume fixed viewpoints, mild misalignment, or only a few changed objects. We introduce
Video-based Scene Change Detection (VSCD), which predicts a pixel-wise change mask for each query frame, given a reference and a query RGB
video of the same indoor space recorded at different times under unconstrained camera motion. The two
videos are not temporally synchronized, and many object instances may appear or disappear. To study this setting, we build a large-scale
benchmark with over 1.1 million frames annotated with pixel-accurate change masks, together with a real-world test set for evaluating transfer beyond simulation. We propose a query-centric multi-reference model that learns temporal matching implicitly from change-mask supervision, aligns candidate reference features to the query via local patch correspondence, and fuses per-candidate change features using frame-level and patch-level confidence before decoding a high-resolution mask once per frame. Our approach achieves state-of-the-art performance against strong image- and
video-based baselines, and we validate its real-world impact by deploying it on a mobile
robot for two downstream applications – visual surveillance and object incremental learning.
Read the full abstract ▾
Computer Vision
AIR: Amortized Image Reconstruction Framework for Self-Supervised Feed-Forward 2D Gaussian Splatting
Zhaojie Zeng, Yuesong Wang, Yawei Luo et al. (+1 more)
2D Gaussian splatting provides an efficient explicit representation for image reconstruction, but existing methods still require costly per-image iterative optimization or rely on handcrafted priors for primitive allocation. We present AIR, a self-supervised feed-forward framework that amortizes iterative Gaussian fitting into a single network pass, eliminating per-image test-time optimization. AIR adopts a stage-wise residual architecture that progressively predicts additional Gaussian primitives from reconstruction residuals, together with an explicit Stage Control mechanism that activates new primitives only in under-reconstructed regions. A Predict–Optimize–Distill training strategy stabilizes multi-stage prediction by distilling short-horizon optimized Gaussian increments back into the predictor. The stabilized predictor is then jointly finetuned across stages and equipped with an image-adaptive quantizer for compact Gaussian storage. Experiments on Kodak and DIV2K show that AIR achieves better reconstruction quality than representative Gaussian-based baselines while reducing encoding time to 160–300\,ms. Code: https://github.com/whoiszzj/AIR.git
Read the full abstract ▾
All authors Zhaojie Zeng, Yuesong Wang, Yawei Luo, Tao Guan