Computer Vision · Cross-listed from cs.RO
Federico Pizzolato, Francesco Pasti and Nicola Bellotto
Terrain
segmentation is a fundamental capability for autonomous mobile
robots operating in unstructured outdoor environments. However, state-of-the-art models are incompatible with the
memory and compute constraints typical of microcontrollers, limiting scalable deployment in small robotics platforms. To address this gap, we develop a complete framework for robust binary terrain
segmentation on a low-cost microcontroller. At the core of our approach we design Nano-U, a highly compact binary
segmentation network with a few thousand parameters. To compensate for the network's minimal capacity, we train Nano-U via Quantization-Aware
Distillation (QAD), combining knowledge
distillation and quantization-aware training. This allows the final quantized model to achieve excellent results on the Botanic Garden
dataset and to perform very well on TinyAgri, a custom agricultural field
dataset with more challenging scenes. We deploy the quantized Nano-U on a commodity microcontroller by extending MicroFlow, a compiler-based inference engine for TinyML implemented in Rust. By eliminating interpreter overhead and dynamic
memory allocation, the quantized model executes on an ESP32-S3 with a minimal
memory footprint and low latency. This compiler-based execution demonstrates a viable and energy-efficient solution for perception on low-cost robotic platforms.
Read the full abstract ▾
Computer Vision
3DReflecNet: A Large-Scale Dataset for 3D Reconstruction of Reflective, Transparent, and Low-Texture Objects
Zhicheng Liang, Haoyi Yu, Boyan Li et al. (+6 more)
Accurate 3D reconstruction of objects with reflective, transparent, or low-texture surfaces still remains notoriously challenging. Such materials often violate key assumptions in multi-view reconstruction pipelines, such as photometric consistency and the availability on distinct
geometric texture cues. Existing
datasets primarily focus on diffuse, textured objects, and therefore provide limited insight into performance under real-world material complexities. We introduce 3DReflecNet, a large-scale hybrid
dataset exceeding 22 TB that is specifically designed to
benchmark and advance 3D
vision methods for these challenging materials. 3DReflecNet combines two types of data: over 120,000 synthetic instances generated via physically-based rendering of more than 12,000 shapes, and over 1,000 real-world objects captured using consumer devices. Together, these data consist of more than 7 million multi-view frames. The
dataset spans diverse materials, complex lighting conditions, and a wide range of
geometric forms, including shapes generated from both real and LLM-synthesized 2D images using
diffusion-based pipelines. To support robust evaluation, we design
benchmarks for five core tasks: image matching, structure-from-motion, novel view synthesis, reflection removal, and relighting. Extensive experiments demonstrate that state-of-the-art methods struggle to maintain accuracy across these settings, highlighting the need for more resilient 3D
vision models.
Read the full abstract ▾
All authors Zhicheng Liang, Haoyi Yu, Boyan Li, Dayou Zhang, Zijian Cao, Tianyi Gong, Junhua Liu, Shuguang Cui, Fangxin Wang
Computer Vision
DetRefiner: Model-Agnostic Detection Refinement with Feature Fusion Transformer
Soichiro Okazaki, Tatsuya Sasaki and Hiroki Ohashi
Open-vocabulary object detection (OVOD) aims to detect both seen and unseen categories, yet existing methods often struggle to generalize to novel objects due to limited integration of global and local contextual cues. We propose DetRefiner, a simple yet effective plug-and-play framework that learns to fuse global and local features to refine open-vocabulary detection. DetRefiner processes global image features and patch-level image features from foundational models (e.g., DINOv3) through a lightweight
Transformer encoder. The encoder produces a class vector capturing image-level attributes and patch vectors representing local region attributes, from which attribute reliability is inferred to recalibrate the base model's confidence. Notably, DetRefiner is trained independently of the base OVOD model, requiring neither access to its internal features nor retraining. At inference, it operates solely on the base detector's predictions, producing auxiliary calibration scores that are merged with the base detector's scores to yield the final refined confidence. Despite this simplicity, DetRefiner consistently enhances multiple OVOD models across COCO, LVIS, ODinW13, and Pascal VOC, achieving gains of up to +10.1 AP on novel categories. These results highlight that learning to fuse global and local representations offers a powerful and general mechanism for advancing open-world object detection. Our codes and models are available at https://github.com/hitachi-rd-cv/detrefiner.
Read the full abstract ▾
Computer Vision
SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation
Longteng Guo, Xuanxu Lin, Dongze Hao et al. (+5 more)
Scientific
reasoning is a key aspect of human intelligence, requiring the integration of
multimodal inputs, domain expertise, and multi-step inference across various subjects. Existing
benchmarks for
multimodal large
language models (MLLMs) often fail to capture the complexity and traceability of
reasoning processes necessary for rigorous evaluation. To fill this gap, we introduce SciVQR, a
multimodal benchmark covering 54 subfields in mathematics, physics, chemistry, geography, astronomy, and biology. SciVQR includes domain-specific visuals, such as equations, charts, and diagrams, and challenges models to combine visual comprehension with
reasoning. The tasks range from basic factual recall to complex, multi-step inferences, with 46% including expert-authored solutions. SciVQR not only evaluates final answers but also examines the
reasoning process, providing insights into how models reach their conclusions. Our evaluation of leading MLLMs, including both proprietary and open-source models, reveals significant limitations in handling complex
multimodal reasoning tasks, underscoring the need for improved multi-step
reasoning and better integration of interdisciplinary knowledge in advancing MLLMs toward true scientific intelligence. The
dataset and evaluation code are publicly available at https://github.com/CASIA-IVA-Lab/SciVQR.
Read the full abstract ▾
All authors Longteng Guo, Xuanxu Lin, Dongze Hao, Tongtian Yue, Pengkang Huo, Jiatong Ma, Yuchen Liu, Jing Liu
Computer Vision
DynGhost: Temporally-Modelled Transformer for Dynamic Ghost Imagings
Vittorio Palladino and Ahmet Enis Cetin
Ghost imaging reconstructs spatial information from a single-pixel bucket detector by correlating structured illumination patterns with scalar intensity measurements. While deep learning approaches have achieved promising results on static scenes, two critical limitations remain unaddressed: existing architectures fail to exploit temporal coherence across frames, leaving dynamic ghost imaging largely unsolved, and they assume additive Gaussian noise models that do not reflect the true Poissonian statistics of real single-
photon hardware. We present DynGhost (Dynamic Ghost Imaging
Transformer), a
transformer architecture that addresses both limitations through alternating spatial and temporal
attention blocks. Our
quantum-aware training framework, based on physically accurate detector simulations (SNSPDs, SPADs, SiPMs) and Anscombe variance-stabilizing normalization, resolves the distribution shift that causes classical models to fail under realistic hardware constraints. Experiments across multiple
benchmarks demonstrate that DynGhost outperforms both traditional reconstruction methods and existing deep learning architectures, with particular gains in dynamic and
photon-starved settings.
Read the full abstract ▾
Computer Vision
Developing a foundation model for high-resolution remote sensing data of the Netherlands
Paul Vermeeren and Heysem Kaya
We develop a foundation model using 1.2m high resolution satellite images of the Netherlands. By combining a Convolutional Neural Network and a
Vision Transformer, the model captures both low- and high-frequency landscape features, such as fine textures, edges, and small objects as well as large terrain structures, elevation patterns, and land-cover distributions. Leveraging temporal data as input, the model learns from broader contextual information across time, allowing the model to exploit the temporal dependencies, such as topographic features, land-cover changes, and seasonal dynamics. These additional constraints reduce feature ambiguity, improve representation learning, and enable better generalization with fewer labeled samples. The foundation model is evaluated on multiple downstream tasks, ranging from use cases within the Netherlands to global benchmarking
datasets. On the vegetation monitoring
dataset of the Netherlands, the model shows clear performance improvements by incorporating temporal information instead of relying on a single time point. Despite using a smaller model and less pretraining data limited to the Netherlands, it achieves competitive results on global
benchmarks when compared to state-of-the-art models. These results demonstrate that the model can learn rich, generalizable representations from limited data, achieving competitive performance on global
benchmarks while using a fraction of the parameters of larger state-of-the-art remote sensing models. To maximize reproducibility and reuse, we made the scripts and the model accessible on GitHub.
Read the full abstract ▾
Computer Vision
A Comparative Study of Machine Learning and Deep Learning for Out-of-Distribution Detection
Jihyeon Baek, Seunghoon Lee, Gitaek Kwon et al. (+1 more)
Out-of-distribution (OOD) detection is essential for building reliable AI systems, as models that produce outputs for invalid inputs cannot be trusted. Although deep learning (DL) is often assumed to outperform traditional machine learning (ML), medical imaging data are typically acquired under standardized protocols, leading to relatively constrained image variability in OOD detection tasks. This motivates a direct comparison between ML and DL approaches in this setting. The two approaches are evaluated on open
datasets comprising over 60,000 fundus and non-fundus images across multiple resolutions. Both approaches achieved an AUROC of 1.000 and accuracies between 0.999 and 1.000 on internal and external validation sets, showing comparable detection performance. The ML approach, however, exhibited substantially lower end-to-end latency while maintaining equivalent accuracy, indicating greater computational efficiency. These results suggest that for OOD detection tasks of limited visual complexity, lightweight ML approaches can achieve DL-level performance with significantly reduced computational cost, supporting practical real-world deployment.
Read the full abstract ▾
All authors Jihyeon Baek, Seunghoon Lee, Gitaek Kwon, Doohyun Park
Computer Vision
What Concepts Lie Within? Detecting and Suppressing Risky Content in Diffusion Transformers
Chenyu Zhang, Lanjun Wang, Yueyang Cheng et al. (+3 more)
The rise of text-to-image~(T2I) models has increasingly raised concerns regarding the generation of risky content, such as sexual, violent, and copyright-protected images, highlighting the need for effective safeguards within the models themselves.Although existing methods have been proposed to eliminate risky concepts from T2I models, they mainly focus on the erasure of risk-associated textual or latent representations, limiting their ability to defend against
adversarial prompts without explicit risk words, especially in recent
Diffusion-
Transformer-based T2I models. To bridge this gap, we investigate how semantic concepts activate the generation process within
Diffusion Transformers~(DiTs) and discover that
attention heads exhibit concept-specific sensitivity. This property enables both the detection and suppression of risky content. Building on this discovery, we propose a training-free inference-time safeguard for image generation in DiTs. Specifically, our method quantifies each textual
token's sensitivity across all
attention heads as an
Attention Head Vector~(AHV), which serves as a discriminative signature for detecting risky generation tendencies. In the inference stage, we propose a momentum-based strategy to dynamically track
token-wise AHVs across denoising steps, and a sensitivity-guided adaptive suppression strategy that suppresses the
attention logits of identified risky
tokens based on head-specific risk scores. Extensive experiments demonstrate that our method effectively suppresses sexual, copyrighted-style, and various harmful content while preserving visual quality, and further exhibits strong robustness against
adversarial prompts and transferability across different DiT-based T2I models. These results establish our method as a practical safeguard for the safe deployment of modern T2I models.
Read the full abstract ▾
All authors Chenyu Zhang, Lanjun Wang, Yueyang Cheng, Ruidong Chen, Wenhui Li, An-an Liu
Computer Vision
MTA-RL: Robust Urban Driving via Multi-modal Transformer-based 3D Affordances and Reinforcement Learning
Guangli Chen, Dianzhao Li, Wenjian Zhong et al. (+2 more)
Robust urban autonomous driving requires reliable 3D scene understanding and stable decision-making under dense interactions. However, existing end-to-end models lack interpretability, while modular pipelines suffer from error propagation across brittle interfaces. This paper proposes MTA-RL, the first framework that bridges perception and control through Multi-modal
Transformer-based 3D Affordances and Reinforcement Learning (RL). Unlike previous fusion models that directly regress actions, RGB images and
LiDAR point clouds are fused using a
transformer architecture to predict explicit, geometry-aware affordance representations. These structured representations serve as a compact observation space, enabling the RL policy to operate purely on predicted driving semantics, which significantly improves sample efficiency and stability. Extensive evaluations in CARLA Town01-03 across varying densities (20-60 background vehicles) show that MTA-RL consistently outperforms state-of-the-art baselines. Trained solely on Town03, our method demonstrates superior zero-shot generalization in unseen towns, achieving up to a 9.0% increase in Route Completion, an 11.0% increase in Total Distance, and an 83.7% improvement in Distance Per Violation. Furthermore, ablation studies confirm that our multi-modal fusion and reward shaping are critical, significantly outperforming image-only and unshaped variants, demonstrating the effectiveness of MTA-RL for robust urban autonomous driving.
Read the full abstract ▾
All authors Guangli Chen, Dianzhao Li, Wenjian Zhong, Bangquan Xie, Ostap Okhrin
Computer Vision
BathyFacto: Refraction-Aware Two-Media Neural Radiance Fields for Bathymetry
Markus Brezovsky, Anatol Günthner, Frederik Schulte et al. (+3 more)
Through-water photogrammetry from
UAV imagery enables shallow-water bathymetry, but refraction at the air–water interface violates the straight-ray assumption of Structure-from-Motion and causes systematic depth bias. We present BathyFacto, a refraction-aware two-media extension of Nerfacto in Nerfstudio for metrically consistent
underwater point clouds on simulated data. BathyFacto uses a shared hash-grid density field with a medium-conditioned color head and traces each camera ray as two segments: a straight air segment to a planar water surface and a refracted water segment computed using Snell's law and known refractive indices. A single proposal-network sampler operates on a virtual straight ray, while a kinked density wrapper corrects water-segment positions before density evaluation. Our pipeline converts photogrammetric reconstructions to Nerfstudio format, estimates the water plane from boundary markers, provides per-pixel medium masks, and supports refraction-corrected point-cloud export with reversible transforms to world and global frames. On a simulated scene with ground truth, BathyFacto achieves a Cloud-to-Mesh signed median deviation of $-0.001$,m and 85.7,% completeness at 0.2,m tolerance in the absolute global frame without rigid-body
alignment. This compares with $+1.370$,m / 11.6,% for Nerfacto and $+1.409$,m / 9.9,% for BathyFacto without refraction. Even after a naive refractive-index depth correction, both baselines remain offset by approximately 0.4,m. Unlike a refraction-corrected Multi-View Stereo reference, which is reliable mainly for near-nadir views, BathyFacto recovers consistent geometry across the full range of camera incidence angles.
Read the full abstract ▾
All authors Markus Brezovsky, Anatol Günthner, Frederik Schulte, Lukas Winiwarter, Boris Jutzi, Gottfried Mandlburger
Computer Vision
V-ABS: Action-Observer Driven Beam Search for Dynamic Visual Reasoning
Zhiwei Ning, Xuanang Gao, Jiaxi Cao et al. (+6 more)
Multimodal large
language models (MLLMs) have achieved remarkable success in general perception, yet complex multi-step visual
reasoning remains a persistent challenge. Although recent agentic approaches incorporate tool use, they often neglect critical execution feedback. Consequently, they suffer from the imagination-action-observer (IAO) bias, a misalignment between prior imagination and observer feedback that undermines
reasoning stability and optimality. To bridge this gap, we introduce V-ABS, an action-observer driven beam search framework that enables deliberate
reasoning through thinker-actor-observer iterations. We also propose an
entropy-based adaptive weighting
algorithm to mitigate the IAO bias by dynamically balancing the confidence scores between the policy priors and the observational feedback. Moreover, we construct a large-scale supervised fine-tuning (SFT)
dataset comprising over 80k samples to guide the model to assign higher prior confidence to correct action paths. Extensive experiments across eight diverse
benchmarks show that V-ABS achieves state-of-the-art performance, delivering an average improvement of 19.7% on the Qwen3-VL-8B baseline and consistent gains across both open-source and proprietary models.
Read the full abstract ▾
All authors Zhiwei Ning, Xuanang Gao, Jiaxi Cao, Gengming Zhang, Shengnan Ma, Wenwen Tong, Hanming Deng, Jie Yang, Wei Liu
Computer Vision
Task-Agnostic Noisy Label Detection via Standardized Loss Aggregation
Inhyuk Park and Doohyun Park
Noisy labels are common in large-scale medical imaging
datasets due to inter-observer variability and ambiguous cases. We propose a statistically grounded and task-agnostic framework, Standardized Loss Aggregation (SLA), for detecting noisy labels at the sample level. SLA quantifies label reliability by aggregating standardized fold-level validation losses across repeated cross-validation runs. This formulation generalizes discrete hard-counting schemes into a continuous
estimator that captures both the frequency and magnitude of performance deviations, yielding interpretable and statistically stable noisiness scores. Experiments on a public fundus
dataset demonstrate that SLA consistently outperforms the hard-counting baseline across all noise levels and converges substantially faster, especially under low noise ratios where subtle loss variations are informative. Samples with high SLA scores indicate potentially ambiguous or mislabeled cases, guiding efficient re-annotation and improving
dataset reliability for any classification task.
Read the full abstract ▾
Computer Vision
Active-SAOOD: Active Sparsely Annotated Oriented Object Detection in Remote Sensing Images
Yu Lin, Jianghang Lin, Kai Ye et al. (+2 more)
Reducing the annotation cost of oriented object detection in remote sensing remains a major challenge. Recently, sparse annotation has gained
attention for effectively reducing annotation redundancy in densely remote sensing scenes. However, (1) the sparse data reliance on class-dependent sampling, and (2) the lack of in-depth investigation into the characteristics of sparse samples hinders its further development. This paper proposes an active learning-based sparsely annotated oriented object detection (SAOOD) method, termed Active-SAOOD. Based on a model state observation module, Active-SAOOD actively selects the most valuable sparse samples at the instance level that are best suited to the current model state, by jointly considering orientation, classification, and localization uncertainty, as well as inter- and intra-class diversity. This design enables SAOOD to operate stably under completely randomly initialized sparse annotations and extends its applicability to broader real-world. Experiments on multiple
datasets demonstrate that Active-SAOOD significantly improves both performance and stability of existing SAOOD methods under various random sparse annotation. In particular, with only 1% annotated ratios, it achieves a 9% performance gain over the baseline, further enhancing the practical value of SAOOD in remote sensing. The code will be public.
Read the full abstract ▾
All authors Yu Lin, Jianghang Lin, Kai Ye, Shengchuan Zhang, Liujuan Cao
Computer Vision
MolSight: Molecular Property Prediction with Images
Aaditya Baranwal, Akshaj Gupta, Yogesh S Rawat et al. (+1 more)
Every molecule ever synthesised can be drawn as a 2D skeletal diagram, yet in modern property prediction this universally available representation has received less focus in favour of molecular graphs, 3D conformers, or billion-parameter
language models, each imposing its own computational and data-engineering overhead. We present $\textbf{MolSight}$, the first systematic large-scale study of
vision-based Molecular Property Prediction (MPP). Using 10
vision architectures, 7 pre-training strategies, and $2\,M$ molecule images, we evaluate performance across 10 downstream tasks spanning physical-property regression, drug-discovery classification, and
quantum-chemistry prediction. To account for the wide variation in structural complexity across pre-training molecules, we further propose a $\textbf{chemistry-informed curriculum}$: five structural complexity descriptors partition the corpus into five tiers of increasing chemical difficulty, consistently outperforming non-curriculum baselines. We show that a single rendered bond-line image, processed by a
vision encoder, is sufficient for competitive molecular property prediction, i.e. $\textit{chemical insight from sight alone}$. The best curriculum-trained configuration achieves the top result on $\textbf{5 of 10}$
benchmarks and top two on $\textbf{all 10}$, at $\textbf{$\textit{80$\times$ lower}$}$ FLOPs than the nearest multi-modal competitor.
Read the full abstract ▾
All authors Aaditya Baranwal, Akshaj Gupta, Yogesh S Rawat, Shruti Vyas
Computer Vision
Improving Temporal Action Segmentation via Constraint-Aware Decoding
Yeo Keat Ee, Debaditya Roy, Chen Li et al. (+2 more)
Temporal action
segmentation (TAS) divides untrimmed
videos into labeled action segments. While fully supervised methods have advanced the field, challenges such as action variability, ambiguous boundaries, and high annotation costs remain, especially in new or low-resource domains. Grammar-based approaches improve
segmentation with structural priors but rely on complex parsing limiting scalability. In this work, we propose a lightweight, constraint-based refinement framework that enhances TAS predictions by integrating statistical structural priors such as transition confidence, action boundary sets, and per-class duration, that can be directly extracted from annotated data. These constraints are integrated into a modified Viterbi decoding
algorithm, allowing inference-time refinement without retraining or added model complexity. Our approach improves both fully and semi-supervised TAS models by correcting structural prediction errors while maintaining high efficiency. Code is available at https://github.com/LUNAProject22/CAD
Read the full abstract ▾
All authors Yeo Keat Ee, Debaditya Roy, Chen Li, Hao Zhang, Basura Fernando
Computer Vision
MicroViTv2: Beyond the FLOPS for Edge Energy-Friendly Vision Transformers
Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu et al. (+2 more)
The
Vision Transformer (ViT) achieves remarkable accuracy across visual tasks but remains computationally expensive for edge deployment. This paper presents MicroViTv2, a lightweight
Vision Transformer optimized for real-device efficiency. Built upon the original MicroViT, the proposed model is designed based on reparameterized design, specifically Reparameterized Patch
Embedding (RepEmbed) and Reparameterized Depth-Wise convolution mixer (RepDW) for faster inference, and introduces the Single Depth-Wise Transposed
Attention (SDTA) to capture long-range dependencies with minimal redundancy. Despite slightly higher FLOPs, MicroViTv2 improves accuracy up to 0.5% compared to its predecessor and surpassing MobileViTv2, EdgeNeXt, and EfficientViT while maintaining fast inference and high energy efficiency on Jetson AGX Orin. Experiments on ImageNet-1K and COCO demonstrate that hardware-aware design and structural re-parameterization are key to achieving high accuracy and low energy consumption, validating the need to evaluate efficiency beyond FLOPs. Code is available at https://github.com/novendrastywn/MicroViT.
Read the full abstract ▾
All authors Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu, Wen-Kai Kuo, Jun-Wei Hsieh
Computer Vision
Scaling Vision Models Does Not Consistently Improve Localisation-Based Explanation Quality
Mateusz Cedro and Marcin Chlebus
Artificial intelligence models are increasingly scaled to improve predictive accuracy, yet it remains unclear whether scale improves the quality of post-hoc explanations. We investigate this relationship by evaluating 11 computer
vision models representing increasing levels of depth and complexity within the ResNet, DenseNet, and
Vision Transformer families, trained from scratch or pretrained, across three image
datasets with ground-truth
segmentation masks. For each model, we generate explanations using five post-hoc explainable AI methods and quantify mask
alignment using two localisation metrics: Relevance Rank Accuracy (Arras et al., 2022) and the proposed Dual-Polarity Precision, which measures positive attributions inside the class mask and negative attributions outside it. Across
datasets and methods, increasing architectural depth and parameter count does not improve explanation quality in most statistical comparisons, and smaller models often match or exceed deeper variants. While pretraining typically improves predictive performance and increases the dependence of explanations on learned weights, it does not consistently increase localisation scores. We also observe scenarios in which models achieve strong predictive performance while localisation precision is near zero, suggesting that performance metrics alone may not indicate whether predictions are based on the annotated regions. These results indicate that larger models do not reliably provide higher-quality explanations, and that explainability should therefore be assessed explicitly during model selection for safety-sensitive deployments.
Read the full abstract ▾
Computer Vision
Thermal-Det: Language-Guided Cross-Modal Distillation for Open-Vocabulary Thermal Object Detection
Yasiru Ranasinghe, Elim Schenck, Florence Yellin et al. (+3 more)
Existing open-vocabulary detectors focus on RGB images and fail to generalize to thermal imagery, where low texture and emissivity variations challenge RGB-based semantics. We present Thermal-Det, the first large
language model (LLM) supervised open-vocabulary detector tailored for thermal images. To enable large-scale training, we develop a synthetic
dataset by converting GroundingCap-1M into the thermal domain and filtering captions to remove RGB-specific terms, yielding over one million thermally aligned samples with bounding boxes, grounding texts, and detailed captions. Thermal-Det jointly optimizes detection, captioning, and cross-modal
distillation objectives. A frozen RGB teacher provides
geometric and semantic pseudo-supervision for paired but unlabeled RGB-thermal data, transferring open-vocabulary knowledge without manual annotation. The model further employs a Thermal-Text
Alignment Head for text calibration and a Modality-Fused Cross-
Attention module for dual-modality
reasoning. Unlike prior domain-adaptation methods, the detector is fully fine-tuned to internalize thermal contrast patterns while preserving
language alignment. Experiments on public
benchmarks show consistent 2-4% AP gains over existing open-vocabulary detectors, establishing a strong foundation for scalable,
language-driven thermal perception.
Read the full abstract ▾
All authors Yasiru Ranasinghe, Elim Schenck, Florence Yellin, Shuowen Hu, Christopher Funk, Vishal M. Patel
Computer Vision
Fashion130K: An E-commerce Fashion Dataset for Outfit Generation with Unified Multi-modal Condition
Yu He, Ting Zhu, Yichun Liu et al. (+6 more)
Recent research work on fashion outfit generation focuses on promoting visual consistency of garments by leveraging key information from reference image and text
prompt. However, the potential of outfit generation remains underexplored, requiring comprehensive e-commercial
dataset and elaborative utilization of multi-modal condition. In this paper, we propose a brand-new e-commerce
dataset, named Fashion130k, with various occasions, models, and garment types. For the consistent generation of garment, we design a framework with Unified Multi-modal Condition (UMC) to align and integrate the text and visual
prompts into generation model. Specifically, we explore an
embedding refiner to extract the unified
embeddings of multi-modal
prompts, within which a Fusion
Transformer is proposed to align the multi-modal
embeddings by adjusting the modality gap between text and image. Based on unified
embeddings, the
attention in generation model is redesigned to emphasis the correlations between
prompts and noise image, inducing that the noise image can select the pivotal
tokens of
prompts for consistent outfit generation. Our
dataset and proposed framework offer a general and nuanced exploration of multi-modal
prompts for generation models. Extensive experiments on real-world applications and
benchmark demonstrate the effectiveness of UMC in visual consistency, achieving promising result than that of SoTA methods.
Read the full abstract ▾
All authors Yu He, Ting Zhu, Yichun Liu, Lichen Ma, Xinyuan Shan, Jingling Fu, Yu Shi, Junshi Huang, Yan Li
Computer Vision
MicroWorld: Empowering Multimodal Large Language Models to Bridge the Microscopic Domain Gap with Multimodal Attributed Property Graph
Manyu Li, Ruian He, Chenxi Ma et al. (+2 more)
Multimodal large
language models (MLLMs) show remarkable potential for scientific
reasoning, yet their performance in specialized domains such as microscopy remains limited by the scarcity of domain-specific training data and the difficulty of encoding fine-grained expert knowledge into model parameters. To bridge the gap, we introduce MicroWorld, a framework for structured-knowledge augmentation with a complementary visual
retrieval pathway, using a
multimodal attributed property graph (MAPG) without domain-specific fine-tuning. MicroWorld extracts biomedical entities and relations via scispaCy or LLM-based triplet mining, aligns images and entities in a shared
embedding space using Qwen3-VL-
Embedding, and assembles a knowledge graph comprising approximately 106K nodes and 2.62M typed edges spanning eight relation categories. At inference time, a graph-augmented
retrieval pipeline matches query entities to the MAPG and injects structured knowledge context into the MLLM
prompt. On MicroVQA, MicroWorld raises Qwen3-VL-8B-Instruct accuracy from 48.8% to 67.1% (+18.3 percentage points; 37.5% relative), exceeding zero-shot GPT-5 (59.4%). The identical retrieved context raises GPT-5 to 71.5%, and GPT-5's own entity extraction reaches 73.3%. On MicroBench, 8B Single-Pass improves overall accuracy by 4.0 percentage points. The benefits are greatest for knowledge-intensive
reasoning; fine-grained visual discrimination and the additional visual-pathway contribution remain limited. Code and accompanying resources: https://github.com/ieellee/MicroWorld.
Read the full abstract ▾
All authors Manyu Li, Ruian He, Chenxi Ma, Weimin Tan, Bo Yan
Computer Vision
Think as Needed: Geometry-Driven Adaptive Perception for Autonomous Driving
Donghyun Kim and Jaehyoung Park
Autonomous driving scenes range from empty highways to dense intersections with dozens of interacting road users, yet current 3D detection models apply a fixed computation budget to every frame, wasting resources on simple scenes while lacking capacity for complex ones. Existing approaches compound this problem:
Transformer-based interaction models scale quadratically with the number of detected objects, and frame-by-frame processing causes the system to immediately forget objects the moment they become occluded. We propose Enhanced HOPE, an adaptive perception architecture that measures the
geometric complexity of each incoming
LiDAR frame using an unsupervised statistical
estimator and routes it through a shallow or deep processing path accordingly, requiring no manual scene labels. To keep interaction modeling efficient, we replace quadratic pairwise
attention with a linear-time subspace-based network that groups nearby objects into clusters and processes them jointly. The computational savings from these two mechanisms free up resources for a persistent temporal
memory module that retains previously detected objects and traffic rules across frames, enabling the system to recall occluded objects seconds after they disappear from view. On the nuScenes and CARLA
benchmarks, Enhanced HOPE reduces latency by 38% on simple scenes with no accuracy loss, improves mean Average Precision by 2.7 points on rare long-tail scenarios, and tracks objects through occlusions lasting over 5 seconds, where all tested baselines fail.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
CFSPMNet: Cross-subject Fourier-guided Spatial-Patch Mamba Network for EEG Motor Imagery Decoding in Stroke Patients
Xiangkai Wang, Yun Zhao, Dongyi He et al. (+5 more)
Motor imagery electroencephalography (MI-
EEG) decoding offers a non-invasive route for post-stroke rehabilitation, but cross-patient use remains difficult because pathological neural reorganization changes task-related
EEG dynamics, aperiodic activity, local excitability, cross-regional coordination, and trial-level
brain-state context. This makes source-learned MI representations unreliable for unseen patients. To address this problem, we propose CFSPMNet, a cross-patient adaptation framework that models post-stroke MI-
EEG as latent neural-state organization. CFSPMNet combines a
Fourier-Reorganized State Mamba Network (FRSM) with Shared-Private Prototype Matching (SPPM). FRSM represents each trial as a latent physiological
token sequence, reorganizes
token states in the
Fourier domain, and uses
Fourier-derived trial context to guide Mamba state-space propagation. SPPM improves target pseudo-label updating by combining semantic confidence with shared-private physiological consistency, filtering confident but physiologically inconsistent target predictions. Leave-one-subject-out experiments on two stroke MI-
EEG datasets show that CFSPMNet outperforms representative CNN-,
Transformer-, Mamba-, and adaptation-based baselines, achieving average accuracies of 68.23% on XW-Stroke and 73.33% on 2019-Stroke, with gains of 5.63 and 8.25 percentage points over the strongest competitors. Ablation, sensitivity, feature-
alignment, pseudo-label selection, and neurophysiological visualization analyses further support the roles of
Fourier-domain
token-state reorganization and calibrated pseudo-label updating. These results suggest that latent neural-state modeling can improve rehabilitation-oriented cross-patient BCI decoding. Code is available at https://github.com/wxk1224/CFSPMNet.
Read the full abstract ▾
All authors Xiangkai Wang, Yun Zhao, Dongyi He, Qingling Xia, Gen Li, Xinlai Xing, Yuchi Pan, Bin Jiang
Computer Vision
ViSRA: A Video-based Spatial Reasoning Agent for Multi-modal Large Language Models
Tingshu Mou, Jiabo He, Renying Wang et al. (+5 more)
Recent advances in Multi-modal Large
Language Models (MLLMs) target 3D spatial intelligence, yet the progress has been largely driven by post-training on curated
benchmarks, leaving the inference-time approach relatively underexplored. In this paper, we take a training-free perspective and introduce ViSRA, a human-aligned
Video-based Spatial
Reasoning Agent, as a framework to probe the spatial
reasoning mechanism of MLLMs. ViSRA elicits spatial
reasoning in a modular and extensible manner by leveraging explicit spatial information from expert models, enabling a plug-and-play flexible paradigm. ViSRA offers two key advantages: (1) human-aligned and transferable 3D understanding rather than task-specific overfitting; and (2) no post-training computational cost along with heavy manual curation of spatial
reasoning datasets. Experimental results demonstrate consistent improvement across a set of MLLMs on both existing
benchmarks and unseen 3D spatial
reasoning tasks, with ViSRA outperforming baselines by up to a 15.6% and 28.9% absolute margin respectively.
Read the full abstract ▾
All authors Tingshu Mou, Jiabo He, Renying Wang, Ce Liu, Hao Yang, Tiehua Zhang, Jingjing Chen, Xingjun Ma
Computer Vision
HYPERPOSE: Hyperbolic Kinematic Phase-Space Attention for 3D Human Pose Estimation
Vinduja Thekkath, Ashish Musale, Ajay Waghumbare et al. (+1 more)
We introduce HYPERPOSE, a novel 3D human pose estimation framework that performs spatio-temporal
reasoning entirely within the Lorentz model of
hyperbolic space $\mathbb{H}^d$ to natively preserve the hierarchical tree
topology of the human skeleton. Current state-of-the-art pose
estimators aim to capture complex joint dynamics by relying on
transformers and graph convolutional networks. Since these architectures operate exclusively in Euclidean space which fundamentally mismatches the inherent tree structure of the human body, these methods inevitably suffer from exponential volume distortion and struggle to maintain structural coherence. To this end, we depart from flat spaces and aim to improve
geometric fidelity with
Hyperbolic Kinematic Phase-Space
Attention (HKPSA), natively
embedding complex joint relationships without distortion, alongside a multi-scale windowed
hyperbolic attention mechanism that efficiently models temporal dynamics in $O(TW)$ complexity. Furthermore, to overcome the well-known instability of training non-Euclidean
manifolds, HYPERPOSE introduces a novel Riemannian loss suite and an uncertainty-weighted curriculum, enforcing physical geodesic constraints like bone length and velocity consistency. Extensive evaluations on the Human3.6M and MPI-INF-3DHP
datasets demonstrate that HYPERPOSE achieves state-of-the-art structural and temporal coherence, significantly reducing both volume distortion and velocity error, while establishing new state-of-the-art
benchmarks in overall positional accuracy.
Read the full abstract ▾
All authors Vinduja Thekkath, Ashish Musale, Ajay Waghumbare, Upasna Singh
Computer Vision
Initiation of Interaction Detection Framework using a Nonverbal Cue for Human-Robot Interaction
Guhnoo Yun, Juhan Yoo, Kijung Kim et al. (+1 more)
This paper describes an initiation of interaction(IoI) detection framework without keywords for human-
robot interaction(HRI) based on audio and
vision sensor fusion in a domestic environment. In the proposed framework, the
robot has its own audio and
vision sensors, and can employ external
vision sensor for stable human detection and tracking. When the user starts to speak while looking at the
robot, the
robot can localize his or her position by its sound source localization together with human tracking information. Then the
robot can detect the IoI if it perceives the face of the speaker faces the
robot. In case that the user does not speak directly, the
robot can also detect the IoI if he or she looks at the
robot for more than predefined periods of time. A state transition model for the proposed IoI detection framework is designed and verified by experiments with a mobile
robot. In order to implement and associate our model in a
robot architecture, all the components are implemented and integrated in the
Robot Operating System(ROS) environment.
Read the full abstract ▾
All authors Guhnoo Yun, Juhan Yoo, Kijung Kim, Dong Hwan Kim
Computer Vision
SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generation
Liangyang Ouyang, Ruicong Liu, Caixin Kang et al. (+2 more)
Video generation has advanced rapidly, producing photorealistic
videos from text or image
prompts. Meanwhile, film production and social robotics increasingly demand multi-person
videos with rich social interactions, including conversations, gestures, and coordinated actions. However, existing models offer no explicit control over interactions, such as who performs which action, when it occurs, and toward whom it is directed. This often results in wrong person performing unintended actions (actor-action mismatch), disordered social dynamics, and wrong action targets. To address these challenges, we present SocialDirector, a training-free interaction controller that enhances the generation model by modulating cross-
attention maps. SocialDirector contains two modules: Social Actor Masking and Directional Reweighting. Social Actor Masking constrains each person's visual
tokens to attend only to their own textual descriptions via a spatiotemporal mask, avoiding actor-action mismatch and disordered social dynamics. Directional Reweighting amplifies
attention to directional words (e.g., "leftward", "right"), leading each action towards its intended target. To evaluate generated social interactions, we annotate existing
datasets with interaction descriptions and build a fully automated evaluation pipeline powered by open-source VLMs. Experiments on different
video generation models show that SocialDirector significantly improves interaction fidelity and approaches the upper bound set by real
videos.
Read the full abstract ▾
All authors Liangyang Ouyang, Ruicong Liu, Caixin Kang, Yifei Huang, Yoichi Sato
Computer Vision
MFVLR: Multi-domain Fine-grained Vision-Language Reconstruction for Generalizable Diffusion Face Forgery Detection and Localization
Yaning Zhang, Tianyi Wang, Zan Gao et al. (+3 more)
The swift advancement in photo-realistic face generation technology has sparked considerable concerns across society and academia, emphasizing the requirement of generalizable face forgery detection and localization methods. Prior works tend to capture face forgery patterns across multiple domains using image modality, other modalities like fine-grained texts are not comprehensively investigated, which restricts the generalization capability of models. Besides, they usually analyze facial images created by GAN, but struggle to identify and localize those synthesized by
diffusion. To solve the problems, in this paper, we devise a novel multi-domain fine-grained
vision-
language reconstruction (MFVLR) model, which explores comprehensive and diverse visual forgery traces via
language-guided face forgery representation learning, to achieve generalizable
diffusion-synthesized face forgery detection and localization (DFFDL). Specifically, we devise a fine-grained
language transformer that studies general fine-grained
language embeddings using
language reconstruction. We propose a multi-domain
vision encoder to capture general and complementary visual forgery patterns across the image and residual domains. A
vision decoder is designed to reconstruct image appearance and achieve forgery localization. Besides, we propose an innovative plug-and-play
vision injection module to enhance the interaction between the
vision and
language embeddings. Extensive experiments and visualizations demonstrate that our network outperforms the state of the art on different settings like cross-generator, cross-forgery, and cross-
dataset evaluations.
Read the full abstract ▾
All authors Yaning Zhang, Tianyi Wang, Zan Gao, Yibo Zhao, Chunjie Ma, Meng Wang
Computer Vision
Explanation-Aware Learning for Enhanced Interpretability in Biomedical Imaging
Zubair Faruqui and Rahul Dubey
Deep neural networks for medical image diagnosis often achieve high predictive accuracy while relying on spurious or clinically irrelevant visual cues, limiting their trustworthiness in practice. Post-hoc explanation methods are widely used to visualize model decisions in the form of saliency maps; however, these explanations do not influence how models learn during training, allowing non-causal or confounding features to persist. This motivates the incorporation of explanation supervision directly into the training objective to guide model
attention toward clinically meaningful regions and promote clinically grounded decision-making. This paper presents a systematic approach to integrate explanation loss into model training and analyzes how different explanation loss designs and supervision strengths influence both predictive performance and spatial faithfulness of explanations. To quantitatively assess interpretability, two complementary explanation performance metrics-annotation coverage and saliency precision-are introduced, enabling rigorous evaluation beyond qualitative visualization. Our experimental results reveal a clear trade-off between explanation quality and explanation loss coefficients. Furthermore, quantitative statistical analysis yields consistently improved explanation
alignment while maintaining comparable accuracy. Experiments were conducted on annotated chest X-ray
datasets; however, the proposed framework is applicable to a broad range of annotated biomedical imaging modalities. Overall, these findings demonstrate that explanation supervision is not a monolithic design choice and provide practical guidance for incorporating explanation loss into training objectives under noisy clinical annotations.
Read the full abstract ▾
Computer Vision
EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs
Jiameng Li, Minye Wu, Jiezhang Cao et al. (+2 more)
Long-form
video understanding remains challenging for
Video Large
Language Models (VideoLLMs), as the dense frame sampling introduces massive visual
tokens while sparse sampling risks missing critical temporal evidence and leading to LLM
hallucination. Existing training-free
token reduction methods either treat
videos equally as static images or rely on segment-level merging heuristics, which weaken fine-grained spatiotemporal modeling and introduce additional overhead. In this paper, we propose EchoPrune, a lightweight and training-free
token pruning method that improves temporal resolution under a fixed LLM-side visual
token budget. Our core idea is to interpret redundant
video tokens as temporal echoes: if a
token is well reconstructed from the previous frame, it is merely a temporally redundant echo; otherwise, it may capture new events, motion, or query-relevant visual evidence. Based on this insight, EchoPrune scores visual
tokens by (i) query-guided crossmodal relevance and (ii) temporal reconstruction error, measured by correspondence matching and echo matching across consecutive frames. The selected
tokens preserve task-relevant cues and temporal novelty while suppressing predictable redundancy, allowing VideoLLMs to observe more frames without increasing the decoding budget. Extensive experiments on LLaVA-OV, Qwen2.5VL, and Qwen3VL across six
video understanding
benchmarks show that EchoPrune enables VideoLLMs to process up to 20x frames under the same
token budget, yielding improved performance (+8.6%) and inference speedup (5.6x for prefilling) on Qwen2.5VL-7B.
Read the full abstract ▾
All authors Jiameng Li, Minye Wu, Jiezhang Cao, Aleksei Tiulpin, Matthew B. Blaschko
Computer Vision
PixelFlowCast: Latent-Free Precipitation Nowcasting via Pixel Mean Flows
Yufeng Zhu, Chunlei Shi, Yongchao Feng et al. (+1 more)
Precipitation nowcasting aims to forecast short-term radar echo sequences for extreme weather warning, where both prediction fidelity and inference efficiency are critical for real-world deployment. However,
diffusion-based models, despite their strong generative capability, suffer from slow inference due to multi-step sampling trajectories, limiting their practical usability. Conditional Flow Matching (CFM) improves efficiency via straightened trajectories, but relies on latent space compression, which inevitably discards high-frequency physical details and degrades fine-grained prediction quality. To address these limitations, we propose PixelFlowCast, a two-stage probabilistic
forecasting framework that achieves both high-efficiency and high-fidelity prediction without latent compression. Specifically, in the first stage, a deterministic model first produces coarse forecasts to capture global
evolution trends. In the subsequent stage, the proposed KANCondNet extracts deep spatiotemporal
evolution features to provide accurate conditional guidance. Based on this, a latent-free, few-step Pixel Mean Flows (PMF) predictor employs an $x$-prediction mechanism to generate high-quality predictions, effectively preserving fine-grained structures while maintaining fast inference. Experiments on the publicly available SEVIR
dataset demonstrate that PixelFlowCast outperforms existing mainstream methods in both prediction accuracy and inference efficiency, particularly for long sequence
forecasting, highlighting its strong potential for real-world operational deployment.
Read the full abstract ▾
All authors Yufeng Zhu, Chunlei Shi, Yongchao Feng, Dan Niu