Computer Vision
Multimodal Scenario Similarity Search for Autonomous Driving
Tamás Matuszka, András Tamásy and Balázs Szolár
Large-scale autonomous-driving
datasets contain vast numbers of recorded scenarios, creating a need for efficient
retrieval methods that can identify situations similar to a given query. Existing approaches typically rely on either visual representations or motion-based descriptions, making it difficult to understand their relative strengths and limitations for scenario
retrieval. In this work, we present a
multimodal framework for autonomous-driving scenario
retrieval that combines visual and trajectory-based representations within a unified
retrieval pipeline. We investigate two trajectory-based approaches: Exo-Trajectory, an explicit matching method based on surrounding-
agent motion, and ScenarioFormer, a
transformer-based representation learned from object trajectories using contrastive learning. We compare these approaches against strong
vision-based baselines and analyze their behavior across a diverse set of driving scenarios. Experimental results show that trajectory representations provide strong
retrieval performance for motion-centric events such as cut-ins, turning maneuvers, and traffic queueing, while visual
embeddings excel when appearance cues are informative. Most importantly, combining visual and trajectory information consistently improves
retrieval quality, yielding the best overall performance. These findings demonstrate that appearance and motion capture are complementary notions of scenario similarity and motivate
multimodal retrieval systems for autonomous-driving data mining,
dataset curation, and scenario-based validation.
Read the full abstract ▾
Computer Vision
SVF-CR: Synchronized Visual-Facial Cross-Refinement for Multimodal Ambivalence and Hesitancy Recognition
Hyein Park, Namho Kim and Junhwa Kim
Ambivalence and hesitancy are subtle behavioral states that are expressed through a combination of verbal content, facial behavior, visual context, and acoustic cues. Effective recognition therefore requires not only extracting informative unimodal representations, but also modeling how temporally aligned behavioral evidence interacts across modalities. In this paper, we propose a synchronized visual-facial cross-refinement framework (SVF-CR) with pairwise
multimodal evidence fusion for ambivalence and hesitancy recognition. The proposed method first extracts whole-
video segment
tokens and cropped-face segment
tokens using the same temporal partition. The synchronized visual and facial
tokens are refined through intra-modal self-
attention and bidirectional visual-facial cross-
attention, allowing whole-
video context and local facial behavior to mutually refine each other before evidence construction. We then construct segment-level visual-facial evidence using consistency and discrepancy modeling, followed by temporal self-
attention and
attention pooling. Textual and acoustic features are lightly refined through context self-
attention and are fused with the enhanced visual-facial evidence at the final decision stage using pairwise evidence fusion. Experiments on the BAH (Behavioral Ambivalence/Hesitancy) public evaluation split show that the proposed synchronized visual-facial cross-refinement improves public macro-F1 over both global visual-face
token fusion and synchronized evidence baselines, achieving a public macro-F1 of 0.7156. Code is available at : https://github.com/hiinnnii/BAH-Challenge-ECCV2026\_SVF-CR.
Read the full abstract ▾
Computer Vision
CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation
Seungyong Lee, Hyun Jun Jang, Sangoh Kim et al. (+1 more)
Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person. Yet most systems give the user little control over how a garment should be worn – its size (loose or fitted), style (e.g., tucked in or untucked, open or closed), and spatial placement on the body. We address this gap with two complementary contributions. First, we define and solve Visual-Instance-
Prompt Segmentation via VIP-SAM: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level
segmentation. Second, we introduce CtrlVTON, a controllable VTO framework that recasts try-on as an image editing problem and adds
segmentation masks as pixel-level control over garment layout, including style, size, and spatial placement on the body. VIP-SAM and CtrlVTON each achieve state-of-the-art results on their respective tasks. In particular, CtrlVTON generates images that follow user-provided layouts far more faithfully than the strongest proprietary editing systems while matching them on garment fidelity.
Read the full abstract ▾
All authors Seungyong Lee, Hyun Jun Jang, Sangoh Kim, Sungjoon Park
Computer Vision
Simon-SR: Spatially Adaptive Modulation and Visual Prompt Adaptation for Text-Reinforced Super-Resolution
Haotong Cheng, Yuxuan Li and Zijie Cui
Severe downsampling makes single image super-resolution (SISR) an ill-posed problem, in which the
language-guided multi-modal methods are especially vulnerable to erroneous priors and require costly textual annotations. To address these issues, we propose Simon-SR, a multi-modal SISR framework leveraging learnable
prompts for efficient semantic mining and robust text-image fusion. Simon-SR treats textual semantics as learnable latent variables. Specifically, the Contrastive
Prompt Learning (CPL) mines instance-level semantics from unannotated images with frozen CLIP encoders, and
Prompt-Guided Spatially Adaptive Refinement (PSAR) injects them through
attention-gated multi-modal fusion. On CUB and COCO2017 at $\times4$ and $\times16$, Simon-SR gains up to 0.50 dB PSNR and 0.0133 SSIM over current SOTA. The project is available at https://github.com/CHT05017/ICAIS26-SimonSR
Read the full abstract ▾
Computer Vision
Dynamic Inverse Rendering for Enhanced Material-Lighting Decomposition
Raza Yunus, Benjamin Ummenhofer, Jan Eric Lenssen et al. (+1 more)
Decomposing outgoing surface radiance into material and illumination during inverse rendering is essential for applications such as relighting and augmented reality, yet it is severely ill-posed since multiple combinations can result in the same observed colour. Capturing an object under multiple lighting conditions usually helps resolve this ambiguity as it constrains the optimization towards correct solutions. In this work, we explore the potential of reconstructing rigidly moving objects – which provides observations of diverse light-surface interactions – to resolve the material-lighting ambiguity in inverse rendering. For this purpose, we introduce a relightable approach that marries object tracking and reconstruction with inverse rendering for general rigidly moving objects. Our experimental analysis on synthetic data demonstrates that motion can be an advantage for disentangling material and lighting: the reconstructed material is significantly more accurate when the object is observed under rigid motion than when it is static. Moreover, results on RGB
videos of real hand-held objects show that our pipeline preserves this advantage even under noisy real-world conditions.
Read the full abstract ▾
All authors Raza Yunus, Benjamin Ummenhofer, Jan Eric Lenssen, Eddy Ilg
Computer Vision
From Classification to Localization and Clinical Validation: Large-Scale Development of a Deep Learning System for Thoracic Disease Detection on Chest Radiographs in Thailand
Isarun Chamveha, Tretap Promwiset, Napat Wanchaitanawong et al. (+3 more)
Chest radiography (CXR) remains the most widely used thoracic imaging modality, yet expert interpretation is constrained by a severe shortage of radiologists in Thailand and across Southeast Asia. Local adaptation of deep learning models to Thai data has been shown to substantially improve accuracy on Thai populations. Here we present the development and comprehensive validation of the chest radiograph analysis model in Inspectra CXR version 5, a deep learning system that performs multi-label thoracic disease classification and weakly supervised lesion localization within a single model. The architecture couples a DenseNet-121 backbone with Attend-and-Compare Modules (ACM) and a Probabilistic Class Activation Map (PCAM) aggregation layer, producing a per-condition classification score and heatmap simultaneously. The model was developed on 874,858 frontal chest radiographs with paired radiologist reports from Siriraj Hospital, Bangkok. On a held-out, radiologist-verified in-domain test set of 19,871 cases, it achieved a mean AUROC of 0.994 (mean sensitivity 92.4%, specificity 98.6%) across nine clinically important conditions. On an independent generalization set of 5,992 cases from 13 hospitals across Thailand, the mean AUROC was 0.970, indicating robust transfer across sites. For localization, evaluated on 4,549 radiologist-annotated cases, the model attained a mean lesion-localization fraction (LLF) of 77.9% at 0.59 non-lesion localizations per image. In a usability evaluation with five thoracic radiologists, the system reached a classification concordance of 93.6%, a localization concordance of 94.7%, and a mean System Usability Scale (SUS) score of 89. These results indicate that a locally developed, localization-capable CXR system can deliver high accuracy, generalize across heterogeneous Thai hospitals, and earn the trust of practicing radiologists.
Read the full abstract ▾
All authors Isarun Chamveha, Tretap Promwiset, Napat Wanchaitanawong, Trongtum Tongdee, Pairash Saiviroonporn, Warasinee Chaisangmongkon
Computer Vision
TextileNet: Towards Zero-shot Text-style Segmentation of Manuscripts
Anguelos Nicolaou, Antonella Ambrosio, Desiree Di Donato et al. (+1 more)
Automatic writer identification systems have progressed remarkably in recent years, yet their deployment in archival paleography remains limited by the scarcity of labeled training data, open scribe sets, and degraded image quality. We present TextileNet, a fully convolutional multi-task network trained exclusively on synthetic data to produce dense pixel-level texture
embeddings, which we transfer zeroshot to historical manuscript analysis. As an original contribution to evaluation methodology, we designed a paleographic visual quiz of 80 pair and triplet questions and administered it to a range from lay participants to senior paleographers under strict anonymity, establishing to our knowledge for the first time a human baseline for script-style discrimination on late medieval text. We employ TextileNet
embeddings to perform zero-shot
retrieval on sub-word granularity for hand and gender identification. Our experimental results help in building the credibility of TextileNet in the paleographic domain, but more than that demonstrate in experimental terms that the question of gender in handwriting needs to be treated with caution.
Read the full abstract ▾
All authors Anguelos Nicolaou, Antonella Ambrosio, Desiree Di Donato, Georg Vogeler
Computer Vision · Cross-listed from cs.CL
Letter Lemmatization: One-to-one and Banded RNNs for Reversing Character-Set Simplification and Abbreviation in Medieval Text
Anguelos Nicolaou, Maria Pia Tiseo, Tamas Kovacs et al. (+2 more)
Medieval document transcribers have very different practices; on top of that, heterogeneous digitization policies have resulted in corpora where the character-set must be viewed as fluid. In this paper we address the problem of changing between character-sets in a flexible manner. We focus on one-to-one character mappings and train characterlevel one-to-one RNNs to undo them with self-supervision; recovering half the CER even with 20 text lines. We analyse the use of these one-to-one networks for HTR post-correction and we see that they obtain significant improvements while totally ignoring ins-dels. We then use the exact same networks with character-level
alignment groundtruth compiled from parallel corpora in a training and inference mode we call Banded RNNs. We use such networks to successfully expand abbreviations in medieval charter transcriptions. Finally we introduce an elaborate heuristic which takes the characters of two arbitrary character-sets and defines a metric encapsulating what we consider to be semantic similarity of characters. We call the construction of such mappings letter lemmatization and present a rich Python library that efficiently performs all presented methods.
Read the full abstract ▾
All authors Anguelos Nicolaou, Maria Pia Tiseo, Tamas Kovacs, Nicolas Renet, Georg Vogeler
Computer Vision
Rethinking Monocular Depth Embedding for Generalized Stereo Matching
Libo Lin, Shuangli Du, Minghua Zhao et al. (+3 more)
Generally,
monocular methods capture rich contextual priors but lack
geometric precision, whereas stereo methods are geometrically accurate yet struggle in textureless and occluded regions. Several approaches attempt to combine their strengths to enhance the generalization of stereo matching (SM) by aligning
monocular depth with stereo information. However, establishing a stable and generalizable
alignment is challenging, and unreliable
monocular cues can substantially degrade performance. This paper rethinks
monocular depth
embedding. First, to prevent shortcut learning, we reduce branch coupling instead of expanding network width. Second, we construct soft constraints instead of hard ones from
monocular depth to improve tolerance to
monocular depth errors. Based on the principles, we integrate
monocular information into both feature extraction and GRU iterations. Specifically, the
monocular depth map is fused with the RGB image to sharpen depth boundary perception and suppress matching ambiguities. The fused image is then used for feature extraction, allowing the contextual features to encode global
geometric information. Furthermore, the
monocular depth gradient feature is employed to guide disparity updates, helping to escape local oscillations. Finally, to address the boundary blurring of supervised disparity caused by data augmentation, we propose an edge confidence estimation method and an edge-aware loss function. Our method achieves state-of-the-art (SOTA) performance on multiple standard
benchmarks, demonstrating excellent generalization while improving accuracy. The code is available at https://github.com/linliboabc-maker/stereo-matching-digital.
Read the full abstract ▾
All authors Libo Lin, Shuangli Du, Minghua Zhao, Zhenzhen You, Shun Lv, Yiguang Liu
Computer Vision
REMIND: RE-Identification with Memory for INDoor Navigation
Pablo Diaz-Pereda, Alejandro Rodriguez-Ramos, David Perez-Saura et al. (+1 more)
Mobile
robots operating indoors must re-identify previously observed objects after long temporal gaps, significant viewpoint changes, and severe illumination variations. This remains a challenging problem: multi-object tracking methods are optimized for short-term association of pedestrians and vehicles at
video rates, person and vehicle re-identification approaches lack persistent
memory mechanisms, and state-of-the-art
video object
segmentation techniques rely on reactive distractor filtering rather than enforcing global identity consistency. To address these limitations, we present REMIND, an online tracker designed for long-term multi-object re-identification of generic indoor objects from
monocular RGB imagery, requiring neither camera pose nor depth. Motivated by evidence from visual cognition that humans rely on accumulated appearance familiarity and spatial context rather than explicit self-localization, REMIND combines frozen DINOv3 features with a dual-bank multi-prototype appearance
memory, part- and background-level descriptors, a neighbour-context
reasoning module exploiting spatial co-occurrence, and joint Hungarian assignment with ambiguity-aware safeguards. On a purpose-built indoor
dataset featuring controlled revisits and dense same-class clutter, REMIND reaches 90.35% IDF1, nearly 20 points above a state-of-the-art
video object
segmentation baseline and more than 36 above a strong tracking-by-detection baseline. On ScanNet++, it attains the highest IDF1 in every setting but one, end-to-end detection over all scenes, where the tracking-by-detection baseline is marginally ahead while REMIND still associates and recovers identities more accurately; it also completes every scene, whereas the
video object
segmentation baseline exhausts GPU
memory on 66.9% under YOLO detections. The complete system, evaluation framework, and
dataset are publicly released.
Read the full abstract ▾
All authors Pablo Diaz-Pereda, Alejandro Rodriguez-Ramos, David Perez-Saura, Pascual Campoy
Computer Vision
Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval
Junmyeong Lee, Chan Hur, ChangSu Choi et al. (+5 more)
Sign
Language Retrieval (SLRet) enables efficient access to sign
language content but remains fragile in fine-grained scenarios where visually similar signs must be distinguished. We show that this limitation does not stem from model capacity, but from ineffective hard negative supervision. Specifically, we formulate fine-grained
retrieval failures as a negative distribution mismatch: semantically distinct yet visually confusable signs are rarely treated as hard negatives, while existing text-based mining strategies fail to capture such visual ambiguity. To address this issue, we propose Sign-Aware Hard Negative Mining (SAN), which constructs hard negatives based on visual confusability in the sign
embedding space rather than linguistic similarity. Experiments on PHOENIX-2014T demonstrate that SAN substantially improves fine-grained
retrieval performance while preserving coarse-grained accuracy, highlighting the importance of aligning negative supervision with visual ambiguity in sign
language retrieval.
Read the full abstract ▾
All authors Junmyeong Lee, Chan Hur, ChangSu Choi, Sukmin Cho, Fitsum Gaim, Eui Jun Hwang, Hoyun Song, KyungTae Lim
Computer Vision
AnythingReality: Robust Online Gaussian Splatting SLAM for Open-Vocabulary VR Scene Exploration
Timofei Kozlov, Dmitrii Maliukov, Andrey Marchenko et al. (+2 more)
We present a novel integrated architecture for robust online 3D Gaussian splatting, real-time VR exploration, and
speech-driven
Vision-
Language-Model interaction. Unlike methods assuming clean depth or external poses, our system combines ORB-SLAM3-based pose estimation with online Gaussian reconstruction for noisy real-world data. A VR pipeline enables immersive exploration of incremental reconstructions; a semantic module transcribes voice commands, generates scene descriptions, and records points of interest. Against state-of-the-art online Gaussian splatting methods, we improve image quality on our
dataset (+14.5% PSNR, +8.6% SSIM, -14.3% LPIPS) and TUM-RGBD (+11.7% PSNR, +7.8% SSIM, -21.6% LPIPS), with comparable or superior frame rates via quality-speed configurations. We achieve an 88% VLM object-recognition rate.
Read the full abstract ▾
All authors Timofei Kozlov, Dmitrii Maliukov, Andrey Marchenko, Miguel Altamirano Cabrera, Dzmitry Tsetserukou
Computer Vision · Cross-listed from cs.LG
All you need is SAMPAT
Jayadeva and Madhur Aswani
The current state of the art in AI/ML rests on deep neural architectures, which, in general, suffer from a lack of interpretability. Interpretability is crucial to gleaning insights while analyzing experimental data, where quantitative predictions may not be adequate for a scientist. We present a three layer neural architecture, SAMPAT (Smooth Approximation via Multivariate
Polynomials and Analytic Transformations), that can provably learn a continuous, everywhere differentiable function, that can approximate any smooth function arbitrarily closely. SAMPAT's approximant can be expressed as a closed and compact algebraic, analytic expression, providing complete interpretability. Experiments on synthetic and
benchmark datasets indicate that SAMPAT yields competitive performance with simpler representations. For many tasks, a two layer SAMPAT suffices. By imposing restrictions on the connectivity between
neurons, SAMPAT may be used to provide a range of approximants, including regular and trigonometric
polynomials, rational expressions, Gaussians, mixtures of Gaussians, as well as arbitrary combinations of the same; without restrictions, it learns a suitable structure. SAMPAT may be used to factorize
polynomials and model nonlinear systems. With the addition of skip connections, a 4 to 6 layer SAMPAT is adequate to represent a substantive range of methods widely used in AI/ML, allowing the choice of a model's family, not just its parameters, to also be optimized as part of the learning process.
Read the full abstract ▾
Computer Vision
Glob3R: Global Structure-from-Motion with 3D Foundation Models
Junyuan Deng, Heng Li, Kejie Qiu et al. (+7 more)
Recent 3D
geometric foundation models, such as VGGT, provide robust feed-forward 3D reconstruction by directly predicting camera poses and 3D scene points from input images. However, their results remain inaccurate, and scaling them to long sequences or large unordered image sets typically requires chunk-wise processing, which can introduce drift and inconsistency. We present Glob3R, a global SfM-style reconstruction built on 3D foundation models. Our key idea is to explicitly optimize feed-forward
geometric predictions. To this end, we augment a frozen Pi3X backbone with a lightweight dense matching head that predicts image warps between selected reference frames and neighboring views. These dense warps are converted into sparse but reliable multi-view feature tracks, which provide correspondence constraints for global optimization. We further introduce a keyframe-based sliding-window association strategy that propagates tracks and relative poses across overlapping windows, enabling scalable reconstruction. Finally, we perform global motion averaging and bundle adjustment to refine camera poses, reduce scale inconsistencies, and recover dense scene geometry. Extensive experiments on indoor, outdoor, large-scale driving, and unordered SfM
benchmarks demonstrate that Glob3R achieves robust and accurate reconstruction. It consistently improves over feed-forward foundation-model baselines and recent scalable reconstruction methods, while being more robust than classical SfM pipelines. The refined poses also lead to higher-quality neural rendering, validating the benefit of combining foundation-model priors with global
geometric optimization. Project page: https://junyuandeng.github.io/Glob3r
Read the full abstract ▾
All authors Junyuan Deng, Heng Li, Kejie Qiu, Lingteng Qiu, Rui Peng, Weichao Shen, Weihao Yuan, Siyu Zhu, Zilong Dong, Ping Tan
Computer Vision · Cross-listed from eess.IV
Joint-Embedding Predictive Architecture for Solar PV Panel Fault Classification
Seyyedhamid Azimidokht, Mehdi Monemi, Abdelhak Kharbouch et al. (+4 more)
The rapid expansion of
solar photovoltaic (PV) systems has increased the need for reliable and scalable fault classification, as manual inspection is impractical at scale. Thermal infrared (IR) imaging provides a non-contact solution for identifying PV faults; however, accurate classification remains challenging due to class imbalance, limited texture information, and subtle thermal differences. In this work, we investigate the applicability of Joint-
Embedding Predictive Architecture (JEPA) for thermal IR PV fault classification across various scenarios and propose JEFFNet (JEPA-EFFicientNet), a multibranch architecture that combines JEPA-based self-supervised representation learning with EfficientNetV2-S-based supervised convolutional feature extraction. JEFFNet fuses semantic representations from a JEPA-pretrained
Vision Transformer with convolutional features from EfficientNetV2-S, enabling complementary feature learning. JEFFNet is evaluated on two public thermal IR
datasets, PVF-10 and InfraredSolarModules (ISM), for both multiclass and derived binary (healthy/faulty) classification. On PVF-10, JEFFNet achieves an F1-score of $93.21$ and an accuracy of $94.33$ in the 10-class task, and an F1-score of $97.53$ and an accuracy of $96.41$ in the derived 2-class task. On ISM, JEFFNet achieves an F1-score of $72.60$ and an accuracy of $83.88$ in the 12-class task, and an F1-score of $94.69$ and an accuracy of $94.78$ in the derived 2-class task. JEFFNet also uses only 108.6M parameters versus 205.91M for GEPFNet, a 47.2% reduction. These results demonstrate that combining self-supervised semantic and supervised convolutional features provides an effective, parameter-efficient solution for thermal IR PV fault classification. The source code is publicly available at https://github.com/Azimi2kht/JEFFNet
Read the full abstract ▾
All authors Seyyedhamid Azimidokht, Mehdi Monemi, Abdelhak Kharbouch, Farid Hamzehaghdam, Mehdi Rasti, Jamshid Aghaei, Emil Kurvinen
Computer Vision
YeTI: You Only Need Two Noisy Images for Real-World sRGB Noise Generation
Jaekyun Ko, Byung Wan Lim, Soomin Lee et al. (+2 more)
Real-world sRGB image denoising remains challenging due to the nonlinear characteristics of sensor noise and the difficulty of acquiring aligned clean-noisy image pairs. Supervised denoisers often overfit to limited paired
datasets, while self-supervised methods still depend on sufficiently diverse noisy observations. These limitations motivate scalable noise synthesis methods that can model real-world noise without clean ground truth or camera metadata. We propose YeTI, a real-world sRGB noise generation framework that learns from only two noisy observations of the same scene. YeTI uses a Reconstruction Autoencoder to disentangle scene structure and noise characteristics, and models the latent noise distribution with a one-step Conditional
Diffusion Transformer trained using consistency objectives. Given a single noisy input at inference time, YeTI generates realistic, signal-dependent noise while preserving the underlying scene content. Extensive experiments demonstrate the effectiveness of YeTI across real-world
benchmarks. We evaluate noise generation on SIDD and further assess generalization on SIDD+, MAI2021, and SID, covering smartphone and diverse consumer-camera sensors. Downstream denoising results on DND further show that denoisers trained with YeTI-synthesized images achieve strong real-world performance, highlighting the practical value of clean-image-free and metadata-free noise generation.
Read the full abstract ▾
All authors Jaekyun Ko, Byung Wan Lim, Soomin Lee, Dongjin Kim, Tae Hyun Kim
Computer Vision
HiHR: Hierarchical Hyperbolic Representation for Aerial-Ground Person Re-Identification
Qiwei Yang and Pingping Zhang
Aerial-Ground Person Re-IDentification (AG-ReID) aims to retrieve the same person across heterogeneous aerial and ground camera platforms. Although great progress has been made, existing methods remain suboptimal due to the direct feature
alignment across views, overlooking view-specific cues. To address this issue, we propose a novel Hierarchical
Hyperbolic Representation (HiHR) framework for AG-ReID. More specifically, we first extract multi-granularity features based on pre-trained visual-text encoders. Then, we propose a Text-guided Multi-granularity Fusion (TMF) to fuse multi-granularity features and enhance the representation ability of identity features. Furthermore, we introduce the Hierarchical
Hyperbolic Learning (HHL) to construct a hierarchical feature structure in a
hyperbolic space. This hierarchy includes a coarse level that ensures identity separability and cross-view consistency, and a fine level that preserves view-specific discriminative cues. As a result, our proposed framework can effectively aggregate view-invariant and view-specific discriminative features for AG-ReID. Extensive experiments on four AG-ReID
benchmarks demonstrate the effectiveness of our framework. The source code is available at https://github.com/YangQiWei3/HiHR.
Read the full abstract ▾
Computer Vision
Causally Debiased Latent Action Model for Embodied Action-Conditioned World Models
Yufan Wei, Kun Zhou, Lingjun Mao et al. (+9 more)
Action-conditioned world models (ACWMs) aim to simulate future observations conditioned on
embodied actions. However, learning controllable ACWMs requires large-scale action-labeled data, which remains costly to collect. Latent action models (LAMs) mitigate this bottleneck by inferring latent actions from
videos without executable action labels, but existing LAMs are typically trained with reconstruction-only objectives and therefore entangle action-relevant dynamics with action-irrelevant visual confounders. With such objectives, LAM favors visual context over action dynamics, therefore downstream ACWMs exhibit residual motion under zero actions and fail to reproduce supplied dynamics. We propose CD-LAM, a causally debiased framework for LAM-based ACWMs. CD-LAM introduces three debiasing objectives: embodiment-centric reconstruction, action-centric contrastive learning, and latent space calibration, which together encourage embodiment-focused, action-aware, and well-calibrated representations. CD-LAM reduces action-following error by up to 42% and 35% when conditioned on latent and
robot actions, while improving fidelity, and matching DreamDojo reference with over 12$\times$ fewer adaptation updates. CD-LAM also adapts LTX-2.3-22B into an ACWM with competitive quality using only 3.2% of DreamDojo-14B's sample exposures. On X-VLA, pretraining with CD-LAM instead of baseline LAM consistently improves success rates by up to 5 percent point across LIBERO, LIBERO-Plus, and RoboTwin C2R.
Read the full abstract ▾
All authors Yufan Wei, Kun Zhou, Lingjun Mao, Ziming Xu, Shuang Liang, Zijun Zhang, Ziqiao Xi, Ruobing Han, Yuchen Yan, Xinyue Wang, Fan Feng, Biwei Huang
Computer Vision
TSR-Ego: Temporally Guided Stereo Refinement Framework for Egocentric 3D Human Pose Estimation
Md Mushfiqur Azam, John Quarles and Kevin Desai
Egocentric 3D human pose estimation from head-mounted stereo cameras is challenging due to fisheye distortion, severe self-occlusion, and frequent truncation of body joints outside the camera field of view. Recent stereo
egocentric methods have improved performance through heatmap lifting, stereo correspondence, and
transformer-based refinement, but they often rely heavily on frame-local evidence or use temporal information only as auxiliary pose-level context. This limits robustness when current-frame stereo cues are weak, occluded, or ambiguous. We propose TSR-Ego, a temporally guided stereo framework that couples short-term motion evidence with projection-guided feature sampling. The model first enriches dense stereo feature maps using a causal depthwise-separable temporal convolution, allowing past visual evidence to influence the feature space before deformable cross-
attention. A single-stage causal stereo decoder then refines learned 3D joint queries through temporal self-
attention, joint self-
attention, and fisheye deformable stereo cross-
attention, using the evolving pose estimate to generate 2D sampling references. Unlike methods that apply temporal
reasoning mainly after pose prediction, TSR-Ego uses motion context to shape both the sampled stereo features and the joint representations while preserving online inference without future frames. Experiments on UnrealEgo2 and UnrealEgo-RW show state-of-the-art performance, with especially strong gains on real-world sequences.
Read the full abstract ▾
Computer Vision
What Pixels Are Enough? SEAMS: Sufficiency Saliency via MSE-Preservation Soft-Masks
Magdalena Trędowicz, Łukasz Struski, Arkadiusz Lewicki et al. (+4 more)
Saliency maps are most useful when they identify the image regions that are sufficient to preserve a model's behaviour. We introduce SEAMS, a sufficiency-based saliency method that directly optimises a soft mask using a preservation objective. Given a frozen differentiable model output, such as a class probability, CLS
embedding, or
token representation, SEAMS searches for a compact mask that preserves the selected output. The approach relies on a simple optimisation framework based on soft masks, a learnable budget, and a three-way image composite generated entirely from the query image. As a result, it requires no auxiliary distractor
dataset, architecture-specific attribution mechanism, or differentiable top-k relaxation. Experiments with frozen ViT-S/16 and ConvNeXt models show that the same optimisation pipeline can generate object-level, class-conditioned, and
token-level explanations by changing only the preserved target. The resulting masks are compact, interpretable, stable across random initialisations, and competitive on insertion and deletion
benchmarks. Our results also indicate that different architectures often rely on different sufficient evidence while achieving similar preservation fidelity, highlighting the architecture-dependent nature of visual explanations.
Read the full abstract ▾
All authors Magdalena Trędowicz, Łukasz Struski, Arkadiusz Lewicki, Karolina Pachota, Andrzej Grudzień, Mateusz Jagła, Jacek Tabor
Computer Vision
Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing
Chenxu Peng, Chongtian zhou, Dicheng Liu et al. (+5 more)
Fusing standard RGB frames with asynchronous event streams has emerged as a definitive paradigm for robust perception in degraded environments. Although unified backbones have recently gained traction in multi-modal
vision, adapting them to the RGB-Event domain remains fundamentally challenging. Existing architectures either resort to decoupled dual encoders that double computational overhead, or adopt generic unified designs that fail to resolve implicit
geometric parallax and cross-spectral aliasing under the extreme representational divide between dense intensity grids and sparse kinematic spikes. To transcend these bottlenecks, we present Evita, the first unified backbone specifically engineered for dedicated dense RGB-Event parsing. To achieve profound modal synergy, Evita explicitly embeds a suite of intrinsic co-learning modules directly into every encoder layer. Specifically, it features
Geometric Parallax Rectification for adaptive spatial
alignment, Harmonic Spectral Resonance for texture transfer exclusively in the complex frequency domain, and Transient Global Routing for event-driven asymmetric
attention. To guarantee robust feature extraction against spatial misalignments and decouple representations from specific event encodings, we construct N-ImageNetV2 alongside a
stochastic event representation mixing pretraining protocol, empowering the network to seamlessly accommodate arbitrary event formats in downstream tasks. Extensive evaluations across the DELIVER, DDD17, and DSEC
benchmarks confirm that Evita establishes new state-of-the-art metrics while delivering a superior accuracy-latency trade-off for real-time
multimodal perception.The code are publicly available at: https://github.com/chaineypung/Evita.
Read the full abstract ▾
All authors Chenxu Peng, Chongtian zhou, Dicheng Liu, Bo-Wen Yin, Yimian Dai, Xialei Liu, Ming-Ming Cheng, Xiang Li
Computer Vision · Cross-listed from cs.AI
MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
Runhan Shi, Quan Zhou, Yuqian Xu et al. (+14 more)
Large
language models (LLMs) are increasingly deployed in online medical consultation, yet existing
benchmarks remain poorly aligned with real clinical practice. Many rely on synthetic conversations or patient simulators, omit patient-uploaded medical images, or evaluate open-ended clinical responses using multiple-choice or lexical-overlap metrics that poorly reflect clinical quality. We introduce MedRealMM, a large-scale
benchmark for
multimodal online medical consultation built from de-identified patient-doctor interactions collected from a nationwide Chinese internet hospital. MedRealMM uses a
Multimodal Clinical Challenge Point (MCCP) extraction framework to identify clinically demanding moments in authentic consultation trajectories and converts each into a standardized next-response generation task while preserving the preceding text-image context. Each instance is paired with a case-specific rubric refined by physicians that rewards clinically desirable behaviors and penalizes unsafe, unsupported, or contradictory responses. The current release contains 5,620 real-world
multimodal cases spanning 64 clinical departments. We evaluate 19 general-purpose and medical-specialized LLMs, including text-only and
multimodal systems. Our results show that image information is critical for reliable clinical performance and that current frontier models remain below the online physician response. Although some frontier models satisfy as many or more positive clinical criteria than physicians, they trigger more negative criteria, indicating that safety-sensitive error avoidance remains a central bottleneck. MedRealMM offers a realistic and reproducible
benchmark for evaluating
multimodal medical
reasoning in real-world online consultation. The
dataset will be publicly available on Hugging Face at https://huggingface.co/
datasets/jdh-algo/MedRealMM.
Read the full abstract ▾
All authors Runhan Shi, Quan Zhou, Yuqian Xu, Shuai Yang, Xin Wu, Zitong Zhou, Hui Liu, Bin Zha, Zheming Wang, Liya Li, Wei Wei, Jinru Ding, Wenrao Pang, Mouxiao Bian, Haoyuan Hu, Jun Xu, Jie Xu
Computer Vision
Super-Generalist: Towards Comprehensive and Accurate Medical Image Understanding via Generalist-Specialist Synergy
Shaoteng Zhang, Weiwei Cao, Wanxing Chang et al. (+9 more)
Medical images require comprehensive and accurate interpretation to support the diagnosis of diverse clincial conditions. Recent
vision-
language generalist models offer broad task coverage and promising zero-shot capabilities, yet often lack fine-grained anatomical and lesion awareness for reliable diagnosis and spatial interpretability. In contrast, supervised specialist models achieve strong performance on specific tasks but typically lack generalization across diseases and anatomies. In this work, we present SuG, a Super-Generalist framework that unifies generalist
vision-
language learning with specialist objectives, enabling both broad generalization and specialist-level diagnostic capability. We perform specialist-enhanced
vision-
language alignment in SuG by incorporating spatial priors from multiple
segmentation experts, including anatomy, class-specific lesion and class-agnostic lesion segmentors that captures lesions beyond anatomies annotated during training. To improve lesion grounding capability, we leverage lesion masks as spatial priors to calibrate text-conditioned visual
attention, encouraging disease-related semantics to focus on clinically relevant regions. We evaluate SuG on extensive chest and abdominal CT
benchmarks, including CT-RATE, Merlin, MedVL-CT69K, and several in-house tumor
datasets. SuG achieves state-of-the-art performance across a wide range of disease diagnosis tasks and surpasses specialist models on several critical tumor diagnosis
benchmarks. Furthermore, SuG demonstrates strong lesion grounding capability, including robust generalization to lesion types lacking class-specific supervision.
Read the full abstract ▾
All authors Shaoteng Zhang, Weiwei Cao, Wanxing Chang, Yutong Xie, Kai Cao, Zaiyi Liu, Yu Shi, Tingbo Liang, Qi Zhang, Ling Zhang, Yong Xia, Jianpeng Zhang
Computer Vision
IB-Flow: Information Bottleneck-Guided CFG Distillation for Few-Step Text-to-Image Generation
Yiting Wang, Jingyi Zhang, Wenhu Zhang et al. (+4 more)
While large-scale text-to-image generative models have achieved unprecedented visual performance, their inherent reliance on multi-step iterative solvers incurs severe inference latency. Few-step
distillation targeting the Classifier-Free Guidance (CFG) trajectory has emerged as the prevalent dual-dimensional compression paradigm. However, existing frameworks remain subjugated by a coarse-grained blind injection paradigm that perpetually enforces a globally static guidance strength while indiscriminately sampling the supervisor timestep. This state-agnostic design completely disregards the intrinsic nature of image generation as a dynamic evolutionary process characterized by progressive
entropy reduction, which not only restricts the performance boundary of few-step compression but also precipitates severe CFG over-conditioning artifacts. To transcend these limitations, we re-examine the
distillation procedure through the theoretical lens of Information Theory, formally modeling it as a dynamic mutual information game constrained by the Information Bottleneck (IB) principle. Specifically, we dismantle traditional blind assumptions via a dual-track adaptive framework. To determine the injection target, we propose an instance-aware selection mechanism that transmutes the intractable KL divergence constraint into a zero-overhead closed-form solution predicated on the local vector field norm. To regulate the injection strength, we introduce an
entropy-aware schedule that dynamically decays alongside the SNR, applying maximal thrust for initial structural anchoring before smoothly reverting to the natural
manifold to refine micro-details. Extensive empirical evaluations corroborate that our framework fundamentally eradicates over-conditioning artifacts, shattering the performance ceiling to achieve SOTA generative fidelity under extremely stringent 2-step configurations.
Read the full abstract ▾
All authors Yiting Wang, Jingyi Zhang, Wenhu Zhang, Ke Chao, Yves Liang, Kun Cheng, Kang Zhao
Computer Vision
Junyi Hu, Zhewen He, Haomian Huang et al. (+2 more)
Sign
language translation (SLT) converts continuous sign
videos into spoken
language text. Gloss-free approaches leverage pre-trained visual encoders and
language models but rely on implicit cross-modal
alignment from translation supervision alone. We present VTaMo, a framework that introduces explicit multi-granularity
alignment at three levels: (1) local
alignment via
entropy-regularized optimal transport with a learnable null
token for fine-grained frame-to-
token correspondences; (2) global
alignment via a learnable orthogonal transformation that calibrates
embedding space geometry through Earth Mover's Distance; and (3) position-aligned contrastive learning for discriminative
token-level representations. Experiments on Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL demonstrate consistent state-of-the-art performance, with ablations confirming the complementary contributions of each component. Code is available at https://github.com/junyi2005/vtamo.
Read the full abstract ▾
All authors Junyi Hu, Zhewen He, Haomian Huang, Aoxiang Yang, Yi Fang
Computer Vision
4D Human-Scene Reconstruction from Low-Overlap Captures
Minhyuk Hwang, Sangmin Kim, Seunguk Do et al. (+2 more)
Existing volumetric capture of dynamic human performance achieves high fidelity with dense camera arrays. However, in real-world scenarios, only a handful of low-overlap cameras are available, which degrades the output quality and leaves large areas unobserved. Recent 4D reconstruction methods have focused on low-overlap settings, yet they still produce noticeable artifacts in under-observed regions.
Video diffusion models have emerged as another option, but they show geometrically inconsistent results for humans. To address these limitations, we propose StudioRecon, a pipeline that reconstructs 4D human scenes from sparse, low-overlap cameras by decoupling background and humans. We densify background supervision by synthesizing hundreds of camera-controlled novel views with a
video diffusion model. We also robustly initialize deformable Gaussian humans with cross-view identity association and triangulated multi-view keypoint fitting. Finally, our recursive enhancement module with motion-adaptive consistency injection harmonizes the composed output, thereby further avoiding remaining artifacts. We achieve state-of-the-art novel view synthesis across four real-world
datasets and demonstrate applications such as novel trajectory rendering and human replacement.
Read the full abstract ▾
All authors Minhyuk Hwang, Sangmin Kim, Seunguk Do, Daneul Kim, Jaesik Park
Computer Vision
Event Burst Trigger: An Availability Backdoor Attack on Event-Based SNN Object Detection
Jaesun Baek, Chanwook Lee and Eun-Kyu Lee
Event-based
vision and spiking neural networks (SNNs) are increasingly adopted for edge intelligence under strict latency and energy constraints. However, the vulnerability of event-based SNN object detection models to availability
backdoor attacks remains insufficiently studied. This paper presents Event Burst Trigger (EBT), an availability
backdoor attack targeting SNN-based object detection models. EBT injects carefully crafted event-based triggers into the training data, which induce temporally concentrated event streams during inference. These burst-like activations increase the number of phantom (i.e., spurious) object candidates, and consequently inflate the computational cost of the post-processing stage, particularly Non-Maximum Suppression (NMS). We evaluate EBT on SpikeYOLO, the state-of-the-art SNN-based object detector, under a poison-only threat model that does not require modifications to the model architecture, loss function, or inference pipeline. Experimental results show that while detection accuracy remains largely preserved, with mAP@0.5 decreasing by less than 0.099, the latency of the NMS stage increases by up to 38%. This indicates that NMS can become a dominant availability bottleneck in event-based SNN object detection. Experiments on an edge platform further show that the proposed attack elevates baseline resource utilization and reduces scheduling slack without inducing conspicuous peaks in resource usage. In addition, STRIP-based
backdoor detection fails to reliably distinguish the proposed attack from benign inputs. These results characterize a previously underexplored availability
backdoor threat in event-based SNN object detection systems.
Read the full abstract ▾
Computer Vision
Peipei Zhu, Yueqing Niu, Lin Zhu et al. (+3 more)
Video anomaly detection (VAD) is critical for automated surveillance but remains fragile under challenging conditions such as illumination variations, fast motion, and complex backgrounds when relying solely on visible light
videos. To address these limitations, we propose EVAD, an event enhanced VAD framework that jointly exploits conventional
video and event streams captured by bio inspired event cameras. Event sensors asynchronously capture brightness changes with high temporal resolution, offering robustness to motion blur and extreme lighting, and providing motion salient cues complementary to
video based visual information. To support multi modal VAD research, we construct a large scale visible event
benchmark comprising 6.3 billion events and 376,368
video frames collected under diverse illumination levels, motion patterns, and background complexities, filling the gap of realistic and scalable
datasets for event based anomaly detection. Building upon this
dataset, we design a contrastive multi modal pretraining framework to learn discriminative event representations by aligning semantic
embeddings across event streams, visible
videos, and textual descriptions. An adaptive fusion module then dynamically integrates event based temporal cues with
video based spatial semantics, improving robustness to environmental disturbances. Experiments on
benchmarks and the proposed TJUTCM Pha
dataset demonstrate that E VAD consistently outperforms methods, validating the effectiveness of event-based sensing for VAD in real world scenarios.
Read the full abstract ▾
All authors Peipei Zhu, Yueqing Niu, Lin Zhu, Guanchong Niu, Yang Yu, Zheng Li
Computer Vision
Integrating Large Language Models and Graph Convolutional Networks for Semi-Supervised Image Classification
Camila Piscioneri Magalhães and Lucas Pascotti Valem
While the growing availability of image data has driven significant advances, labeling
datasets remains costly and time-consuming. Therefore, semi-supervised approaches such as Graph Convolutional Networks (GCNs), which learn from both labeled and unlabeled data, have emerged as a promising solution. One of the primary challenges in applying GCNs to image classification is graph construction, since, unlike in citation networks or similar domains, images typically do not come with a predefined structural representation. For visual data, most studies construct graphs based on the similarity between feature vectors from pretrained deep learning backbones, typically by employing kNN or reciprocal kNN
algorithms. Although Large
Language Models (LLMs) have shown remarkable capability in capturing high-level semantics, their integration with GCNs for image classification remains underexplored. Aiming to fill this gap, our approach uses a
Vision Language Model (VLM) to generate textual image descriptions, which are then processed by an LLM to estimate semantic similarity scores between connected images. These scores guide the
pruning of edges in kNN and reciprocal kNN graphs, filtering out semantically irrelevant neighbors. Experimental results reveal that leveraging LLMs for graph refinement can improve classification accuracy, particularly for kNN graphs and some backbones. The source code is publicly available at http://gcnllm.lucasvalem.com.
Read the full abstract ▾
Computer Vision
Equivariant Filter for High Performance Image Tracking using an Event Camera
Angus Apps, Yixiao Ge, Timothy L. Molloy et al. (+1 more)
Image tracking is the problem of estimating the transformation that relates a moving image of a scene to an original reference image. The problem is important in control of autonomous vehicles or
robots, where the image encodes information about the motion of the camera or environment, as well as in pure computer
vision applications. In this paper, we present an equivariant filter design for high performance tracking of planar image transformations using an event camera. The design exploits the Asynchronous Event Blob (AEB) tracker (Wang et al., 2024) to extract feature-position measurements from the raw event stream, and an equivariant filter to compute an affine image translation and rotation using the special Euclidean group
symmetry. The equivariant filter incorporates an equivalent-measurement update step that de-correlates the (highly temporally correlated) feature-position measurements provided by the AEB tracker. We evaluate the design experimentally using two
datasets involving general and fast rotational motion. We
benchmark results against direct optimisation (estimating the relative transformation from the raw blob tracks), and a covariance intersection approach for overcoming data correlation. Our design provides smooth image tracking for features moving up to 7000 pixels per second on the image plane.
Read the full abstract ▾
All authors Angus Apps, Yixiao Ge, Timothy L. Molloy, Robert Mahony