Computer Vision
Retrieved but Not Delivered: Multimodal Memory Delivery for Long-Term Agents
Yuhang Jiang, Qingwei Liao, Kaize Yin et al. (+3 more)
Work on
memory for
multimodal agents optimizes what is written, updated and retrieved. Between
retrieval and the answer, however, is a stage that
multimodal memory evaluations do not isolate: what of the retrieved
memory reaches the model, and in what form. We call it delivery, and a controlled decomposition on MemLens locates the remaining room there. With the retrieved evidence set exactly fixed, delivering the original pixels instead of withholding them raises accuracy by 13.87 points on an 8B backbone, whereas making
retrieval perfect on those same messages improves it by 2.31. Delivery is the larger term on all three MemLens backbones and grows with backbone strength;
retrieval grows too, without closing the gap. We propose DeliverMem, an instantiation of delivery as three decisions: keep the original modality, give each item a readable identity, and state when it was seen, with a
retrieval-side adapter for the one property delivery cannot supply. Each is measured against a delivery-matched control that alters only its own variable. DeliverMem leads the strongest published
memory agent on MemLens at all four context lengths, and beats DMV-Bench's own strongest method at every setting on both backbones. On MemLens it does this on a tenth to a seventieth of the input. Each decision helps only where the question lacks what it supplies, and is null elsewhere. A single fixed configuration nonetheless leads both
benchmarks, without training any component or modifying the stored records. Project page: https://avalon-s.github.io/DeliverMem/
Read the full abstract ▾
All authors Yuhang Jiang, Qingwei Liao, Kaize Yin, Xingling Liu, Luca Cuomo, Silvio Bacci
Computer Vision
Jihoo Jung, Suho Yoo, Jeongsoo Choi et al. (+5 more)
Smart-home assistants are expected to handle diverse, realistic requests that arise in daily life. In such interactions, users often rely on the surrounding
multimodal context-pointing at objects or referring to what they see or hear, leaving their requests underspecified in
language alone. Existing smart-home
benchmarks, however, express user requests solely through
language, leaving context-dependent real-world requests underexplored. To bridge this gap, we introduce OmniSmartHome, a
multimodal smart-home
benchmark where each spoken request is paired with the surrounding visual and spatial-audio context, providing complementary cues to disambiguate underspecified requests. OmniSmartHome comprises 1,360 synthetic and 272 real-world episodes. We evaluate 16 omnimodal large
language models (Omni-LLMs) and reveal that, while they perform strongly when
speech alone sufficiently conveys the user's intent, performance drops substantially when resolving it requires
reasoning over
multimodal contextual cues. As a simple
agent baseline, we provide PROME (PROcedural
Memory for
multimodal Evidence gathering), which equips
agents with specialized audio-visual perception tools and procedural
memory for orchestrating their use. PROME generally improves performance across six Omni-LLMs. Demos and examples are available at https://omni-smart-home.github.io
Read the full abstract ▾
All authors Jihoo Jung, Suho Yoo, Jeongsoo Choi, Hyebin Cho, Tae Wook Haam, Hyeonggon Ryu, Sumin Park, Joon Son Chung
Computer Vision
Attribution Gaps in Zero-Training LLM+OVOD Pipelines: A Fine-Grained Analysis of the CAAP–SNAP Discrepancy
Yu-Feng Yen
LAOD and similar zero-training LLM+open-vocabulary-detector (OVOD) pipelines score two things separately: class-agnostic localization accuracy (CAAP) and semantic naming accuracy (SNAP). The two consistently diverge, and nobody has asked why. This paper asks why, on the full 5,000-image COCO-Val split (27,273 detections) rather than the small subset the original work evaluated on. Object visual complexity turns out not to be the driver – small and occluded objects are, if anything, localized better than large ones. Vocabulary novelty is: once the LLM's wording falls outside the detector's native category set, localization accuracy falls from 80.9% to 31.6%. That drop is not spread evenly across unfamiliar phrasing, though. Almost all of it comes from cases where the novel wording actually names a different object than the one COCO annotated (true synonyms still score 89.3%; semantically unrelated "noise" labels score 12.0%). A closer look at a further failure subset tells a similar story: 78-88% of what looks like complete localization failure is really the model correctly finding a real object that COCO's non-exhaustive 80-category scheme simply never labeled, not
hallucination. Swap the detector backbone (YOLO-World for Grounding DINO) or the LLM (Gemma-3 for Qwen2.5-VL) and both the effect and its rough size hold up, so this looks like a general property of the pipeline family rather than a quirk of one model pairing. The upshot is that a large share of the apparent CAAP–SNAP gap traces back to closed-category annotation limits rather than a real grounding failure, which matters for how we detect
hallucination, analyze failure modes, and design evaluation for grounded
multimodal systems meant to work in the open world.
Read the full abstract ▾
Computer Vision
CFCH: Coarse-Fine Collaborative Hierarchical Learning for Anterior Segment Disease Analysis
Peng Wang, Haohan Zou, Yanlin Wu et al. (+3 more)
Accurate classification of anterior segment diseases is crucial for ophthalmic screening and diagnosis. However, slit-lamp image analysis remains challenging due to substantial variability in imaging conditions and the intrinsic anatomical-disease hierarchy of ocular pathologies. Existing methods typically formulate this task as a flat multi-class classification problem, ignoring the structured dependency between anatomical regions (e.g., cornea, conjunctiva, and lens) and disease manifestations.To address these limitations, we propose CFCH, a Coarse-Fine Collaborative Hierarchical learning framework that explicitly models anatomical context and disease semantics through a dual-branch architecture. To enable effective cross-granularity collaboration, CFCH introduces semantic and cross-granularity
attention consistency constraints, encouraging aligned yet complementary feature learning across branches. In addition, we construct AS-9K, a large-scale anterior segment
dataset with 8975 images covering 12 common disease categories. To the best of our knowledge, AS-9K is the largest publicly available
dataset for anterior segment image classification. Extensive experiments on two anterior segment
datasets demonstrate that CFCH outperforms state-of-the-art methods. Qualitative visualizations further show more focused and lesion-relevant activation responses, validating the effectiveness of the proposed framework. Code will be available at https://github.com/ybupengwang/CFCH.
Read the full abstract ▾
All authors Peng Wang, Haohan Zou, Yanlin Wu, Xueshuo Xie, Yan Wang, Tao Li
Computer Vision
Feature Space Guidance for Breast Cancer Classification in DCE-MRI
Benjamin Hamm, Yannick Kirchhoff, Maximilian Rokuss et al. (+6 more)
Dynamic contrast enhanced breast MRI (DCE-MRI) is a powerful clinical tool for breast cancer detection, providing high resolution anatomical detail together with rich temporal contrast information. However, high dimensional 4D inputs, small lesions, and heterogeneous acquisition protocols across clinical sites hinder robust automated classification of healthy, benign, and malignant cases. To address these challenges, we propose a framework that dynamically analyzes latent representations to adapt to protocol-specific characteristics. Spatial variability is mitigated by reducing confounding background uptake and compensating for misalignment caused by deformable soft tissue. Additionally, relationships in the latent space across phases are leveraged to select the most informative temporal features, improving robustness to protocol-specific temporal variability. Finally, task specific discriminative features are promoted through large scale supervised lesion segmentation pretraining, which substantially enhances downstream finetuning. Evaluated under leave-one-center-out validation on the ODELIA
dataset and the held-out AMBL cohort, the proposed framework substantially outperforms finetuned radiology foundation models and prior methods, improving mean AUROC by nearly 8 points and balanced accuracy by 4 points over the strongest baseline. Additionally, our method achieved first place in the MICCAI ODELIA Breast MRI Challenge 2025, further demonstrating its effectiveness for robust breast cancer classification. We publicly release our codebase under https://github.com/MIC-DKFZ/CURIAtor.
Read the full abstract ▾
All authors Benjamin Hamm, Yannick Kirchhoff, Maximilian Rokuss, Moritz Langenberg, Constantin Ulrich, Tassilo Wald, Jeremias Traub, Karol Gotkowski, Klaus Maier-Hein
Computer Vision
Harnessing Coupled Stream Completion For Human-Object Interaction Modeling
Dawei Guan, Di Yang and Jiangtao Wang
Text-conditioned human-object interaction (HOI) generation requires body motion, object trajectories & rotations, and hand articulation to remain coordinated. These components differ in scale and dynamics, but must agree on contact, relative pose, and timing. A shared representation may limit the distinct structure of each stream, while independent generation prevents each stream from responding to changes in the others. Latent supervision alone also does not directly constrain contact after decoding. We propose TRACE, a continuous latent framework that keeps stream states separate and couples their updates. TRACE encodes body, object, and hand motion into separate latents and predicts each stream velocity from the complete current interaction state.
Geometric losses on decoded motion further constrain contact and object-relative motion over time. The same model supports completion of any single absent stream from the other two. Frozen flow features also serve as input to a
language model for HOI understanding. Experiments on InterAct, OMOMO, and BEHAVE show that joint completion training improves generation and that frozen flow features improve understanding over raw-motion encoding. On InterAct, TRACE achieves the highest contact precision, recall, and F1 among the compared methods.
Read the full abstract ▾
Computer Vision
In-Flight KV Cache with Clean Anchors for Faster Autoregressive Video Diffusion
Yikai Wang, Xiao Han, Mengmeng Xu et al. (+9 more)
Few-step autoregressive
video diffusion generates a long
video by splitting the
video into temporal chunks and generating chunk-by-chunk, each through a short sequence of denoising stages. To memorize chunks that are already generated, previous methods reconstruct a clean or less-noisy key–value (KV) cache by additional forwards to build the cache without advancing an output latent. However, every denoising forward itself already computes the in-flight KV of the current chunk. We introduce FlashForward, which directly reuses this cache to avoid the heavy cache-update-only model forwards. After the current chunk completes one denoising stage, its stage-specific cache is already available for the next chunk. Assigning one GPU to each stage therefore lets different chunks occupy different stages concurrently. This early availability has a quality cost: the resulting stage-matched history is noisy, causing appearance and motion drift among chunks. To complement it, FlashForward produces sparse auxiliary clean anchor latents before the corresponding region is generated so the generation trajectories can be stabilized by this two-sided conditioning. The two
memories operate at different temporal scales: sparse clean anchor KV supplies coarse, long-range two-sided structural guidance, while dense stage-matched history preserves fine, recent
evolution. With up to four GPUs, FlashForward runs $1.16$–$1.69\times$ faster than HiAR and $1.42$–$2.92\times$ faster than Self-Forcing for 16 FPS
videos of 20 seconds or longer across 1.3B and 14B backbone scales at 480p and 720p. On VBench, for the 1.3B model at 480p, it achieves higher scores and remains stable at longer durations, demonstrating that FlashForward generates high-quality and temporally consistent
videos across durations of 20s, 35s and 65s at a much faster generation speed.
Read the full abstract ▾
All authors Yikai Wang, Xiao Han, Mengmeng Xu, Juan Camilo Perez, Yiannis Douratsos, Sen He, Zijian Zhou, Fei Zhang, Zhaochong An, Juan-Manuel Perez-Rua, Chen Change Loy, Tao Xiang
Computer Vision
RIPE-MambaSpike: Resolution-Independent Spiking-State-Space Interfaces for Parameter-Efficient Event-Based Vision
Md Muhiminul Islam, Shoaib Ahmed Dipu and Sayeed Shafayet Chowdhury
Spiking-Mamba hybrids reach strong accuracy on event-based
vision, but existing designs often require tens of millions of parameters. Much of that cost comes from how the spiking front-end is connected to the state-space backbone rather than from the hybrid architecture itself. In a representative model, a single resolution-dependent projection accounts for 33.55M of 36.25M parameters. To that end, we introduce RIPE-MambaSpike (Resolution-Independent, Parameter-Efficient), which replaces that projection with a hierarchical multi-resolution bridge of fixed channel width. Its deployed footprint is 0.870M parameters, constant at fixed time steps and widths across a 43x range of input areas. Reparameterized spiking stages, temporal decoupled modulation, and a dynamic convex-hull-bounded dual-stream membrane-potential
attention preserve accuracy under this compact design. Result-wise, RIPE-MambaSpike is pareto-optimal on CIFAR10-DVS, N-Caltech101, and DailyDVS-200. Notably, on the 200-class DailyDVS-200, a scaled 8.04M configuration achieves 45.7% top-1 accuracy, the best reported spiking result on that
benchmark, and outperforms prior spiking methods with 3.0-15.1x fewer parameters than dense ANNs. Overall, our findings demonstrate that competitive event-based recognition does not require resolution-dependent parameter growth. Code is available at https://github.com/MuhiminOsim/RIPE-MambaSpike.
Read the full abstract ▾
Computer Vision
DepthBench: Measuring How Residual Connections Enable More Computational Depth
Keyu Wang, Yangyi Huang, Jiale Kang et al. (+3 more)
Depth is a natural way to increase the computational capacity in
Transformers, yet the contribution of deeper layers can diminish as depth grows larger. Recent approaches enhance normalization (\text{e.g.}, LayerNorm Scaling) or residual connections (\text{e.g.}, mHC, AttnRes) to enable better information flow and depth utilization. However, it remains unclear whether they truly translate increased architectural depth into effective computational depth, and whether their reported gains stem from better access to information across depth, or unaccounted-for confounding factors. In this paper, we introduce DepthBench, a controlled
benchmark for studying computational depth across various architectures. We systematically vary the width–depth aspect ratio ($d_{\text{model}}/n_{\text{layer}}$) from shallow–wide to deep–narrow shapes, while keeping the model size and pre-training recipe fixed. Across 10 representative architectures, we find that the benefit of allocating more capacity to depth is strongly architecture-dependent. Standard Pre-LN and most of its norm- and scaling-based variants provide little benefit and can even degrade performance as models become deeper and narrower, whereas HC and Full AttnRes improve consistently even at extreme deep shapes. These gains extend beyond pre-training loss and consistently translate into improved domain-specific performance and effective computation. Controlled layer-level analyses further show that the gains of HC and Full AttnRes are associated with more effective utilization of additional layers, revealing distinct mechanisms of computational depth across architectures. Overall, our results identify residual connection design as a key determinant of whether depth can serve as a meaningful scaling axis by enabling additional architectural depth to translate into effective computation.
Read the full abstract ▾
All authors Keyu Wang, Yangyi Huang, Jiale Kang, David González-Martínez, Weiyang Liu, Shiwei Liu
Computer Vision
SetOPD: From Few Visual Exemplars to Multimodal Candidate Sets for Remote-Sensing Open-Prompt Detection
Jinlong Hu, Yi Zhang, Zhiqi Xia et al. (+2 more)
Open-
prompt detectors allow users to specify targets with text, visual exemplars, or both. We argue that existing designs underuse the visual modality in two ways. First, multiple exemplars are commonly compressed into a single class-level
embedding. This textualizes visual prompting: the resulting vector plays the role of another class name, is often aligned to or injected into the text pathway, and may be suboptimal when only a few heterogeneous exemplars are available. Second, existing methods interact primarily in
prompt or representation space, before modality-specific detection states are formed. We address both issues from a set perspective: \setopd preserves modality-specific decoding states from a shared
prompt-conditioned initialization and performs explicit
multimodal collaboration at the candidate-state level. For the first issue, we introduce \br prompting, which reads every boxed exemplar in its full scene context and decomposes the pooled support evidence into a Base anchor and a learned Residual correction; the resulting
prompt has fixed capacity regardless of the number of examples and drives its own visual detection pathway. For the second, we recast text–visual collaboration from representation-level fusion into a candidate-set modeling problem. Paired-Query Arbitration (\pqa) then performs explicit cross-modal state arbitration only after modality-specific candidate states have been formed. The two readers share query initialization so their candidates are paired by index; a learned gate arbitrates within each pair, followed by a permutation-equivariant module that reasons over the fused set.
Read the full abstract ▾
All authors Jinlong Hu, Yi Zhang, Zhiqi Xia, Yikang Zhou, Shunping Ji
Computer Vision
UnStep: Training-Free Acceleration of Causal Video Diffusion with Fewer Steps Than Distillation
Youssef Mansour, Enis Simsar, Fadime Sener et al. (+4 more)
Distilling bidirectional multi-step
video diffusion transformers into few-step causal models has become a common approach for streaming
video generation. While these few-step students are significantly faster than the teachers they are distilled from, they remain slow for real-time generation. In this work we present UnStep, a training-free wrapper that accelerates few-step causal
video models at inference by running them with fewer
diffusion transformer (DiT) steps than during
distillation and limiting the temporal window retained in the
attention KV cache. We propose two inference-only mechanisms to recover quality lost by step reduction and
attention windowing: renoising the generated latent frames to a near-clean level and reusing the existing clean-cache pass to refine them, and applying truncated SVD to the DiT
attention value and output projections. We also accelerate inference with a quality-preserving runtime stack for the DiT and VAE decoder, including more efficient
attention calls and KV indexing, fused Triton RoPE with cached coefficients, and VAE decoding with optimized
memory layout, precision, and convolution kernels. By reducing computation and optimizing the runtime stack, UnStep sets a new throughput regime for causal
video diffusion, by running substantially faster than current methods, reaching 50 FPS on a single H100 without quality loss, and 77 FPS on GB200, all without retraining.
Read the full abstract ▾
All authors Youssef Mansour, Enis Simsar, Fadime Sener, Markos Georgopoulos, Albert Pumarola, Ali Thabet, Edgar Schoenfeld
Computer Vision · Cross-listed from cs.AI
LocalProp: Neuro-Localized Memory-Efficient Backpropagation
Diana-Nicoleta Grigore, Iuliana Georgescu and Radu Tudor Ionescu
The current deep learning training paradigm employs end-to-end backpropagation, regardless of the training stage, i.e. pre-training or fine-tuning. However, backpropagating through the entire model is neither biologically plausible nor
memory efficient, since learning inside the
brain is highly localized. Therefore, we propose LocalProp, a training procedure that locally updates the weights of a model. Our neuro-localized weight updates follow the "pre-training then fine-tuning" paradigm, where the pre-training is based on I-JEPA. After locally updating the weights, a
pruning operation is performed, followed by a short final fine-tuning phase.
Pruning helps by sending the learning signal from higher blocks to lower blocks. We perform experiments on several
datasets, including large-scale
benchmarks such as ImageNet, and empirically show that LocalProp reaches good performance at a fraction of GPU peak
memory. By varying the number of jointly optimized blocks, we identify gradient-propagation span as a practical control over the accuracy-
memory trade-off.
Read the full abstract ▾
Computer Vision
Cross-Domain Few-Shot Writer Adaptation for Real-World Handwritten Mathematical Expression Recognition
Paulo Grane Gabriel Silva, Lorenz Bernard Marqueses and Joel Ilao
Handwritten mathematical expression recognition (HMER) refers to the task of recognizing and converting handwritten mathematics into a parsable markup
language, usually LaTeX. No current state-of-the-art-competitive system adjusts to the way a specific person writes, and the domain gap between training images (usually digital or perfectly binarized) and images physically taken with a camera used in inference has received fairly little
attention for this specific problem. We characterize this domain gap through fragmentation and stroke-width analyses of the images as well as introduce a writer-adaptive fine-tuning pipeline to MFH-CoMER in an attempt to address it. We further introduce sample author-specific
datasets, consisting of five handwriting category subsets from two authors, and evaluate using a McNemar's test and permutation tests adapted to limited data. Results suggest an increase in expression recognition rate and a decrease in CER for digital handwriting but more varied results for physical handwriting, with the model struggling for handwriting articles that the base model can already evaluate well. Nonetheless, adapted models were found to have improved results for four out of five subsets. Statistical testing results suggest a consistency in improvement for two of the tested author-specific subsets. Our results point towards the potential feasibility of writer adaptation for the HMER task.
Read the full abstract ▾
Computer Vision
GeoCR: Learning a Generalist Cloud Removal Prior from Heterogeneous Observations
Jeonghyeok Do and Munchurl Kim
Cloud removal methods are typically specialized to individual
datasets and input configurations, limiting reuse across sensors, spectral bands, and observation settings. We introduce GeoCR, a generalist model that unifies RGB-only-based CR and multispectral-based CR from single- or multi-temporal cloudy observations, with optional SAR guidance, within a single network. To accommodate different spectral and sensing domains, compact input and output stems extend a pretrained RGB autoencoder while keeping its encoder and decoder trunks frozen. This shared latent interface enables a single flow
transformer to jointly model clean RGB and non-RGB latents, conditioned on separate cloudy-observation streams and optional SAR
tokens. Through joint pretraining on the training splits of ten
datasets comprising 883,331 cloud-free target images, GeoCR learns a shared cloud removal prior across these heterogeneous configurations. The same pretrained checkpoint supports direct inference without
dataset-specific fine-tuning and efficient adaptation through low-rank adaptation (LoRA). We evaluate GeoCR against general image restoration and cloud removal methods on test splits of the contributing
datasets under full-band and RGB-only settings. GeoCR achieves the best FID and DISTS on full-band SEN12MS-CR and Sen2_MTC_New and RGB-only CUHK-CR2, outperforming existing models and demonstrating the effectiveness of a reusable generative model across diverse settings.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Models
Tam Le Thi Thanh, Hoang Tran Van, Hong-Hanh Nguyen-Le et al. (+1 more)
In tri-modal visual question answering (VQA), auxiliary text is commonly used to complement visual and textual inputs, yet its reliability is often uncontrolled. While prior work studies modality conflicts in general, it remains unclear how different types of unreliable auxiliary text affect answer selection under fixed image-question-option contexts. In this work, we show that the most harmful auxiliary text is not necessarily the most factually incorrect, but the one that aligns with the question while contradicting the image and favoring a specific distractor, leading to systematic redirection of model predictions. To isolate this effect, we introduce the Textual Reliability Ladder, a controlled diagnostic protocol that decomposes auxiliary text along three axes: image consistency, question relevance, and option support. Across multiple
datasets (ScienceQA, VCR, A-OKVQA, Causal-VidQA) and recent VLMs, we find that such distractor-supporting text induces the largest accuracy drops (up to 53.1%) and concentrates errors on specific incorrect options. To mitigate this failure mode, we propose a training-free inference-time intervention that explicitly counteracts this redirection effect via noise-stability steering and dynamic grounding, reducing redirected errors while largely preserving performance under faithful text. Our results highlight that auxiliary-text reliability must be understood at the decision level, rather than solely through factual correctness, and provide a practical pathway toward more robust tri-modal
reasoning.
Read the full abstract ▾
All authors Tam Le Thi Thanh, Hoang Tran Van, Hong-Hanh Nguyen-Le, Thanh Duc Ngo
Computer Vision
JEPA Learns What the Mask Leaves Unrecoverable
Peng Xie and Amr Alanwar
Joint-
embedding predictive architectures are unusually sensitive to how the input is masked: block masks work, scattered masks do not, and the explanations are empirical. We give a measurement account. A mask is a linear measurement, and in a compactly supported wavelet basis every atom whose support lies inside the hidden region falls in the measurement's null space and leaves no trace in the data. The JEPA loss asks only that the encoded context suffice for the target, so a target that a low-level prior can recover admits a shortcut, one the moving-average target encoder can make self-consistent. What removes the shortcut is the coarse-scale content the mask leaves unrecoverable, provided enough context stays within reach of each target. We score that content before training and test the account's distinctive predictions in 151 pre-training runs. On ImageNet-100, strip masks match blocks in area and contiguity yet are recoverable, and they land at 40.3% linear top-1, beside random masks at 40.8%, against 64.3% for blocks; within one geometry family, the placements that leave the least unrecoverable content lose 6.5 points to those that leave the most, over five seed pairs; pixel targets span 7 points where latent targets span 25; and against a frozen target the gap between random and block masks, 19 points on the same kind of GPU, closes to 1.5, so the geometry acts through the target the encoder produces for itself. On UCF101 the masking ratio decides which condition, content or reach, binds; removing whole frames, unrecoverable in space but recoverable from neighbouring frames, is worst at both ratios; and on V-JEPA's own masks, batching them intact instead of truncated changes little (36.0% against 35.1%), whereas making 100 target
tokens inside the blocks visible lifts them to 48.7% and hiding 100 context
tokens outside the blocks does not (33.7%).
Read the full abstract ▾
Computer Vision
Back-Tracking from Clarity: Self-Learning to See Text from Afar
Duc-Tri Tran, Phi Le Nguyen and Minh Hoai
We propose a self-supervised framework designed to enhance the capability of scene text detectors in identifying and recognizing text in scenarios where instances are shown at significant distances, typically small, blurred, and frequently missed by conventional models. Our approach leverages the high-fidelity performance of existing text spotting models on large, clear text as a foundational supervisor. By temporally back-tracking these high-confidence detections through
video sequences, we automatically synthesize pseudo-labels for preceding frames where the distant text is still visually degraded or undersized. These pseudo-labels enable training a student model specialized for early text detection, without requiring any manual annotation. The success of this approach depends on accurate pseudo-label generation, for which we develop a dedicated scene text tracker capable of maintaining consistent text identities across challenging
video sequences. In addition, we propose SceneText50, a diverse multilingual outdoor
dataset to facilitate training and evaluation. Experiments show that our framework significantly improves early detection accuracy and robustness across varied scenes and
languages. Code and data are at \href{https://github.com/trid2912/BackTrackingText}{https://github.com/trid2912/BackTrackingText}.
Read the full abstract ▾
Computer Vision
Can Motion-Language Models Ground Structure? STRIDE for Evaluating the Evaluators
Lixing Tan, Qing Xia, Yuting Guo et al. (+2 more)
Motion-
language models are typically scored by motion-
language evaluators, but how well these evaluators ground
language structure remains unclear. Here, we introduce the Structure grounding via Temporal-order, Reflection, and Identity Diagnostic Evaluation (STRIDE)
benchmark to systematically evaluate the ability of evaluators to track temporal order, mirror reflection, and action identity. STRIDE comprises $5{,}869$ triples, each consisting of a motion, its original caption, and a perturbed caption, spanning both short and long descriptions. We likelihood-balance caption pairs to reduce text-only bias and estimate each evaluator's caption preference under unrelated motions to measure the discrimination gain from matched motions relative to this baseline. Our experiments reveal weak structural grounding and severe deficits in mirror sensitivity among the audited evaluators, which commonly used evaluation protocols fail to expose. To understand why these limitations go undetected in standard tests, we examine the evaluators more closely. We find that text-only priors alone can solve naive perturbation tests on existing
datasets, while common
retrieval and distributional metrics barely respond to structural corruption introduced by mirroring ground-truth motions. These findings suggest a natural intervention: structural hard negatives. Our experiments show that a simple modification to contrastive learning substantially improves performance on temporal order and mirror reflection. The
benchmark and code will be released.
Read the full abstract ▾
All authors Lixing Tan, Qing Xia, Yuting Guo, Shuai Li, Aimin Hao
Computer Vision
REMEDY: How Far Is Video Generation from Medical Education World Models?
Lixing Tan, Yanghao Zhou, Qing Xia et al. (+3 more)
Recent
video generation models produce realistic
videos and show potential as a foundation for world models. These advances create opportunities for generating medical teaching demonstrations, which requires both convincing visual quality and precise procedural actions. However, whether current generators can meet these requirements has not been measured. To address this problem, we introduce Readiness Evaluation of Medical Education Demonstration sYnthesis (REMEDY), to our knowledge, the first
benchmark for AI-generated medical teaching demonstrations. REMEDY provides 900 first frames from real demonstration
videos, covering 12 tasks across four scenarios: operating room, imaging, clinic and bedside, and resuscitation. Five contemporary open-source
video generation models produce 4,500
videos from these frames. We combine task-specific clinical checklists with
video and motion quality metrics. Evaluation covers four dimensions: clinical action following, clinical profiles,
video quality, and motion quality. Our results show that realistic appearance and temporal consistency do not ensure correct clinical actions. Even the most advanced MiniMax-H3 achieves only 28.25% on strict clinical success rate, and fine-grained clinical actions remain challenging. These findings establish a foundation and roadmap for developing future medical education world models.
Read the full abstract ▾
All authors Lixing Tan, Yanghao Zhou, Qing Xia, Yuting Guo, Shuai Li, Aimin Hao
Computer Vision
Seeing Parts, Reasoning about Worlds: Visual Inference under Partial Observation
Wei Wang, Wenqiao Zhang, Yutong Lin et al. (+2 more)
World modeling under partial observation requires
reasoning about the complete worlds that remain compatible with limited visual evidence. Occluded objects and unseen regions can leave several world states possible; additional views can exclude alternatives and strengthen the conclusions supported by the observations. We introduce WorldScope to study this process through possible-world semantics, evidence-grounded data, and learned visual representations. WorldScope-1.2M provides 1.2 million English question-answer pairs spanning eight world properties, ten task interfaces, and three observation protocols. Its answers encode confirmed facts, supported bounds, and unresolved possibilities. Complementary supervision comprises 4,800 certified counterworld groups with equivalent base observations and different hidden object configurations and query answers. These groups provide physical witnesses of ambiguity and training-only labels for world compatibility and view-induced exclusions. We propose WorldFlow, which composes cross-view entity evidence and support-surface coverage into an image-subset evidence
lattice. Counterworld compatibility and transition objectives train subset representations to reflect how new observations constrain possible worlds. A shared answer generator uses these representations to predict the strongest supported conclusion. WorldScope-Bench evaluates claim judgments and evidence-dependent conclusions as views are selected, combined, removed, or ordered. On its 5,000-question test set, WorldFlow reaches 64.34% exact accuracy, improving over the same backbone trained on QA alone by 24.88 percentage points. It retains 50.43% accuracy on the 3,000 questions from structure-disjoint scenes.
Read the full abstract ▾
All authors Wei Wang, Wenqiao Zhang, Yutong Lin, Jun Xiao, Yueting Zhuang
Computer Vision
QuacamFM: Quaternion-Constrained Flow Matching for Camera Pose Estimation
Bao-Long Tran, Cuong Le, Tahereh Dehdarirad et al. (+2 more)
Camera pose estimation from multi-view images remains a challenge in computer
vision. Traditional methods often address this problem using Structure-from-Motion (SfM) with bundle adjustment. However, camera poses estimated from sparse views are inherently ambiguous due to insufficient
geometric constraints. Recent work leverages probabilistic models, such as
diffusion models, to generate multiple camera pose hypotheses and therefore capture this uncertainty better. Most of these methods represent camera rotations using unit quaternions, but treat them as unconstrained 4D vectors during the generative processes, thereby ignoring the unit-norm constraint of quaternions. Unconstrained quaternions create non-smooth and suboptimal generation trajectories. To this end, we propose *QuacamFM*, a quaternion-constrained flow matching framework for camera pose estimation that preserves unit quaternion representations throughout the entire flow trajectory. We design the optimal transport of the quaternion flows using smooth spherical linear interpolation. Experiments on CO3Dv2 demonstrate our method's advantage in camera pose accuracy over
diffusion-based methods and classical SfM approaches. We further show that our quaternion-constrained formulation outperforms the naive application of standard flow matching to 4D quaternion vectors on sparse-view camera pose estimation. Finally, it is observed that QuacamFM generalizes well across
datasets and in-the-wild examples.
Read the full abstract ▾
All authors Bao-Long Tran, Cuong Le, Tahereh Dehdarirad, Fredrik Viksten, Per-Erik Forssén
Computer Vision
De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift
Mengyuan Liu, Yuhang Wen, Yi Zhang et al. (+4 more)
Skeleton sequences can represent both individual actions and multi-entity interactions, encompassing human bodies, hands, objects, and
robots. Existing approaches to recognize skeleton-based actions and interactions usually adopt a late fusion strategy, which expects individuals are independent and identically distributed to train a robust weight-shared entity encoder. However, observed entity bias in various skeletal data violates this assumption, leading to suboptimal optimization of backbone models that might produce wrong recognition results. This bias arises from the world coordinate system's initial configuration, where the choice of origin often creates bias in representation. To this end, we propose a Convex Hull Adaptive Shift based normalization method to reduce Entity bias (CHASE), improving performance across a variety of skeleton-based action and interaction recognition tasks. To adaptively apply plausible shifts to the input skeletons, we formulate a plug-and-play parameterized network that ensures the relocated world origin lies within the skeleton convex hull, which avoids non-convergence by limiting the search space. To further minimize entity bias, we incorporate an auxiliary objective that leverages pair-wise distribution distances to guide network optimization. To support both single- and multi-entity actions, we propose a sub-entity strategy that offers a consistent formulation for both scenarios. Moreover, CHASE demonstrates compatibility with various intra-skeleton modalities, such as bones and velocities, highlighting its adaptability. Essentially, our method works as a normalization approach to reduce entity bias, enabling subsequent classifiers to achieve improved recognition performance across diverse settings. Extensive experiments on 7
datasets verify our approach by seamlessly integrating with various backbones and significantly boosting their performance.
Read the full abstract ▾
All authors Mengyuan Liu, Yuhang Wen, Yi Zhang, Songtao Wu, Hong Liu, Junsong Yuan, Beichen Ding
Computer Vision
CityToolVQA: Tool-Augmented Visual Question Answering for 3D Spatial Cognition in Urban Low-Altitude Environments
Boao Yu, Yingzhen Nie, Yue Hu et al. (+2 more)
CityToolVQA addresses the weak performance of
Vision-
Language Models (VLMs) on quantitative tasks in urban low-altitude visual question answering. We divide the seven tasks into qualitative and quantitative groups: qualitative questions are answered directly by the VLM, whereas quantitative questions are handled by an external visual-
geometric toolchain that performs object grounding, segmentation, depth back-projection, and spatial computation. The toolchain can be attached to different VLMs in a zero-shot manner; a Depth-Assisted
Prompt Inference (DAPI) fallback is triggered when the main-chain detection is invalid or unreliable, and CityToolVQA-SFT adapts the 8B backbone to tool-conditioned inputs. On the 73,324-question Open3D-VQA-v2 test set, CityToolVQA-SFT (Qwen3-VL-8B) reaches 67.6% overall accuracy, and attaching the toolchain to ten open-source VLMs improves quantitative-task accuracy by 12.7-36.9 percentage points. These results indicate that externalizing explicit 3D
geometric computation effectively complements the limited ability of RGB-only VLMs to estimate metric distances and object sizes.
Read the full abstract ▾
All authors Boao Yu, Yingzhen Nie, Yue Hu, Zhengqiu Zhu, Rusheng Ju
Computer Vision
Jian Tang, Jiawei Fan, Qiannan Zhou et al. (+3 more)
Diffusion Transformers (DiTs) have become the standard backbone for high-quality generative modeling, yet deploying them in conditional generation tasks remains computationally prohibitive because bidirectional joint
attention repeatedly processes large reference streams. While existing optimization schemes mitigate generic temporal redundancy, they typically rely on coarse-grained static reuse and overlook the distinct dynamics of references and targets. Specifically, we observe that reference representations often evolve slowly along the generation trajectory, while the target often assigns little
attention mass to them; reference drift and this target-to-reference exposure jointly shape how strongly stale reference states affect the target. To exploit these patterns, we introduce \RefAdapt, a training-free framework for adaptive control of joint
attention between references and targets. Instead of rigid static strategies, \RefAdapt combines consecutive target-Q change with previously observed target-to-reference
attention mass to control reference computation adaptively at block granularity. Under ultra-few-step settings, \RefAdapt enables speedups of up to $2.097\times$ on 4-step MiniMax H3 and $3.54\times$ on 8-step Qwen Image Edit, while maintaining comparable visual quality.
Read the full abstract ▾
All authors Jian Tang, Jiawei Fan, Qiannan Zhou, Qingbin Liu, Jiang Bian, Zang Li
Computer Vision
Toward On-Chip Training of Spiking Neural Networks for Dense Event-Based Vision
Maxime Vaillant, Axel Carlier, Lai Xing Ng et al. (+2 more)
Event cameras provide low-latency, asynchronous visual sensing for resource-constrained robotics. Spiking neural networks (SNNs) process event streams naturally, but training deep SNNs with backpropagation through time (BPTT) requires substantial
memory and remains difficult on neuromorphic hardware. Local learning avoids this by restricting error propagation to local blocks, but existing methods mainly target classification rather than dense prediction. We introduce DELL (Dense Event-driven Local Learning), a block wise scheme for dense event-based
vision that replaces global gradient propagation with local dense supervision. Learnable, spatially structured local heads supervise each block at its appropriate resolution while preserving temporal dynamics within blocks. We evaluate DELL on optical-flow regression and semantic segmentation with a fully spiking U-shaped architecture. On DSEC optical flow, DELL reduces peak training
memory by 39.6% relative to end-to-end BPTT while improving accuracy, reaching 1.670 px endpoint error on the official test
benchmark versus 1.941 px for the same backbone trained end-to-end. Block detachment behaves more like a regularizer than a constraint. DECOLLE, the existing local-learning baseline, relies on fixed random local read-outs poorly suited to dense regression, resulting in a 3.9x higher endpoint error; learnable local heads recover this loss and outperform end-to-end training across all optical-flow metrics. On segmentation, they recover most of the performance gap, although DELL remains a few mIoU points behind end-to-end training. With 2.3M parameters, 24x fewer than the strongest SNN baseline, the backbone remains competitive with the SNN state of the art on DSEC. These results extend local learning to dense event-based prediction while substantially reducing training
memory.
Read the full abstract ▾
All authors Maxime Vaillant, Axel Carlier, Lai Xing Ng, Christophe Hurter, Benoit R. Cottereau
Computer Vision
Endo-TSR: Temporal Spectral Modeling of Appearance and Motion for Endoscopic Reconstruction
Taoyu Wu, Yiyi Miao, Qi Shao et al. (+4 more)
Endoscopic scene reconstruction requires modeling tissue motion and temporal appearance while recovering fine surface detail. Deformable Gaussian models provide explicit trajectories, but their fixed colour coefficients lack a dedicated temporal representation for photometric changes. We propose Endo-TSR, which augments deformable Gaussian splatting with bounded
Fourier colour residuals and independent translation residuals on shared temporal frequencies. The colour residuals capture local appearance changes, while a Matérn spectral prior regularises motion corrections. Multi-scale
Laplacian supervision guides tissue-detail recovery during joint image fitting. Extensive experiments on the EndoNeRF and StereoMIS
datasets demonstrate state-of-the-art rendering quality, with the highest PSNR across all evaluated sequences. Ablation studies show that temporal appearance yields the largest PSNR gain among the tested component additions, while appearance and detail supervision jointly improve rendering with fixed Gaussian counts.
Read the full abstract ▾
All authors Taoyu Wu, Yiyi Miao, Qi Shao, Zhuoxiao Li, Zhe Tang, Limin Yu, Baoru Huang
Computer Vision
RefCompose: Multi-Reference Image Generation via LoRA-Conditioned Diffusion
Sai Sri Teja Kuppa, Parth Shinde, Priyadharsan Balaji S et al. (+2 more)
Filmmakers and visual artists routinely need to compose multiple references, actors, locations, props, cultural elements, into a single coherent shot, but existing tools either fail to scale past a handful of references or destroy fine grained subject identity in the process, since per reference tokenization scales
memory linearly with reference count $N$ and generated content often departs from the given references rather than reproducing them. We propose RefCompose, a pixel space compositional conditioning framework that decouples where things go from what they look like, via a single fixed resolution reference canvas that keeps conditioning size constant regardless of reference count. Spatial layout is induced at inference time from a frozen
diffusion transformer and extracted via Grounding DINO, requiring no LLM or dedicated layout model, while dual stream LoRA adapters inject a layout derived depth map and the encoded canvas through separate low rank streams, disentangling
geometric scaffolding from localized appearance. On the Dense Layout protocol, RefCompose consistently outperforms layout based and state of the art multi reference baselines on color, texture, shape, spatial accuracy, and identity/content preservation at higher reference counts, all with constant inference
memory, making it a practical building block for multi subject cinematic composition at production scale.
Read the full abstract ▾
All authors Sai Sri Teja Kuppa, Parth Shinde, Priyadharsan Balaji S, Jinka Harshavardhan, Sriprabha Ramanarayanan
Computer Vision
An End-to-End Latent-Rollout Approach for Pushing Few-Step ImageNet-$256$ Generation to FID $1.11$ without Fréchet Losses
Xiaoran Xu and Yujing Wang
Iterative generation poses a joint optimization problem across steps, as intermediate predictions shape subsequent computations and ultimately determine the final output distribution. Few-step generators distilled from pretrained
diffusion and flow-matching models make such optimization computationally practical end to end. We build on this opportunity with a distill-then-refine approach that uses teacher imitation to establish a strong initialization for a few-step rollout in latent space, then shifts to end-to-end refinement of the complete latent rollout against real data. We introduce FiST (Flow-in-Stage
Transformer), an architecture that composes learned latent-state transitions in a few stages using a shared
Transformer, with optional cross-stage hidden communication.
Distillation applies teacher-forced regression to selected states along teacher trajectories; refinement replaces this supervision with
adversarial and auxiliary classification objectives on the final latent output. A trainable discriminator module operates on semantically rich features extracted from clean real and generated latents by a frozen SiT backbone pretrained with REPA. All training takes place in latent space, without image decoding. During refinement, FiST consumes its own intermediate predictions, and endpoint gradients pass through every generation stage. For class-conditional generation on ImageNet at $256\times256$, our approach achieves FID 1.11 (IS 282) with three stages and FID 1.15 (IS 280) with two. These results demonstrate competitive few-step generation through learned distribution-level supervision, without explicit Fréchet-distance minimization. Ablations characterize how
distillation, pretrained checkpoint choices, refinement supervision, and cross-stage hidden communication affect generation quality.
Read the full abstract ▾
Computer Vision
StegGNN: Learning Graphical Representation for Image Steganography
Abhinav Kumar, Shorya Singhal, Agam Pandey et al. (+2 more)
Image steganography refers to
embedding secret messages within cover images while maintaining imperceptibility. Recent advances in deep learning - primarily driven by Convolutional Neural Networks (CNNs) and architectures such as inverse neural networks, autoencoders, and generative
adversarial networks - have led to notable progress. However, these frameworks are primarily built on CNN architectures, which treat images as regular grids and are limited by their receptive field size and a bias toward spatial locality. In parallel, Graph Neural Networks (GNNs) have recently demonstrated strong adaptability in several computer
vision tasks, achieving state-of-the-art performance with architectures such as
Vision GNN (ViG). This work moves in that direction and introduces StegGNN - a novel autoencoder-based, cover-agnostic image steganography framework based on GNNs. By modeling images as graph structures, our approach leverages the representational flexibility of GNNs over the grid-based rigidity of conventional CNNs. We conduct extensive experiments on standard
benchmark datasets to evaluate visual quality and imperceptibility. Our results show that our GNN-based method performs comparably to existing CNN
benchmarks. These findings suggest that GNNs provide a promising alternative representation for steganographic
embedding and open the field of deep learning-based steganography to further exploration of GNN-based architectures.
Read the full abstract ▾
All authors Abhinav Kumar, Shorya Singhal, Agam Pandey, Tushar Kumar, Sukrit Jindal
Computer Vision
Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation for Extreme Visual Token Reduction
Junxian Li, Ruixuan Yang, Tianao Zhang et al. (+3 more)
Visual
token reduction is an effective way to accelerate
multimodal large
language models (MLLMs), but performance deteriorates rapidly under extremely low
token budgets. Existing work has explored both visual-
token selection and training-based adaptation to reduced visual inputs. We take a step further by asking how a heavily compressed MLLM should learn from the states induced by its own generations. This setting naturally calls for on-policy self-
distillation: a heavily compressed model is supervised on the states induced by its own generations, while its full-
token counterpart serves as an information-rich teacher. Based on this insight, we propose LT-OPD, a training framework for extreme visual-
token reduction. The student rolls out responses with only a small fraction of visual
tokens, and a frozen full-
token copy of the same MLLM provides distributional supervision along these student-generated trajectories. To stabilize on-policy learning when visual evidence is severely limited, we further introduce a budget-level curriculum that progressively decreases the
token budget during training. Across nine
benchmarks on Qwen3.5-4B, LT-OPD raises average retained performance under 5% visual-
token retention from 68.6% to 82.3%, outperforming training-free, training-based, and reinforcement-learning baselines at the same budget. The gains transfer consistently to Qwen3.5-9B, GLM-4.6V-9B, and LLaVA-OV-1.5-4B. LT-OPD also reduces KV-cache usage by 85.2% and prefill FLOPs by 85.4% without additional inference overhead, demonstrating that on-policy learning can substantially recover capabilities lost to extreme visual-
token reduction.
Read the full abstract ▾
All authors Junxian Li, Ruixuan Yang, Tianao Zhang, Tiange Xu, Weisheng Dong, Yulun Zhang