Computer Vision
Evaluating an Adaptive Multispectral Turret System for Autonomous Tracking Across Variable Illumination Conditions
Aahan Sachdeva, Dhanvinkumar Ganeshkumar, James E. Gallagher et al. (+2 more)
Autonomous robotic platforms are playing a growing role across the emergency services sector, supporting missions such as search and rescue operations in disaster zones and reconnaissance. However, traditional red-green-blue (RGB) detection pipelines struggle in low-light environments, and thermal-based systems lack color and texture information. To overcome these limitations, we present an adaptive framework that fuses RGB and long-wave infrared (LWIR)
video streams at multiple fusion ratios and dynamically selects the optimal detection model for each illumination condition. We trained 33 You Only Look Once (YOLO) models on over 22,000 annotated images spanning three light levels: no-light (<10 lux), dim-light (10-1000 lux), and full-light (>1000 lux). To integrate both modalities, fusion was performed by blending aligned RGB and LWIR frames at eleven ratios, from full RGB (100/0) to full LWIR (0/100) in 10% increments. Evaluation showed that the best full-light model (80/20 RGB-LWIR) and dim-light model (90/10 fusion) achieved 92.8% and 92.0% mean confidence; both significantly outperformed the YOLOv5 nano (YOLOv5n) and YOLOv11 nano (YOLOv11n) baselines. Under no-light conditions, the top 40/60 fusion reached 71.0%, exceeding baselines though not statistically significant. Adaptive RGB-LWIR fusion improved detection confidence and reliability across all illumination conditions, enhancing autonomous robotic
vision performance.
Read the full abstract ▾
All authors Aahan Sachdeva, Dhanvinkumar Ganeshkumar, James E. Gallagher, Tyler Treat, Edward J. Oughton
Computer Vision · Cross-listed from cs.LG
Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Models
Zheng Xing and Weibing Zhao
Unsupervised human motion
segmentation (HMS) can be effectively achieved using subspace clustering techniques. However, traditional methods overlook the role of temporal semantic exploration in HMS. This paper explores the use of temporal
vision semantics (TVS) derived from human motion sequences, leveraging the image-to-text capabilities of a large
language model (LLM) to enhance subspace clustering performance. The core idea is to extract textual motion information from consecutive frames via LLM and incorporate this learned information into the subspace clustering framework. The primary challenge lies in learning TVS from human motion sequences using LLM and integrating this information into subspace clustering. To address this, we determine whether consecutive frames depict the same motion by querying the LLM and subsequently learn temporal neighboring information based on its response. We then develop a TVS-integrated subspace clustering approach, incorporating subspace
embedding with a temporal regularizer that induces each frame to share similar subspace
embeddings with its temporal neighbors. Additionally,
segmentation is performed based on subspace
embedding with a temporal constraint that induces the grouping of each frame with its temporal neighbors. We also introduce a feedback-enabled framework that continuously optimizes subspace
embedding based on the
segmentation output. Experimental results demonstrate that the proposed method outperforms existing state-of-the-art approaches on four
benchmark human motion
datasets.
Read the full abstract ▾
Computer Vision
Scalable Deep Subspace Clustering Network
Nairouz Mrabah, Mohamed Bouguessa and Sihem Sami
Subspace clustering methods face inherent scalability limits due to the $O(n^3)$ cost (with $n$ denoting the number of data samples) of constructing full $n\times n$ affinities and performing spectral decomposition. While deep learning-based approaches improve feature extraction, they maintain this computational bottleneck through exhaustive pairwise similarity computations. We propose SDSNet (Scalable Deep Subspace Network), a deep subspace clustering framework that achieves $\mathcal{O}(n)$ complexity through (1) landmark-based approximation, avoiding full affinity
matrices, (2) joint optimization of auto-encoder reconstruction with self-expression objectives, and (3) direct spectral clustering on factorized representations. The framework combines convolutional auto-encoders with subspace-preserving constraints. Experimental results demonstrate that SDSNet achieves comparable clustering quality to state-of-the-art methods with significantly improved computational efficiency.
Read the full abstract ▾
Computer Vision
A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding
Christina Liu, Alan Q. Wang, Joy Hsu et al. (+2 more)
Recent tool-use frameworks powered by
vision-
language models (VLMs) improve image understanding by grounding model predictions with specialized tools. Broadly, these frameworks leverage VLMs and a pre-specified toolbox to decompose the prediction task into multiple tool calls (often deep learning models) which are composed to make a prediction. The dominant approach to composing tools is using text, via function calls embedded in VLM-generated code or natural
language. However, these methods often perform poorly on medical image understanding, where salient information is encoded as spatially-localized features that are difficult to compose or fuse via text alone. To address this, we propose a tool-use framework for medical image understanding called the Tool Bottleneck Framework (TBF), which composes VLM-selected tools using a learned Tool Bottleneck Model (TBM). For a given image and task, TBF leverages an off-the-shelf medical VLM to select tools from a toolbox that each extract clinically-relevant features. Instead of text-based composition, these tools are composed by the TBM, which computes and fuses the tool outputs using a neural network before outputting the final prediction. We propose a simple and effective strategy for TBMs to make predictions with any arbitrary VLM tool selection. Overall, our framework not only improves tool-use in medical imaging contexts, but also yields more interpretable, clinically-grounded predictors. We evaluate TBF on tasks in histopathology and dermatology and find that these advantages enable our framework to perform on par with or better than deep learning-based classifiers, VLMs, and state-of-the-art tool-use frameworks, with particular gains in data-limited regimes. The project details and the code are available at https://christinaliu2020.github.io/tbm/.
Read the full abstract ▾
All authors Christina Liu, Alan Q. Wang, Joy Hsu, Jiajun Wu, Ehsan Adeli
Computer Vision
Understanding Virality: A Rubric based Vision-Language Model Framework for Short-Form Edutainment Evaluation
Arnav Gupta, Gurekas Singh Sahney, Hardik Rathi et al. (+4 more)
Evaluating short-form
video content requires moving beyond surface-level quality metrics toward human-aligned,
multimodal reasoning. While existing frameworks like VideoScore-2 assess visual and semantic fidelity, they do not capture how specific audiovisual attributes drive real audience engagement. In this work, we propose a data-driven evaluation framework that uses
Vision-
Language Models (VLMs) to extract unsupervised audiovisual features, clusters them into interpretable factors, and trains a regression-based evaluator to predict engagement on short-form edutainment
videos. Our curated YouTube Shorts
dataset enables systematic analysis of how VLM-derived features relate to human engagement behavior. Experiments show strong correlations between predicted and actual engagement, demonstrating that our lightweight, feature-based evaluator provides interpretable and scalable assessments compared to traditional metrics (e.g., SSIM, FID). By grounding evaluation in both
multimodal feature importance and human-centered engagement signals, our approach advances toward robust and explainable
video understanding.
Read the full abstract ▾
All authors Arnav Gupta, Gurekas Singh Sahney, Hardik Rathi, Abhishek Chandwani, Ishaan Gupta, Pratik Narang, Dhruv Kumar
Computer Vision · Cross-listed from eess.IV
A Graph-Augmented knowledge Distillation based Dual-Stream Vision Transformer with Region-Aware Attention for Gastrointestinal Disease Classification with Explainable AI
Md Assaduzzaman, Nushrat Jahan Oyshi and Eram Mahamud
The accurate classification of gastrointestinal diseases from endoscopic and histopathological imagery remains a significant challenge in medical diagnostics, mainly due to the vast data volume and subtle variation in inter-class visuals. This study presents a hybrid dual-stream deep learning framework built on teacher-student knowledge
distillation, where a high-capacity teacher model integrates the global contextual
reasoning of a Swin
Transformer with the local fine-grained feature extraction of a
Vision Transformer. The student network was implemented as a compact Tiny-ViT structure that inherits the teacher's semantic and morphological knowledge via soft-label
distillation, achieving a balance between efficiency and diagnostic accuracy. Two carefully curated Wireless Capsule Endoscopy
datasets, encompassing major GI disease classes, were employed to ensure balanced representation and prevent inter-sample bias. The proposed framework achieved remarkable performance with accuracies of 0.9978 and 0.9928 on
Dataset 1 and
Dataset 2 respectively, and an average AUC of 1.0000, signifying near-perfect discriminative capability. Interpretability analyses using Grad-CAM, LIME, and Score-CAM confirmed that the model's predictions were grounded in clinically significant tissue regions and pathologically relevant morphological cues, validating the framework's transparency and reliability. The Tiny-ViT demonstrated diagnostic performance with reduced computational complexity comparable to its
transformer-based teacher while delivering faster inference, making it suitable for resource-constrained clinical environments. Overall, the proposed framework provides a robust, interpretable, and scalable solution for AI-assisted GI disease diagnosis, paving the way toward future intelligent endoscopic screening that is compatible with clinical practicality.
Read the full abstract ▾
Computer Vision
HiStream: Efficient High-Resolution Video Generation via Redundancy-Eliminated Streaming
Haonan Qiu, Shikun Liu, Zijian Zhou et al. (+10 more)
High-resolution
video generation, while crucial for digital media and film, is computationally bottlenecked by the quadratic complexity of
diffusion models, making practical inference infeasible. To address this, we introduce HiStream, an efficient autoregressive framework that systematically reduces redundancy across three axes: i) Spatial Compression: denoising at low resolution before refining at high resolution with cached features; ii) Temporal Compression: a chunk-by-chunk strategy with a fixed-size anchor cache, ensuring stable inference speed; and iii) Timestep Compression: applying fewer denoising steps to subsequent, cache-conditioned chunks. On 1080p
benchmarks, our primary HiStream model (i+ii) achieves state-of-the-art visual quality while demonstrating up to 76.2x faster denoising compared to the Wan2.1 baseline and negligible quality loss. Our faster variant, HiStream+, applies all three optimizations (i+ii+iii), achieving a 107.5x acceleration over the baseline, offering a compelling trade-off between speed and quality, thereby making high-resolution
video generation both practical and scalable.
Read the full abstract ▾
All authors Haonan Qiu, Shikun Liu, Zijian Zhou, Zhaochong An, Weiming Ren, Zhiheng Liu, Jonas Schult, Sen He, Shoufa Chen, Yuren Cong, Tao Xiang, Ziwei Liu, Juan-Manuel Perez-Rua
Computer Vision
Beyond Memorization: A Multi-Modal Ordinal Regression Benchmark to Expose Popularity Bias in Vision-Language Models
Li-Zhong Szu-Tu, Ting-Lin Wu, Chia-Jui Chang et al. (+2 more)
We expose a significant popularity bias in state-of-the-art
vision-
language models (VLMs), which achieve up to 34% higher accuracy on famous buildings compared to ordinary ones, indicating a reliance on memorization over generalizable understanding. To systematically investigate this, we introduce the largest open
benchmark for this task: the YearGuessr
dataset, a collection of 55,546 building images with multi-modal attributes from 157 countries, annotated with continuous ordinal labels of their construction year (1001-2024), GPS data, and page-view counts as a proxy for popularity. Using this
dataset, we frame the construction year prediction task as ordinal regression and introduce popularity-aware interval accuracy metrics to quantify this bias. Our resulting
benchmark of 30+ models, including our YearCLIP model, confirms that VLMs excel on popular, memorized items but struggle significantly with unrecognized subjects, exposing a critical flaw in their
reasoning capabilities. Project page: https://sytwu.github.io/BeyondMemo/
Read the full abstract ▾
All authors Li-Zhong Szu-Tu, Ting-Lin Wu, Chia-Jui Chang, He Syu, Yu-Lun Liu
Computer Vision
Streaming Video Instruction Tuning
Jiaer Xia, Peixian Chen, Mengdan Zhang et al. (+2 more)
We present Streamo, a real-time streaming
video LLM that serves as a general-purpose interactive assistant. Unlike existing online
video models that focus narrowly on question answering or captioning, Streamo performs a broad spectrum of streaming
video tasks, including real-time narration, action understanding, event captioning, temporal event grounding, and time-sensitive question answering. To develop such versatility, we construct Streamo-Instruct-465K, a large-scale instruction-following
dataset tailored for streaming
video understanding. The
dataset covers diverse temporal contexts and multi-task supervision, enabling unified training across heterogeneous streaming tasks. After training end-to-end on the instruction-following
dataset through a streamlined pipeline, Streamo exhibits strong temporal
reasoning, responsive interaction, and broad generalization across a variety of streaming
benchmarks. Extensive experiments show that Streamo bridges the gap between offline
video perception models and real-time
multimodal assistants, making a step toward unified, intelligent
video understanding in continuous
video streams.
Read the full abstract ▾
All authors Jiaer Xia, Peixian Chen, Mengdan Zhang, Xing Sun, Kaiyang Zhou
Computer Vision
Fast SAM2 with Text-Driven Token Pruning
Avilasha Mandal, Chaoning Zhang, Fachrina Dewi Puspitasari et al. (+6 more)
Segment Anything Model 2 (SAM2), a
vision foundation model has significantly advanced in
prompt-driven
video object
segmentation, yet their practical deployment remains limited by the high computational and
memory cost of processing dense visual
tokens across time. The SAM2 pipelines typically propagate all visual
tokens produced by the image encoder through downstream temporal
reasoning modules, regardless of their relevance to the target object, resulting in reduced scalability due to quadratic
memory attention overhead. In this work, we introduce a text-guided
token pruning framework that improves inference efficiency by selectively reducing
token density prior to temporal propagation, without modifying the underlying
segmentation architecture. Operating after visual encoding and before
memory based propagation, our method ranks
tokens using a lightweight routing mechanism that integrates local visual context, semantic relevance derived from object-centric textual descriptions (either user-provided or automatically generated), and uncertainty cues that help preserve ambiguous or boundary critical regions. By retaining only the most informative
tokens for downstream processing, the proposed approach reduces redundant computation while maintaining
segmentation fidelity. Extensive experiments across multiple challenging
video segmentation benchmarks demonstrate that post-encoder
token pruning provides a practical and effective pathway to efficient,
prompt-aware
video segmentation, achieving up to 42.50 percent faster inference and 37.41 percent lower GPU
memory usage compared to the unpruned baseline SAM2, while preserving competitive J and F performance. These results highlight the potential of early
token selection to improve the scalability of
transformer-based
video segmentation systems for real-time and resource-constrained applications.
Read the full abstract ▾
All authors Avilasha Mandal, Chaoning Zhang, Fachrina Dewi Puspitasari, Xudong Wang, Jiaquan Zhang, Caiyan Qin, Guoqing Wang, Yang Yang, Heng Tao Shen
Computer Vision
TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learning
Varun Belagali, Saarthak Kapse, Pierre Marza et al. (+12 more)
The interpretation of small tiles in large whole slide images (WSI) often needs a larger image context. We introduce TICON, a
transformer-based tile representation contextualizer that produces rich, contextualized
embeddings for ”any” application in computational pathology. Standard tile encoder-based pipelines, which extract
embeddings of tiles stripped from their context, fail to model the rich slide-level information essential for both local and global tasks. Furthermore, different tile-encoders excel at different downstream tasks. Therefore, a unified model is needed to contextualize
embeddings derived from ”any” tile-level foundation model. TICON addresses this need with a single, shared encoder, pretrained using a masked modeling objective to simultaneously unify and contextualize representations from diverse tile-level pathology foundation models. Our experiments demonstrate that TICON-contextualized
embeddings significantly improve performance across many different tasks, establishing new state-of-the-art results on tile-level
benchmarks (i.e., HEST-Bench, THUNDER, CATCH) and slide-level
benchmarks (i.e., Patho-Bench). Finally, we pretrain an aggregator on TICON to form a slide-level foundation model, using only 11K WSIs, outperforming SoTA slide-level foundation models pretrained with up to 350K WSIs.
Read the full abstract ▾
All authors Varun Belagali, Saarthak Kapse, Pierre Marza, Srijan Das, Zilinghan Li, Sofiène Boutaj, Pushpak Pati, Srikar Yellapragada, Tarak Nath Nandi, Ravi K Madduri, Joel Saltz, Prateek Prasanna, Stergios Christodoulidis, Maria Vakalopoulou, Dimitris Samaras
Computer Vision · Cross-listed from cs.LG
Does the Data Processing Inequality Reflect Practice? On the Utility of Low-Level Tasks
Roy Turgeman and Tom Tirer
The data processing inequality is an information-theoretic principle stating that the information content of a signal cannot be increased by processing the observations. In particular, it suggests that there is no benefit in enhancing the signal or encoding it before addressing a classification problem. This assertion can be proven to be true for the case of the optimal Bayes classifier. However, in practice, it is common to perform "low-level" tasks before "high-level" downstream tasks despite the overwhelming capabilities of modern deep neural networks. In this paper, we aim to understand when and why low-level processing can be beneficial for classification. We present a comprehensive theoretical study of a binary classification setup, where we consider a classifier that is tightly connected to the optimal Bayes classifier and converges to it as the number of training samples increases. We prove that for any finite number of training samples, there exists a pre-classification processing that improves the classification accuracy. We also explore the effect of class separation, training set size, and class balance on the relative gain from this procedure. We support our theory with an empirical investigation of the theoretical setup. Finally, we conduct an empirical study where we investigate the effect of denoising and encoding on the performance of practical deep classifiers on
benchmark datasets. Specifically, we vary the size and class distribution of the training set, and the noise level, and demonstrate trends that are consistent with our theoretical results.
Read the full abstract ▾
Computer Vision
AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents
Yue Cao, Yingyao Wang, Pi Bu et al. (+10 more)
Graphical user interface (GUI)
agents can substantially improve productivity by automating frequently executed long-latency tasks on mobile devices. However, existing evaluation
benchmarks are still constrained to limited applications, simple tasks, and coarse-grained metrics. To address this, we introduce AndroidLens, a challenging evaluation framework for mobile GUI
agents, comprising 571 long-latency tasks in both Chinese and English environments, each requiring an average of more than 26 steps to complete. The framework features: (1) tasks derived from real-world user scenarios across 38 domains, covering complex types such as multi-constraint, multi-goal, and domain-specific tasks; (2) static evaluation that preserves real-world anomalies and allows multiple valid paths to reduce bias; and (3) dynamic evaluation that employs a milestone-based scheme for fine-grained progress measurement via Average Task Progress (ATP). Our evaluation indicates that even the best models reach only a 12.7% task success rate and 50.47% ATP. We also underscore key challenges in real-world environments, including environmental anomalies, adaptive exploration, and long-term
memory retention.
Read the full abstract ▾
All authors Yue Cao, Yingyao Wang, Pi Bu, Jingxuan Xing, Wei Jiang, Zekun Zhu, Junpeng Ma, Sashuai Zhou, Tong Lu, Jun Song, Yu Cheng, Yuning Jiang, Bo Zheng
Computer Vision
Post-Processing Mask-Based Table Segmentation for Structural Coordinate Extraction
Suren Bandara
Structured data extraction from tables plays a crucial role in document image analysis for scanned documents and digital archives. Although many methods have been proposed to detect table structures and extract
cell contents, accurately identifying table segment boundaries (rows and columns) remains challenging, particularly in low-resolution or noisy images. In many real-world scenarios, table data are incomplete or degraded, limiting the adaptability of
transformer-based methods to noisy inputs. Mask-based edge detection techniques have shown greater robustness under such conditions, as their sensitivity can be adjusted through threshold tuning; however, existing approaches typically apply masks directly to images, leading to noise sensitivity, resolution loss, or high computational cost. This paper proposes a novel multi-scale signal-processing method for detecting table edges from table masks. Row and column transitions are modeled as one-dimensional signals and processed using Gaussian convolution with progressively increasing variances, followed by statistical thresholding to suppress noise while preserving stable structural edges. Detected signal peaks are mapped back to image coordinates to obtain accurate segment boundaries. Experimental results show that applying the proposed approach to column edge detection improves
Cell-Aware
Segmentation Accuracy (CASA) a layout-aware metric evaluating both textual correctness and correct
cell placement from 67% to 76% on the PubLayNet-1M
benchmark when using TableNet with PyTesseract OCR. The method is robust to resolution variations through zero-padding and scaling strategies and produces optimized structured tabular outputs suitable for downstream analysis.
Read the full abstract ▾
Computer Vision
Toward Real-Time Surgical Scene Segmentation via a Spike-Driven Video Transformer with Spike-Informed Pretraining
Shihao Zou, Jingjing Li, Wei Ji et al. (+5 more)
Modern
surgical systems increasingly rely on intelligent scene understanding to improve intra-operative safety and situational awareness, with
surgical scene
segmentation playing a fundamental role in fine-grained
surgical perception. Although recent ANN models, especially large foundation models, have achieved impressive accuracy, their high computational and energy demands often hinder deployment in resource-constrained operative environments. To address this challenge, we explore SNN as a highly efficient paradigm. However, its performance in
surgical scene
segmentation remains constrained by sparse spike representations and limited annotated
surgical data. We therefore propose SpikeSurgSeg, the first spike-driven
video Transformer for
surgical scene
segmentation. It preserves the real-time and energy-efficient advantages of SNN, while achieving competitive performance against most ANN models in data-scarce
surgical scenarios. Specifically, we introudce a spike-informed pretraining strategy based on MAE, where mask generation is guided by spike firing activity to better align with sparse spike representations, together with a layer-wise tube masking scheme that reduces information leakage and encourages contextual
reasoning. To further strengthen semantic representation, we introduce multi-spectral knowledge
distillation, which aligns teacher ANN and student SNN features in the frequency domain, where the mismatch between continuous activation patterns and spike-driven temporal representations can be effectively mitigated. Built on the pretrained SNN encoder, we further design a lightweight spike-driven
segmentation head. Extensive experiments on EndoVis18 and our in-house SurgBleed
dataset show that SpikeSurgSeg achieves mIoU comparable to SOTA ANN models while reducing inference latency by at least 8x. Notably, it delivers over 20x speedup relative to most foundation models.
Read the full abstract ▾
All authors Shihao Zou, Jingjing Li, Wei Ji, Jincai Huang, Kai Wang, Guo Dan, Weixin Si, Yi Pan
Computer Vision
GriDiT: Factorized Grid-Based Diffusion for Efficient Long Image Sequence Generation
Snehal Singh Tomar, Alexandros Graikos, Arjun Krishna et al. (+2 more)
Modern deep learning methods typically treat image sequences as large tensors of sequentially stacked frames. However, is this straightforward representation ideal given the current state-of-the-art (SoTA)? In this work, we address this question in the context of generative models and aim to devise a more effective way of modeling image sequence data. Observing the inefficiencies and bottlenecks of current SoTA image sequence generation methods, we showcase that rather than working with large tensors, we can improve the generation process by factorizing it into first generating the coarse sequence at low resolution and then refining the individual frames at high resolution. We train a generative model solely on grid images comprising subsampled frames. Yet, we learn to generate image sequences, using the strong self-
attention mechanism of the
Diffusion Transformer (DiT) to capture correlations between frames. In effect, our formulation extends a 2D image generator to operate as a low-resolution 3D image-sequence generator without introducing any architectural modifications. Subsequently, we super-resolve each frame individually to add the sequence-independent high-resolution details. This approach offers several advantages and can overcome key limitations of the SoTA in this domain. Compared to existing image sequence generation models, our method achieves superior synthesis quality and improved coherence across sequences. It also delivers high-fidelity generation of arbitrary-length sequences and increased efficiency in inference time and training data usage. Furthermore, our straightforward formulation enables our method to generalize effectively across diverse data domains, which typically require additional priors and supervision to model in a generative context. Our method consistently outperforms SoTA in quality and inference speed (at least twice-as-fast) across
datasets.
Read the full abstract ▾
All authors Snehal Singh Tomar, Alexandros Graikos, Arjun Krishna, Dimitris Samaras, Klaus Mueller
Computer Vision
ACD: Direct Conditional Control for Video Diffusion Models via Attention Supervision
Weiqi Li, Zehao Zhang, Liang Lin et al. (+1 more)
Controllability is a fundamental requirement in
video synthesis, where accurate
alignment with conditioning signals is essential. Existing classifier-free guidance methods typically achieve conditioning indirectly by modeling the joint distribution of data and conditions, which often results in limited controllability over the specified conditions. Classifier-based guidance enforces conditions through an external classifier, but the model may exploit this mechanism to raise the classifier score without genuinely satisfying the intended condition, resulting in
adversarial artifacts and limited effective controllability. In this paper, we propose
Attention-Conditional
Diffusion (ACD), a novel framework for direct conditional control in
video diffusion models via
attention supervision. By aligning the model's
attention maps with external control signals, ACD achieves better controllability. To support this, we introduce a sparse 3D-aware object layout as an efficient conditioning signal, along with a dedicated Layout ControlNet and an automated annotation pipeline for scalable layout integration. Extensive experiments on
benchmark video generation
datasets demonstrate that ACD delivers superior
alignment with conditioning inputs while preserving temporal coherence and visual fidelity, establishing an effective paradigm for conditional
video synthesis.
Read the full abstract ▾
All authors Weiqi Li, Zehao Zhang, Liang Lin, Guangrun Wang
Computer Vision
AnyAD: Unified Any-Modality Anomaly Detection in Incomplete Multi-Sequence MRI
Changwei Wu, Yifei Chen, Yuxin Du et al. (+7 more)
Reliable anomaly detection in
brain MRI remains challenging due to the scarcity of annotated abnormal cases and the frequent absence of key imaging modalities in real clinical workflows. Existing single-class or multi-class anomaly detection (AD) models typically rely on fixed modality configurations, require repetitive training, or fail to generalize to unseen modality combinations, limiting their clinical scalability. In this work, we present a unified Any-Modality AD framework that performs robust anomaly detection and localization under arbitrary MRI modality availability. The framework integrates a dual-pathway DINOv2 encoder with a feature distribution
alignment mechanism that statistically aligns incomplete-modality features with full-modality representations, enabling stable inference even with severe modality dropout. To further enhance semantic consistency, we introduce an Intrinsic Normal Prototypes (INPs) extractor and an INP-guided decoder that reconstruct only normal anatomical patterns while naturally amplifying abnormal deviations. Through randomized modality masking and indirect feature completion during training, the model learns to adapt to all modality configurations without re-training. Extensive experiments on BraTS2018, MU-Glioma-Post, and Pretreat-MetsToBrain-Masks demonstrate that our approach consistently surpasses state-of-the-art industrial and medical AD baselines across 7 modality combinations, achieving superior generalization. This study establishes a scalable paradigm for
multimodal medical AD under real-world, imperfect modality conditions. Our source code is available at https://github.com/wuchangw/AnyAD.
Read the full abstract ▾
All authors Changwei Wu, Yifei Chen, Yuxin Du, Mingxuan Liu, Jinying Zong, Beining Wu, Jie Dong, Feiwei Qin, Yunkang Cao, Qiyuan Tian
Computer Vision
DreaMontage: Arbitrary Frame-Guided One-Shot Video Generation
Jiawei Liu, Junqiao Li, Jiangfan Deng et al. (+11 more)
The "one-shot" technique represents a distinct and sophisticated aesthetic in filmmaking. However, its practical realization is often hindered by prohibitive costs and complex real-world constraints. Although emerging
video generation models offer a virtual alternative, existing approaches typically rely on naive clip concatenation, which frequently fails to maintain visual smoothness and temporal coherence. In this paper, we introduce DreaMontage, a comprehensive framework designed for arbitrary frame-guided generation, capable of synthesizing seamless, expressive, and long-duration one-shot
videos from diverse user-provided inputs. To achieve this, we address the challenge through three primary dimensions. (i) We integrate a lightweight intermediate-conditioning mechanism into the DiT architecture. By employing an Adaptive Tuning strategy that effectively leverages base training data, we unlock robust arbitrary-frame control capabilities. (ii) To enhance visual fidelity and cinematic expressiveness, we curate a high-quality
dataset and implement a Visual Expression SFT stage. In addressing critical issues such as subject motion rationality and transition smoothness, we apply a Tailored DPO scheme, which significantly improves the success rate and usability of the generated content. (iii) To facilitate the production of extended sequences, we design a Segment-wise Auto-Regressive (SAR) inference strategy that operates in a
memory-efficient manner. Extensive experiments demonstrate that our approach achieves visually striking and seamlessly coherent one-shot effects while maintaining computational efficiency, empowering users to transform fragmented visual materials into vivid, cohesive one-shot cinematic experiences.
Read the full abstract ▾
All authors Jiawei Liu, Junqiao Li, Jiangfan Deng, Gen Li, Siyu Zhou, Zetao Fang, Shanshan Lao, Zengde Deng, Jianing Zhu, Tingting Ma, Jiayi Li, Yunqiu Wang, Qian He, Xinglong Wu
Computer Vision · Cross-listed from cs.LG
Improving the Convergence Rate of Ray Search Optimization for Query-Efficient Hard-Label Attacks
Xinjie Xu, Shuyu Cheng, Dongwei Xu et al. (+2 more)
In hard-label black-box
adversarial attacks, where only the top-1 predicted label is accessible, the prohibitive query complexity poses a major obstacle to practical deployment. In this paper, we focus on optimizing a representative class of attacks that search for the optimal ray direction yielding the minimum $\ell_2$-norm perturbation required to move a benign image into the
adversarial region. Inspired by Nesterov's Accelerated Gradient (NAG), we propose a momentum-based
algorithm, ARS-OPT, which proactively estimates the gradient with respect to a future ray direction inferred from accumulated momentum. We provide a theoretical analysis of its convergence behavior, showing that ARS-OPT enables more accurate directional updates and achieves faster, more stable optimization. To further accelerate convergence, we incorporate surrogate-model priors into ARS-OPT's gradient estimation, resulting in PARS-OPT with enhanced performance. The superiority of our approach is supported by theoretical guarantees under standard assumptions. Extensive experiments on ImageNet and CIFAR-10 demonstrate that our method surpasses 13 state-of-the-art approaches in query efficiency.
Read the full abstract ▾
All authors Xinjie Xu, Shuyu Cheng, Dongwei Xu, Qi Xuan, Chen Ma
Computer Vision
SegMo: Segment-aligned Text to 3D Human Motion Generation
Bowen Dang, Lin Wu, Xiaohang Yang et al. (+2 more)
Generating 3D human motions from textual descriptions is an important research problem with broad applications in
video games, virtual reality, and augmented reality. Recent methods align the textual description with human motion at the sequence level, neglecting the internal semantic structure of modalities. However, both motion descriptions and motion sequences can be naturally decomposed into smaller and semantically coherent segments, which can serve as atomic
alignment units to achieve finer-grained correspondence. Motivated by this, we propose SegMo, a novel Segment-aligned text-conditioned human Motion generation framework to achieve fine-grained text-motion
alignment. Our framework consists of three modules: (1) Text Segment Extraction, which decomposes complex textual descriptions into temporally ordered phrases, each representing a simple atomic action; (2) Motion Segment Extraction, which partitions complete motion sequences into corresponding motion segments; and (3) Fine-grained Text-Motion
Alignment, which aligns text and motion segments with contrastive learning. Extensive experiments demonstrate that SegMo improves the strong baseline on two widely used
datasets, achieving an improved TOP 1 score of 0.553 on the HumanML3D test set. Moreover, thanks to the learned shared
embedding space for text and motion segments, SegMo can also be applied to
retrieval-style tasks such as motion grounding and motion-to-text
retrieval.
Read the full abstract ▾
All authors Bowen Dang, Lin Wu, Xiaohang Yang, Zheng Yuan, Zhixiang Chen
Computer Vision
Leveraging Lightweight Entity Extraction for Scalable Event-Based Image Retrieval
Dao Sy Duy Minh, Huynh Trung Kiet, Nguyen Lam Phu Quy et al. (+2 more)
Retrieving images from natural
language descriptions is a core task at the intersection of computer
vision and natural
language processing, with wide-ranging applications in search engines, media archiving, and digital content management. However, real-world image-text
retrieval remains challenging due to vague or context-dependent queries, linguistic variability, and the need for scalable solutions. In this work, we propose a lightweight two-stage
retrieval pipeline that leverages event-centric entity extraction to incorporate temporal and contextual signals from real-world captions. The first stage performs efficient candidate filtering using BM25 based on salient entities, while the second stage applies BEiT-3 models to capture deep
multimodal semantics and rerank the results. Evaluated on the OpenEvents v1
benchmark, our method achieves a mean average precision of 0.559, substantially outperforming prior baselines. These results highlight the effectiveness of combining event-guided filtering with long-text
vision-
language modeling for accurate and efficient
retrieval in complex, real-world scenarios. Our code is available at https://github.com/PhamPhuHoa-23/Event-Based-Image-
Retrieval
Read the full abstract ▾
All authors Dao Sy Duy Minh, Huynh Trung Kiet, Nguyen Lam Phu Quy, Phu-Hoa Pham, Tran Chi Nguyen
Computer Vision · Cross-listed from cs.AI
RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic
Le Wang, Zonghao Ying, Xiao Yang et al. (+7 more)
Embodied agents powered by
vision-
language models (VLMs) are increasingly capable of executing complex real-world tasks, yet they remain vulnerable to hazardous instructions that may trigger unsafe behaviors. Runtime safety guardrails, which intercept hazardous actions during task execution, offer a promising solution due to their flexibility. However, existing defenses often rely on static rule filters or
prompt-level control, which struggle to address implicit risks arising in dynamic, temporally dependent, and context-rich environments. To address this, we propose RoboSafe, a hybrid
reasoning runtime safeguard for
embodied agents through executable predicate-based safety logic. RoboSafe integrates two complementary
reasoning processes on a Hybrid Long-Short Safety
Memory. We first propose a Backward Reflective
Reasoning module that continuously revisits recent trajectories in short-term
memory to infer temporal safety predicates and proactively triggers replanning when violations are detected. We then propose a Forward Predictive
Reasoning module that anticipates upcoming risks by generating context-aware safety predicates from the long-term safety
memory and the
agent's
multimodal observations. Together, these components form an adaptive, verifiable safety logic that is both interpretable and executable as code. Extensive experiments across multiple
agents demonstrate that RoboSafe substantially reduces hazardous actions (-36.8% risk occurrence) compared with leading baselines, while maintaining near-original task performance. Real-world evaluations on physical robotic arms further confirm its practicality. Code will be released upon acceptance.
Read the full abstract ▾
All authors Le Wang, Zonghao Ying, Xiao Yang, Quanchen Zou, Zhenfei Yin, Tianlin Li, Jian Yang, Yaodong Yang, Aishan Liu, Xianglong Liu
Computer Vision
Latent Implicit Visual Reasoning
Kelvin Li, Chuyi Shang, Leonid Karlinsky et al. (+3 more)
While Large
Multimodal Models (LMMs) have made significant progress, they remain largely text-centric, relying on
language as their core
reasoning modality. As a result, they are limited in their ability to handle
reasoning tasks that are predominantly visual. Recent approaches have sought to address this by supervising intermediate visual steps with helper images, depth maps, or image crops. However, these strategies impose restrictive priors on what "useful" visual abstractions look like, add heavy annotation costs, and struggle to generalize across tasks. To address this critical limitation, we propose Latent Implicit Visual
Reasoning (LIVR), a task-agnostic mechanism that trains LMMs to discover and use latent visual
reasoning tokens without explicit intermediate supervision. These
tokens attend globally and re-encode the image in a task-adaptive way, enabling the model to extract relevant visual information without hand-crafted supervision. LIVR consistently outperforms direct supervised fine-tuning across diverse
vision-centric tasks and multiple LMM backbones. In broader comparisons, LIVR remains competitive with or outperforms prior text-based and explicit-visual-intermediate
reasoning methods, while requiring no additional intermediate supervision such as helper images, bounding boxes, image crops, depth maps, or chain-of-thought annotations. Our project page can be found here: https://www.chuyishang.com/livr/
Read the full abstract ▾
All authors Kelvin Li, Chuyi Shang, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, Roei Herzig
Computer Vision
Human Motion Estimation with Everyday Wearables
Siqi Zhu, Yixuan Li, Junfu Li et al. (+4 more)
While on-body device-based human motion estimation is crucial for applications such as XR interaction, existing methods often suffer from poor wearability, expensive hardware, and cumbersome calibration, which hinder their adoption in daily life. To address these challenges, we present EveryWear, a lightweight and practical human motion capture approach based entirely on everyday wearables: a smartphone, smartwatch, earbuds, and smart glasses equipped with one forward-facing and two downward-facing cameras, requiring no explicit calibration before use. We introduce Ego-Elec, a 9-hour real-world
dataset covering 56 daily activities across 17 diverse indoor and outdoor environments, with ground-truth 3D annotations provided by the motion capture (MoCap), to facilitate robust research and benchmarking in this direction. Our approach employs a
multimodal teacher-student framework that integrates visual cues from
egocentric cameras with inertial signals from consumer devices. By training directly on real-world data rather than synthetic data, our model effectively eliminates the sim-to-real gap that constrains prior work. Experiments demonstrate that our method outperforms baseline models, validating its effectiveness for practical full-body motion estimation.
Read the full abstract ▾
All authors Siqi Zhu, Yixuan Li, Junfu Li, Qi Wu, Zan Wang, Haozhe Ma, Wei Liang
Computer Vision · Cross-listed from cs.RO
Schrödinger's Navigator: Imagining an Ensemble of Futures for Zero-Shot Object Navigation
Yu He, Da Huang, Zhenyang Liu et al. (+5 more)
Zero-shot object
navigation (ZSON) requires
robots to find target objects in unseen environments without task-specific fine-tuning or pre-built maps, a key capability for general-purpose service
robots. Yet methods that perform well in simulation often degrade in cluttered real-world scenes with severe occlusion and latent hazards, where large unseen regions make single-scene inference brittle and unsafe. We propose Schrödinger's Navigator, a belief-aware framework that reasons at inference time over multiple trajectory-conditioned imagined 3D futures. Given candidate paths, a trajectory-conditioned 3D world model predicts hypothetical observations and maintains a superposition of plausible scene realizations rather than committing to one map. An adaptive occluder-aware sampler directs imagination to uncertainty-critical regions, while a Future-Aware Value Map (FAVM) aggregates imagined futures for robust, proactive action selection. Experiments in simulation and on a physical Go2 quadruped show that Schrödinger's Navigator outperforms strong ZSON baselines, improving hidden-target discovery and risk-aware waypoint selection in occlusion-heavy
navigation scenarios. These results highlight imagined 3D futures as a scalable and generalizable strategy for zero-shot
navigation in uncertain real-world environments.
Read the full abstract ▾
All authors Yu He, Da Huang, Zhenyang Liu, Zixiao Gu, Qiang Sun, Guangnan Ye, Yanwei Fu, Yu-Gang Jiang
Computer Vision
VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
Brigitta Malagurski Törtei, Yasser Dahou, Ngoc Dung Huynh et al. (+5 more)
Vision-
Language Models (VLMs) have achieved remarkable progress across tasks such as visual question answering and image captioning. Yet, the extent to which these models perform visual
reasoning as opposed to relying on linguistic priors remains unclear. To address this, we introduce VisRes Bench, a
benchmark designed to study visual
reasoning in naturalistic settings without contextual
language supervision. Analyzing model behavior across three levels of complexity, we uncover clear limitations in perceptual and relational visual
reasoning capacities. VisRes isolates distinct
reasoning abilities across its levels. Level 1 probes perceptual completion and global image matching under perturbations such as blur, texture changes, occlusion, and rotation; Level 2 tests rule-based inference over a single attribute (e.g., color, count, orientation); and Level 3 targets compositional
reasoning that requires integrating multiple visual attributes. Across more than 19,000 controlled task images, we find that state-of-the-art VLMs perform near random under subtle perceptual perturbations, revealing limited abstraction beyond pattern recognition. We conclude by discussing how VisRes provides a unified framework for advancing abstract visual
reasoning in
multimodal research.
Read the full abstract ▾
All authors Brigitta Malagurski Törtei, Yasser Dahou, Ngoc Dung Huynh, Wamiq Reyaz Para, Phúc H. Lê Khac, Ankit Singh, Sofian Chaybouti, Sanath Narayan
Computer Vision
UltraShape 1.0: High-Fidelity 3D Shape Generation via Scalable Geometric Refinement
Tanghui Jia, Dongyu Yan, Dehao Hao et al. (+11 more)
In this report, we introduce UltraShape 1.0, a scalable 3D
diffusion framework for high-fidelity 3D geometry generation. The proposed approach adopts a two-stage generation pipeline: a coarse global structure is first synthesized and then refined to produce detailed, high-quality geometry. To support reliable 3D generation, we develop a comprehensive data processing pipeline that includes a novel watertight processing method and high-quality data filtering. This pipeline improves the
geometric quality of publicly available 3D
datasets by removing low-quality samples, filling holes, and thickening thin structures, while preserving fine-grained
geometric details. To enable fine-grained geometry refinement, we decouple spatial localization from
geometric detail synthesis in the
diffusion process. We achieve this by performing voxel-based refinement at fixed spatial locations, where voxel queries derived from coarse geometry provide explicit positional anchors encoded via RoPE, allowing the
diffusion model to focus on synthesizing local
geometric details within a reduced, structured solution space. Our model is trained exclusively on publicly available 3D
datasets, achieving strong
geometric quality despite limited training resources. Extensive evaluations demonstrate that UltraShape 1.0 performs competitively with existing open-source methods in both data processing quality and geometry generation. All code and trained models will be released to support future research.
Read the full abstract ▾
All authors Tanghui Jia, Dongyu Yan, Dehao Hao, Yang Li, Kaiyi Zhang, Xianyi He, Lanjiong Li, Yuhan Wang, Jinnan Chen, Lutao Jiang, Qishen Yin, Long Quan, Ying-Cong Chen, Li Yuan
Computer Vision
Towards Arbitrary Motion Completing via Hierarchical Continuous Representation
Chenghao Xu, Guangtao Lyu, Qi Liu et al. (+3 more)
Physical motions are inherently continuous, and higher camera frame rates typically contribute to improved smoothness and temporal coherence. For the first time, we explore continuous representations of human motion sequences, featuring the ability to interpolate, inbetween, and even extrapolate any input motion sequences at arbitrary frame rates. To achieve this, we propose a novel parametric activation-induced hierarchical implicit representation framework, referred to as NAME, based on Implicit Neural Representations (INRs). Our method introduces a hierarchical temporal encoding mechanism that extracts features from motion sequences at multiple temporal scales, enabling effective capture of intricate temporal patterns. Additionally, we integrate a custom parametric activation function, powered by
Fourier transformations, into the MLP-based decoder to enhance the expressiveness of the continuous representation. This parametric formulation significantly augments the model's ability to represent complex motion behaviors with high accuracy. Extensive evaluations across several
benchmark datasets demonstrate the effectiveness and robustness of our proposed approach.
Read the full abstract ▾
All authors Chenghao Xu, Guangtao Lyu, Qi Liu, Jiexi Yan, Muli Yang, Cheng Deng
Computer Vision · Cross-listed from physics.med-ph
Equivariant Multiscale Learned Invertible Reconstruction for Cone Beam CT: From Simulated to Real Data
Nikita Moriakov, Efstratios Gavves, Jonathan H. Mason et al. (+3 more)
Cone Beam CT (CBCT) is an important imaging modality nowadays, however lower image quality of CBCT compared to more conventional Computed
Tomography (CT) remains a limiting factor in CBCT applications. Deep learning reconstruction methods are a promising alternative to classical analytical and iterative reconstruction methods, but applying such methods to CBCT is often difficult due to the lack of ground truth data,
memory limitations and the need for fast inference at clinically-relevant resolutions. In this work we propose LIRE++, an end-to-end rotationally-equivariant multiscale learned invertible primal-dual scheme for fast and
memory-efficient CBCT reconstruction.
Memory optimizations and multiscale reconstruction allow for fast training and inference, while rotational equivariance improves parameter efficiency. LIRE++ was trained on simulated projection data from a fast quasi-Monte Carlo CBCT projection simulator that we developed as well. Evaluated on synthetic data, LIRE++ gave an average improvement of 1 dB in Peak Signal-to-Noise Ratio over alternative deep learning baselines. On real clinical data, LIRE++ improved the average Mean Absolute Error between the reconstruction and the corresponding planning CT by 10 Hounsfield Units with respect to current proprietary state-of-the-art hybrid deep-learning/iterative method.
Read the full abstract ▾
All authors Nikita Moriakov, Efstratios Gavves, Jonathan H. Mason, Carmen Seller-Oria, Jonas Teuwen, Jan-Jakob Sonke