Computer Vision
DiffVC: A Non-autoregressive Framework Based on Diffusion Model for Video Captioning
Junbo Wang, Liangyu Fu, Yuke Li et al. (+4 more)
Current
video captioning methods usually use an encoder-decoder structure to generate text autoregressively. However, autoregressive methods have inherent limitations such as slow generation speed and large cumulative error. Furthermore, the few non-autoregressive counterparts suffer from deficiencies in generation quality due to the lack of sufficient
multimodal interaction modeling. Therefore, we propose a non-autoregressive framework based on
Diffusion model for
Video Captioning (DiffVC) to address these issues. Its parallel decoding can effectively solve the problems of generation speed and cumulative error. At the same time, our proposed discriminative conditional
Diffusion Model can generate higher-quality textual descriptions. Specifically, we first encode the
video into a visual representation. During training, Gaussian noise is added to the textual representation of the ground-truth caption. Then, a new textual representation is generated via the discriminative denoiser with the visual representation as a conditional constraint. Finally, we input the new textual representation into a non-autoregressive
language model to generate captions. During inference, we directly sample noise from the Gaussian distribution for generation. Experiments on MSVD, MSR-VTT, and VATEX show that our method can outperform previous non-autoregressive methods and achieve comparable performance to autoregressive methods, e.g., it achieved a maximum improvement of 9.9 on the CIDEr and improvement of 2.6 on the B@4, while having faster generation speed. The source code will be available soon.
Read the full abstract ▾
All authors Junbo Wang, Liangyu Fu, Yuke Li, Yining Zhu, Ya Jing, Xuecheng Wu, Jiangbin Zheng
Computer Vision
AdaSpark: Adaptive Sparsity for Efficient Long-Video Understanding
Handong Li, Zikang Liu, Longteng Guo et al. (+10 more)
Processing long-form
videos with
Video Large
Language Models (
Video-LLMs) is computationally prohibitive. Current efficiency methods often compromise fine-grained perception through irreversible information disposal or inhibit long-range temporal modeling via rigid, predefined sparse patterns. This paper introduces AdaSpark, an adaptive sparsity framework designed to address these limitations. AdaSpark first partitions
video inputs into 3D spatio-temporal cubes. It then employs two co-designed, context-aware components: (1) Adaptive Cube-Selective
Attention (AdaS-Attn), which adaptively selects a subset of relevant
video cubes to attend for each query
token, and (2) Adaptive
Token-Selective FFN (AdaS-FFN), which selectively processes only the most salient
tokens within each cube. An
entropy-based (Top-p) selection mechanism adaptively allocates computational resources based on input complexity. Experiments demonstrate that AdaSpark significantly reduces computational load by up to 57% FLOPs while maintaining comparable performance to dense models and preserving fine-grained, long-range dependencies, as validated on challenging hour-scale
video benchmarks.
Read the full abstract ▾
All authors Handong Li, Zikang Liu, Longteng Guo, Tongtian Yue, Yepeng Tang, Xinxin Zhu, Chuanyang Zheng, Ziming Wang, Zhibin Wang, Jun Song, Cheng Yu, Bo Zheng, Jing Liu
Computer Vision
DinoRADE: Full Spectral Radar-Camera Fusion with Vision Foundation Model Features for Multi-class Object Detection in Adverse Weather
Christof Leitgeb, Thomas Puchleitner, Max Peter Ronecker et al. (+1 more)
Reliable and weather-robust perception systems are essential for safe autonomous driving and typically employ multi-modal sensor configurations to achieve comprehensive environmental awareness. While recent automotive FMCW Radar-based approaches achieved remarkable performance on detection tasks in adverse weather conditions, they exhibited limitations in resolving fine-grained spatial details particularly critical for detecting smaller and vulnerable road users (VRUs). Furthermore, existing research has not adequately addressed VRU detection in adverse weather
datasets such as K-Radar. We present DinoRADE, a Radar-centered detection pipeline that processes dense Radar tensors and aggregates
vision features around transformed reference points in the camera perspective via deformable cross-
attention.
Vision features are provided by a DINOv3
Vision Foundation Model. We present a comprehensive performance evaluation on the K-Radar
dataset in all weather conditions and are among the first to report detection performance individually for five object classes. Additionally, we compare our method with existing single-class detection approaches and outperform recent Radar-camera approaches by 12.1%. The code is available under https://github.com/chr-is-tof/RADE-Net.
Read the full abstract ▾
All authors Christof Leitgeb, Thomas Puchleitner, Max Peter Ronecker, Daniel Watzenig
Computer Vision
Tensor-Augmented Convolutional Neural Networks: Enhancing Expressivity with Generic Tensor Kernels
Chia-Wei Hsing and Wei-Lin Tu
Convolutional Neural Networks (CNNs) excel at extracting local features hierarchically, but their performance in capturing complex correlations hinges heavily on deep architectures, which are usually computationally demanding and difficult to interpret. To address these issues, we propose a physically-guided shallow model: tensor-augmented CNN (TACNN), which replaces conventional convolution kernels with generic tensors to enhance representational capacity. This choice is motivated by the fact that an order-$N$ tensor naturally encodes an arbitrary
quantum superposition state in the
Hilbert space of dimension $d^N$, where $d$ is the local physical dimension, thus offering substantially richer expressivity. Furthermore, in our design the convolution output of each layer becomes a multilinear form capable of capturing high-order feature correlations, thereby equipping a shallow multilayer architecture with an expressive power competitive to that of deep CNNs. On the Fashion-MNIST
benchmark, TACNN demonstrates clear advantages over conventional CNNs, achieving remarkable accuracies with only a few layers. In particular, a TACNN with only two convolution layers attains a test accuracy of 93.7$\%$, surpassing or matching considerably deeper models such as VGG-16 (93.5$\%$) and GoogLeNet (93.7$\%$). These findings highlight TACNN as a promising framework that strengthens model expressivity while preserving architectural simplicity, paving the way towards more interpretable and efficient deep learning models.
Read the full abstract ▾
Computer Vision
AtlasOCR: Building the First Open-Source Darija OCR Model with Vision Language Models
Imane Momayiz, Soufiane Ait Elaouad, Abdeljalil Elmajjodi et al. (+1 more)
Darija, the Moroccan Arabic dialect, is rich in visual content yet lacks specialized Optical Character Recognition (OCR) tools. This paper introduces AtlasOCR, the first open-source Darija OCR model built by fine-tuning a 3B parameter
Vision Language Model (VLM). We detail our comprehensive approach, from curating a unique Darija-specific
dataset leveraging both synthetic generation with our OCRSmith library and carefully sourced real-world data, to implementing efficient fine-tuning strategies. We utilize QLoRA and Unsloth for parameter-efficient training of Qwen2.5-VL 3B and present comprehensive ablation studies optimizing key hyperparameters. Our evaluation on the newly curated AtlasOCRBench and the established KITAB-Bench demonstrates state-of-the-art performance, challenging larger models and highlighting AtlasOCR's robustness and generalization capabilities for both Darija and standard Arabic OCR tasks.
Read the full abstract ▾
All authors Imane Momayiz, Soufiane Ait Elaouad, Abdeljalil Elmajjodi, Haitame Bouanane
Computer Vision
Emanuele Balloni, Emanuele Frontoni, Chiara Matti et al. (+3 more)
Decoding visual information from electroencephalography (
EEG) has recently achieved promising results, primarily focusing on reconstructing two-dimensional (2D) images from
brain activity. However, the reconstruction of three-dimensional (3D) representations remains largely unexplored. This limits the
geometric understanding and reduces the applicability of neural decoding in different contexts. To address this gap, we propose
Brain3D, a
multimodal architecture for
EEG-to-3D reconstruction based on
EEG-to-image decoding. It progressively transforms neural representations into the 3D domain using geometry-aware generative
reasoning. Our pipeline first produces visually grounded images from
EEG signals, then employs a
multimodal large
language model to extract structured 3D-aware descriptions, which guide a
diffusion-based generation stage whose outputs are finally converted into coherent 3D meshes via a single-image-to-3D model. By decomposing the problem into structured stages, the proposed approach avoids direct
EEG-to-3D mappings and enables scalable
brain-driven 3D generation. We conduct a comprehensive evaluation comparing the reconstructed 3D outputs against the original visual stimuli, assessing both semantic
alignment and
geometric fidelity. Experimental results demonstrate strong performance of the proposed architecture, achieving up to 85.4% 10-way Top-1
EEG decoding accuracy and 0.648 CLIPScore, supporting the feasibility of
multimodal EEG-driven 3D reconstruction.
Read the full abstract ▾
All authors Emanuele Balloni, Emanuele Frontoni, Chiara Matti, Marina Paolanti, Roberto Pierdicca, Emiliano Santarnecchi
Computer Vision
EEG2Vision: A Multimodal EEG-Based Framework for 2D Visual Reconstruction in Cognitive Neuroscience
Emanuele Balloni, Emanuele Frontoni, Chiara Matti et al. (+3 more)
Reconstructing visual stimuli from non-invasive electroencephalography (
EEG) remains challenging due to its low spatial resolution and high noise, particularly under realistic low-density electrode configurations. To address this, we present
EEG2
Vision, a modular, end-to-end
EEG-to-image framework that systematically evaluates reconstruction performance across different
EEG resolutions (128, 64, 32, and 24 channels) and enhances visual quality through a
prompt-guided post-reconstruction boosting mechanism. Starting from
EEG-conditioned
diffusion reconstruction, the boosting stage uses a
multimodal large
language model to extract semantic descriptions and leverages image-to-image
diffusion to refine geometry and perceptual coherence while preserving
EEG-grounded structure. Our experiments show that semantic decoding accuracy degrades significantly with channel reduction (e.g., 50-way Top-1 Acc from 89% to 38%), while reconstruction quality slight decreases (e.g., FID from 76.77 to 80.51). The proposed boosting consistently improves perceptual metrics across all configurations, achieving up to 9.71% IS gains in low-channel settings. A user study confirms the clear perceptual preference for boosted reconstructions. The proposed approach significantly boosts the feasibility of real-time
brain-2-image applications using low-resolution
EEG devices, potentially unlocking this type of applications outside laboratory settings.
Read the full abstract ▾
All authors Emanuele Balloni, Emanuele Frontoni, Chiara Matti, Marina Paolanti, Roberto Pierdicca, Emiliano Santarnecchi
Computer Vision
ABMAMBA: Multimodal Large Language Model with Aligned Hierarchical Bidirectional Scan for Efficient Video Captioning
Daichi Yashima, Shuhei Kurita, Yusuke Oda et al. (+3 more)
In this study, we focus on
video captioning by fully open
multimodal large
language models (MLLMs). The comprehension of visual sequences is challenging because of their intricate temporal dependencies and substantial sequence length. The core
attention mechanisms of existing
Transformer-based approaches scale quadratically with the sequence length, making them computationally prohibitive. To address these limitations, we propose Aligned Hierarchical Bidirectional Scan Mamba (ABMamba), a fully open MLLM with linear computational complexity that enables the scalable processing of
video sequences. ABMamba extends Deep State Space Models as its
language backbone, replacing the costly quadratic
attention mechanisms, and employs a novel Aligned Hierarchical Bidirectional Scan module that processes
videos across multiple temporal resolutions. On standard
video captioning
benchmarks such as VATEX and MSR-VTT, ABMamba demonstrates competitive performance compared to typical MLLMs while achieving approximately three times higher throughput.
Read the full abstract ▾
All authors Daichi Yashima, Shuhei Kurita, Yusuke Oda, Shuntaro Suzuki, Seitaro Otsuki, Komei Sugiura
Computer Vision
Guiding a Diffusion Model by Swapping Its Tokens
Weijia Zhang, Yuehao Liu, Shanyan Guan et al. (+4 more)
Classifier-Free Guidance (CFG) is a widely used inference-time technique to boost the image quality of
diffusion models. Yet, its reliance on text conditions prevents its use in unconditional generation. We propose a simple method to enable CFG-like guidance for both conditional and unconditional generation. The key idea is to generate a perturbed prediction via simple
token swap operations, and use the direction between it and the clean prediction to steer sampling towards higher-fidelity distributions. In practice, we swap pairs of most semantically dissimilar
token latents in either spatial or channel dimensions. Unlike existing methods that apply perturbation in a global or less constrained manner, our approach selectively exchanges and recomposes
token latents, allowing finer control over perturbation and its influence on generated samples. Experiments on MS-COCO 2014, MS-COCO 2017, and ImageNet
datasets demonstrate that the proposed Self-Swap Guidance (SSG), when applied to popular
diffusion models, outperforms previous condition-free methods in image fidelity and
prompt alignment under different set-ups. Its fine-grained perturbation granularity also improves robustness, reducing side-effects across a wider range of perturbation strengths. Overall, SSG extends CFG to a broader scope of applications including both conditional and unconditional generation, and can be readily inserted into any
diffusion model as a plug-in to gain immediate improvements.
Read the full abstract ▾
All authors Weijia Zhang, Yuehao Liu, Shanyan Guan, Wu Ran, Yanhao Ge, Wei Li, Chao Ma
Computer Vision
Adapting Foundation Models for Annotation-Efficient Adnexal Mass Segmentation in Cine Images
Francesca Fati, Alberto Rota, Adriana V. Gregory et al. (+9 more)
Adnexal mass evaluation via ultrasound is a challenging clinical task, often hindered by subjective interpretation and significant inter-observer variability. While automated
segmentation is a foundational step for quantitative risk assessment, traditional fully supervised convolutional architectures frequently require large amounts of pixel-level annotations and struggle with domain shifts common in medical imaging. In this work, we propose a label-efficient
segmentation framework that leverages the robust semantic priors of a pretrained DINOv3 foundational
vision transformer backbone. By integrating this backbone with a Dense Prediction
Transformer (DPT)-style decoder, our model hierarchically reassembles multi-scale features to combine global semantic representations with fine-grained spatial details. Evaluated on a clinical
dataset of 7,777 annotated frames from 112 patients, our method achieves state-of-the-art performance compared to established fully supervised baselines, including U-Net, U-Net++, DeepLabV3, and MAnet. Specifically, we obtain a Dice score of 0.945 and improved boundary adherence, reducing the 95th-percentile Hausdorff Distance by 11.4% relative to the strongest convolutional baseline. Furthermore, we conduct an extensive efficiency analysis demonstrating that our DINOv3-based approach retains significantly higher performance under data starvation regimes, maintaining strong results even when trained on only 25% of the data. These results suggest that leveraging large-scale self-supervised foundations provides a promising and data-efficient solution for medical image
segmentation in data-constrained clinical environments. Project Repository: https://github.com/FrancescaFati/MESA
Read the full abstract ▾
All authors Francesca Fati, Alberto Rota, Adriana V. Gregory, Anna Catozzo, Maria C. Giuliano, Mrinal Dhar, Luigi De Vitis, Annie T. Packard, Francesco Multinu, Elena De Momi, Carrie L. Langstraat, Timothy L. Kline
Computer Vision
3DrawAgent: Teaching LLM to Draw in 3D with Early Contrastive Experience
Hongcan Xiao, Xinyue Xiao, Yilin Wang et al. (+2 more)
Sketching in 3D space enables expressive
reasoning about shape, structure, and spatial relationships, yet generating 3D sketches through natural
language remains a major challenge. In this work, we introduce 3DrawAgent, a training-free,
language-driven framework for 3D sketch generation that leverages large
language models (LLMs) to sequentially draw 3D Bezier curves under
geometric feedback. Unlike prior 2D sketch
agents, our method introduces a relative experience optimization strategy that adapts the recently proposed Group Reward Policy Optimization (GRPO) paradigm. Instead of relying on explicit ground-truth supervision, we construct pairwise comparisons among generated sketches, with each pair consisting of a relatively better and a worse result based on CLIP-based perceptual rewards and LLM-based fine-grained qualitative assessment. These experiences are then used to iteratively refine the prior knowledge of 3D drawing, enabling black-box reinforcement of the model's 3D awareness. This design allows our model to self-improve its spatial understanding and drawing quality without parameter updates. Experiments show that 3DrawAgent can generate complex and coherent 3D Bezier sketches from diverse textual
prompts, exhibit emergent
geometric reasoning, and generalize to novel shapes, establishing a new paradigm for advancing the field of training-free 3D sketch intelligence.
Read the full abstract ▾
All authors Hongcan Xiao, Xinyue Xiao, Yilin Wang, Yue Zhang, Yonggang Qi
Computer Vision
LINE: LLM-based Iterative Neuron Explanations for Vision Models
Vladimir Zaigrajew, Michał Piechota, Gaspar Sekula et al. (+2 more)
Interpreting the concepts encoded by individual
neurons in deep neural networks is a crucial step towards understanding their complex decision-making processes and ensuring AI safety. Despite recent progress in
neuron labeling, existing methods often limit the search space to predefined concept vocabularies or produce overly specific descriptions that fail to capture higher-order, global concepts. We introduce LINE, a novel, training-free iterative approach tailored for open-vocabulary concept labeling in
vision models. Operating in a strictly gray-box setting, LINE leverages a large
language model and a text-to-image generator to iteratively propose and refine concepts in a closed loop, guided by activation history. We demonstrate that LINE achieves state-of-the-art performance across multiple model architectures, yielding AUC improvements of up to 0.15 on ImageNet and 0.05 on Places365, while discovering, on average, 25% of new concepts missed by predefined vocabularies. Beyond identifying the top concept, LINE provides a complete generation history, enabling polysemanticity evaluation and producing supporting visual explanations that rival gradient-dependent activation maximization methods. The LINE source code is available at https://github.com/CredibleAI/LINE
Read the full abstract ▾
All authors Vladimir Zaigrajew, Michał Piechota, Gaspar Sekula, Paweł Gelar, Przemysław Biecek
Computer Vision
Beyond Mamba: Enhancing State-space Models with Deformable Dilated Convolutions for Multi-scale Traffic Object Detection
Jun Li, Yingying Shi, Zhixuan Ruan et al. (+2 more)
In a real-world traffic scenario, varying-scale objects are usually distributed in a cluttered background, which poses great challenges to accurate detection. Although current Mamba-based methods can efficiently model long-range dependencies, they still struggle to capture small objects with abundant local details, which hinders joint modeling of local structures and global semantics. Moreover, state-space models exhibit limited hierarchical feature representation and weak cross-scale interaction due to flat sequential modeling and insufficient spatial inductive biases, leading to sub-optimal performance in complex scenes. To address these issues, we propose a Mamba with Deformable Dilated Convolutions Network (MDDCNet) for accurate traffic object detection in this study. In MDDCNet, a well-designed hybrid backbone with successive Multi-Scale Deformable Dilated Convolution (MSDDC) blocks and Mamba blocks enables hierarchical feature representation from local details to global semantics. Meanwhile, a Channel-Enhanced Feed-Forward Network (CE-FFN) is further devised to overcome the limited channel interaction capability of conventional feed-forward networks, whilst a Mamba-based
Attention-Aggregating Feature Pyramid Network (A^2FPN) is constructed to achieve enhanced multi-scale feature fusion and interaction. Extensive experimental results on public
benchmark and real-world
datasets demonstrate the superiority of our method over various advanced detectors. The code is available at https://github.com/Bettermea/MDDCNet.
Read the full abstract ▾
All authors Jun Li, Yingying Shi, Zhixuan Ruan, Nan Guo, Jianhua Xu
Computer Vision · Cross-listed from cs.CR
PrivFedTalk: Privacy-Aware Federated Diffusion with Identity-Stable Adapters for Personalized Talking-Head Generation
Soumya Mazumdar, Vineet Kumar Rakesh and Tapas Samanta
Talking-head generation has advanced rapidly with
diffusion-based generative models, but training usually depends on centralized face-
video and
speech datasets, raising major
privacy concerns. The problem is more acute for personalized talking-head generation, where identity-specific data are highly sensitive and often cannot be pooled across users or devices. PrivFedTalk is presented as a
privacy-aware
federated framework for personalized talking-head generation that combines conditional latent
diffusion with parameter-efficient identity adaptation. A shared
diffusion backbone is trained across clients, while each client learns lightweight LoRA identity adapters from local private audio-visual data, avoiding raw data sharing and reducing communication cost. To address heterogeneous client distributions, Identity-Stable
Federated Aggregation (ISFA) weights client updates using
privacy-safe scalar reliability signals computed from on-device identity consistency and temporal stability estimates. Temporal-Denoising Consistency (TDC) regularization is introduced to reduce inter-frame drift, flicker, and identity drift during
federated denoising. To limit update-side
privacy risk, secure aggregation and client-level differential
privacy are applied to adapter updates. The implementation supports both low-
memory GPU execution and multi-GPU client-parallel training on heterogeneous shared hardware. Comparative experiments on the present setup across multiple training and aggregation conditions with PrivFedTalk, FedAvg, and FedProx show stable
federated optimization and successful end-to-end training and evaluation under constrained resources. The results support the feasibility of
privacy-aware personalized talking-head training in
federated environments, while suggesting that stronger component-wise,
privacy-utility, and qualitative claims need further standardized evaluation.
Read the full abstract ▾
Computer Vision
Rotation Equivariant Convolutions in Deformable Registration of Brain MRI
Arghavan Rezvani, Kun Han, Anthony T. Wu et al. (+2 more)
Image registration is a fundamental task that aligns anatomical structures between images. While CNNs perform well, they lack rotation equivariance - a rotated input does not produce a correspondingly rotated output. This hinders performance by failing to exploit the rotational
symmetries inherent in anatomical structures, particularly in
brain MRI. In this work, we integrate rotation-equivariant convolutions into deformable
brain MRI registration networks. We evaluate this approach by replacing standard encoders with equivariant ones in three baseline architectures, testing on multiple public
brain MRI
datasets. Our experiments demonstrate that equivariant encoders have three key advantages: 1) They achieve higher registration accuracy while reducing network parameters, confirming the benefit of this anatomical inductive bias. 2) They outperform baselines on rotated input pairs, demonstrating robustness to orientation variations common in clinical practice. 3) They show improved performance with less training data, indicating greater sample efficiency. Our results demonstrate that incorporating
geometric priors is a critical step toward building more robust, accurate, and efficient registration models.
Read the full abstract ▾
All authors Arghavan Rezvani, Kun Han, Anthony T. Wu, Pooya Khosravi, Xiaohui Xie
Computer Vision · Cross-listed from cs.RO
Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous Vehicles
Jiawei Liu, Xun Gong, Fen Fang et al. (+6 more)
Most Human-Machine Interaction (HMI) research overlooks the maneuvering needs of passengers in autonomous driving (AD). Natural
language offers an intuitive interface, yet translating passenger open-ended instructions into control signals, without sacrificing interpretability and traceability, remains a challenge. This study proposes an instruction-realization framework that leverages a large
language model (LLM) to interpret instructions, generates executable scripts that schedule multiple model predictive control (MPC)-based motion planners based on real-time feedback, and converts planned trajectories into control signals. This scheduling-centric design decouples semantic
reasoning from vehicle control at different timescales, establishing a transparent, traceable decision-making chain from high-level instructions to low-level actions. Due to the absence of high-fidelity evaluation tools, this study introduces a
benchmark for open-ended instruction realization in a closed-loop setting. Comprehensive experiments reveal that the framework significantly improves task-completion rates over instruction-realization baselines, reduces LLM query costs, achieves safety and compliance on par with specialized AD approaches, and exhibits considerable tolerance to LLM inference latency. For more qualitative illustrations and a clearer understanding.
Read the full abstract ▾
All authors Jiawei Liu, Xun Gong, Fen Fang, Muli Yang, Bohao Qu, Yunfeng Hu, Hong Chen, Xulei Yang, Qing Guo
Computer Vision
Component-Adaptive and Lesion-Level Supervision for Improved Small Structure Segmentation in Brain MRI
Minh Sao Khue Luu, Evgeniy N. Pavlovskiy and Bair N. Tuchinov
Small lesions in
brain MRI are hard to segment because they occupy a tiny fraction of the volume and are dominated by background and larger lesions during voxel-wise optimization, so a model can reach a high Dice similarity coefficient (DSC) while missing many of them. We propose CATMIL, a training objective that adds two auxiliary terms to the standard nnU-Net Dice and cross-
entropy loss without changing the architecture. The Component-Adaptive Tversky (CAT) term weights lesion voxels by the inverse size of their connected component, so each lesion contributes nearly equally regardless of volume. The lesion-level Multiple Instance Learning (MIL) term treats each lesion as a bag of voxels and penalizes lesions with no detected voxel. For multiple sclerosis lesion
segmentation on MSLesSeg, CATMIL achieves the highest small-lesion recall (0.873 vs. 0.796 for Dice+CE; 95% CI of the difference +0.030 to +0.157, higher in all six test patients) and about 48% fewer missed lesions, with comparable DSC and HD95. The gain holds for lesions of at least 3 mm in diameter, the clinical reading size (recall 0.944 vs. 0.870). Standard losses produce no probability response to most small lesions they miss, so no threshold can recover them. The cost is more small false-positive components; a simple component-size filter removes most of them while keeping the sensitivity gain, and at matched lesion-wise precision CATMIL detects more small lesions with higher lesion-wise F1. An ablation attributes the detection gain to the MIL term. On a second
dataset, 3D-MR-MS, CATMIL with the same loss weights again improves small-lesion recall, at a larger false-positive cost and slightly lower DSC. Code: https://github.com/luumsk/SmallLesionMRI
Read the full abstract ▾
Computer Vision
Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding
Xuezhen Tu, Jingyu Wu, Fangyu Kang et al. (+4 more)
Spatio-Temporal
Video Grounding requires jointly localizing target objects across both temporal and spatial dimensions based on natural
language queries, posing fundamental challenges for existing
Multimodal Large
Language Models (MLLMs). We identify two core challenges: entangled spatio-temporal
alignment, arising from coupling two heterogeneous sub-tasks within the same autoregressive output space, and dual-domain visual
token redundancy, where target objects exhibit simultaneous temporal and spatial sparsity, rendering the overwhelming majority of visual
tokens irrelevant to the grounding query. To address these, we propose Bridge-STG, an end-to-end framework that decouples temporal and spatial localization while maintaining semantic coherence. While decoupling is the natural solution to this
entanglement, it risks creating a semantic gap between the temporal MLLM and the spatial decoder. Bridge-STG resolves this through two pivotal designs: the Spatio-Temporal Semantic Bridging (STSB) mechanism with Explicit Temporal
Alignment (ETA) distills the MLLM's temporal
reasoning context into enriched bridging queries as a robust semantic interface; and the Query-Guided Spatial Localization (QGSL) module leverages these queries to drive a purpose-built spatial decoder with multi-layer interactive queries and positive/negative frame sampling, jointly eliminating dual-domain visual
token redundancy. Extensive experiments across multiple
benchmarks demonstrate that Bridge-STG achieves state-of-the-art performance among MLLM-based methods. Bridge-STG improves average m\_vIoU from $26.4$ to $34.3$ on VidSTG and demonstrates strong cross-task transfer across various fine-grained
video understanding tasks under a unified multi-task training regime.
Read the full abstract ▾
All authors Xuezhen Tu, Jingyu Wu, Fangyu Kang, Qingpeng Nong, Kaijin Zhang, Chaoyue Niu, Fan Wu
Computer Vision
SearchAD: Large-Scale Rare Image Retrieval Dataset for Autonomous Driving
Felix Embacher, Jonas Uhrig, Marius Cordts et al. (+1 more)
Retrieving rare and safety-critical driving scenarios from large-scale
datasets is essential for building robust autonomous driving (AD) systems. As
dataset sizes continue to grow, the key challenge shifts from collecting more data to efficiently identifying the most relevant samples. We introduce SearchAD, a large-scale rare image
retrieval dataset for AD containing over 423k frames drawn from 11 established
datasets. SearchAD provides high-quality manual annotations of more than 513k bounding boxes covering 90 rare categories. It specifically targets the needle-in-a-haystack problem of locating extremely rare classes, with some appearing fewer than 50 times across the entire
dataset. Unlike existing
benchmarks, which focused on instance-level
retrieval, SearchAD emphasizes semantic image
retrieval with a well-defined data split, enabling text-to-image and image-to-image
retrieval, few-shot learning, and fine-tuning of multi-modal
retrieval models. Comprehensive evaluations show that text-based methods outperform image-based ones due to stronger inherent semantic grounding. While models directly aligning spatial visual features with
language achieve the best zero-shot results, and our fine-tuning baseline significantly improves performance, absolute
retrieval capabilities remain unsatisfactory. With a held-out test set on a public
benchmark server, SearchAD establishes the first large-scale
dataset for
retrieval-driven data curation and long-tail perception research in AD: https://iis-esslingen.github.io/searchad/
Read the full abstract ▾
All authors Felix Embacher, Jonas Uhrig, Marius Cordts, Markus Enzweiler
Computer Vision · Cross-listed from cs.AI
PASK: Toward Intent-Aware Proactive Agents with Long-Term Memory
Zhifei Xie, Zongzheng Hu, Fangda Ye et al. (+10 more)
Proactivity is a core expectation for AGI. Prior work remains largely confined to laboratory settings, leaving a clear gap in real-world proactive
agent: depth, complexity, ambiguity, precision and real-time constraints. We study this setting, where useful intervention requires inferring latent needs from ongoing context and grounding actions in evolving user
memory under latency and long-horizon constraints. We first propose DD-MM-PAS (Demand Detection,
Memory Modeling, Proactive
Agent System) as a general paradigm for streaming proactive AI
agent. We instantiate this paradigm in Pask, with streaming IntentFlow model for DD, a hybrid
memory (workspace, user, global) for long-term MM, PAS infra framework and introduce how these components form a closed loop. We also introduce LatentNeeds-Bench, a real-world
benchmark built from user-consented data and refined through thousands of rounds of human editing. Experiments show that IntentFlow matches leading Gemini3-Flash models under latency constraints, while identifying deeper user intent.
Read the full abstract ▾
All authors Zhifei Xie, Zongzheng Hu, Fangda Ye, Xin Zhang, Haobo Chai, Zihang Liu, Pengcheng Wu, Guibin Zhang, Yue Liao, Xiaobin Hu, Deheng Ye, Chunyan Miao, Shuicheng Yan
Computer Vision
Few-Shot Incremental 3D Object Detection in Dynamic Indoor Environments
Yun Zhu, Jianjun Qian, Jian Yang et al. (+2 more)
Incremental 3D object perception is a critical step toward
embodied intelligence in dynamic indoor environments. However, existing incremental 3D detection methods rely on extensive annotations of novel classes for satisfactory performance. To address this limitation, we propose FI3Det, a Few-shot Incremental 3D Detection framework that enables efficient 3D perception with only a few novel samples by leveraging
vision-
language models (VLMs) to learn knowledge of unseen categories. FI3Det introduces a VLM-guided unknown object learning module in the base stage to enhance perception of unseen categories. Specifically, it employs VLMs to mine unknown objects and extract comprehensive representations, including 2D semantic features and class-agnostic 3D bounding boxes. To mitigate noise in these representations, a weighting mechanism is further designed to re-weight the contributions of point- and box-level features based on their spatial locations and feature consistency within each box. Moreover, FI3Det proposes a gated
multimodal prototype imprinting module, where category prototypes are constructed from aligned 2D semantic and 3D
geometric features to compute classification scores, which are then fused via a
multimodal gating mechanism for novel object detection. As the first framework for few-shot incremental 3D object detection, we establish both batch and sequential evaluation settings on two
datasets, ScanNet V2 and SUN RGB-D, where FI3Det achieves strong and consistent improvements over baseline methods. Code is available at https://github.com/zyrant/FI3Det.
Read the full abstract ▾
All authors Yun Zhu, Jianjun Qian, Jian Yang, Jin Xie, Na Zhao
Computer Vision
SAT: Selective Aggregation Transformer for Image Super-Resolution
Dinh Phu Tran, Thao Do, Saad Wazir et al. (+3 more)
Transformer-based approaches have revolutionized image super-resolution by modeling long-range dependencies. However, the quadratic computational complexity of vanilla self-
attention mechanisms poses significant challenges, often leading to compromises between efficiency and global context exploitation. Recent window-based
attention methods mitigate this by localizing computations, but they often yield restricted receptive fields. To mitigate these limitations, we propose Selective Aggregation
Transformer (SAT). This novel
transformer efficiently captures long-range dependencies, leading to an enlarged model receptive field by selectively aggregating key-value
matrices (reducing the number of
tokens by 97%) via our Density-driven
Token Aggregation
algorithm while maintaining the full resolution of the query
matrix. This design significantly reduces computational costs, resulting in lower complexity and enabling scalable global interactions without compromising reconstruction fidelity. SAT identifies and represents each cluster with a single aggregation
token, utilizing density and isolation metrics to ensure that critical high-frequency details are preserved. Experimental results demonstrate that SAT outperforms the state-of-the-art method PFT by up to 0.22dB, while the total number of FLOPs can be reduced by up to 27%.
Read the full abstract ▾
All authors Dinh Phu Tran, Thao Do, Saad Wazir, Seongah Kim, Seon Kwon Kim, Daeyoung Kim
Computer Vision
MotionScape: A Motion-Stratified UAV Video Benchmark for World Modeling and Future Video Generation
Zile Guo, Zhan Chen, Xiaoxuan Liu et al. (+5 more)
Unmanned aerial vehicles (
UAVs) are increasingly crucial for low-altitude autonomy and complex environment understanding. World models enable
UAVs to anticipate how future states may evolve under potential actions, providing predictive support for autonomous decision-making. For
video world models, future
video generation serves as a means of simulating future visual states, providing a direct basis for evaluating their predictive capability. However, existing
UAV video resources typically focus on specific control signals or simulated environments. Standardized
benchmarks for multi-condition
UAV-view future
video generation are still limited. To bridge this gap, we introduce MotionScape, a real-world
UAV-view
benchmark comprising 228 high-resolution
video clips totaling 62,700 frames, with semantic annotations of weather and illumination conditions, scene environment, and camera-viewpoint motion. The clips are stratified into low-, medium-, and high-motion strata using optical-flow-based motion intensity. MotionScape supports Text2World, Image2World, and
Video2World evaluation under a unified future-generation protocol. Technical validation with representative baseline models assesses performance across conditioning settings and motion strata, providing a unified platform for evaluating the future visual simulation capabilities of
UAV world models.
Read the full abstract ▾
All authors Zile Guo, Zhan Chen, Xiaoxuan Liu, Enze Zhu, Kan Wei, Yongkang Zou, Sijia Ma, Lei Wang
Computer Vision
SceneScribe-1M: A Large-Scale Video Dataset with Comprehensive Geometric and Semantic Annotations
Yunnan Wang, Kecheng Zheng, Jianyuan Wang et al. (+8 more)
The convergence of 3D
geometric perception and
video synthesis has created an unprecedented demand for large-scale
video data that is rich in both semantic and spatio-temporal information. While existing
datasets have advanced either 3D understanding or
video generation, a significant gap remains in providing a unified resource that supports both domains at scale. To bridge this chasm, we introduce SceneScribe-1M, a new large-scale, multi-modal
video dataset. It comprises one million in-the-wild
videos, each meticulously annotated with detailed textual descriptions, precise camera parameters, dense depth maps, and consistent 3D point tracks. We demonstrate the versatility and value of SceneScribe-1M by establishing
benchmarks across a wide array of downstream tasks, including
monocular depth estimation, scene reconstruction, and dynamic point tracking, as well as generative tasks such as text-to-
video synthesis, with or without camera control. By open-sourcing SceneScribe-1M, we aim to provide a comprehensive
benchmark and a catalyst for research, fostering the development of models that can both perceive the dynamic 3D world and generate controllable, realistic
video content.
Read the full abstract ▾
All authors Yunnan Wang, Kecheng Zheng, Jianyuan Wang, Minghao Chen, David Novotny, Christian Rupprecht, Yinghao Xu, Xing Zhu, Wenjun Zeng, Xin Jin, Yujun Shen
Computer Vision
DP-DeGauss: Dynamic Probabilistic Gaussian Decomposition for Egocentric 4D Scene Reconstruction
Tingxi Chen, Zhengxue Cheng, Houqiang Zhong et al. (+3 more)
Egocentric video is crucial for next-generation 4D scene reconstruction, with applications in AR/VR and
embodied AI. However, reconstructing dynamic first-person scenes is challenging due to complex ego-motion, occlusions, and hand-object interactions. Existing decomposition methods are ill-suited, assuming fixed viewpoints or merging dynamics into a single foreground. To address these limitations, we introduce DP-DeGauss, a dynamic probabilistic Gaussian decomposition framework for
egocentric 4D reconstruction. Our method initializes a unified 3D Gaussian set from COLMAP priors, augments each with a learnable category probability, and dynamically routes them into specialized deformation branches for background, hands, or object modeling. We employ category-specific masks for better disentanglement and introduce brightness and motion-flow control to improve static rendering and dynamic reconstruction. Extensive experiments show that DP-DeGauss outperforms baselines by +1.70dB in PSNR on average with SSIM and LPIPS gains. More importantly, our framework achieves the first and state-of-the-art disentanglement of background, hand, and object components, enabling explicit, fine-grained separation, paving the way for more intuitive ego scene understanding and editing.
Read the full abstract ▾
All authors Tingxi Chen, Zhengxue Cheng, Houqiang Zhong, Su Wang, Rong Xie, Li Song
Computer Vision
Object-Centric Stereo Ranging for Autonomous Driving: From Dense Disparity to Census-Based Template Matching
Qihao Huang
Accurate depth estimation is critical for autonomous driving perception systems, particularly for long range vehicle detection on highways. Traditional dense stereo matching methods such as Block Matching (BM) and Semi Global Matching (SGM) produce per pixel disparity maps but suffer from high computational cost, sensitivity to radiometric differences between stereo cameras, and poor accuracy at long range where disparity values are small. In this report, we present a comprehensive stereo ranging system that integrates three complementary depth estimation approaches: dense BM/SGM disparity, object centric Census based template matching, and
monocular geometric priors, within a unified detection ranging tracking pipeline. Our key contribution is a novel object centric Census based template matching
algorithm that performs GPU accelerated sparse stereo matching directly within detected bounding boxes, employing a far close divide and conquer strategy, forward backward verification, occlusion aware sampling, and robust multi block aggregation. We further describe an online calibration refinement framework that combines auto rectification offset search, radar stereo voting based disparity correction, and object level radar stereo association for continuous extrinsic drift compensation. The complete system achieves real time performance through asynchronous GPU pipeline design and delivers robust ranging across diverse driving conditions including nighttime, rain, and varying illumination.
Read the full abstract ▾
Computer Vision
Lighting-grounded Video Generation with Renderer-based Agent Reasoning
Ziqi Cai, Taoyu Yang, Zheng Chang et al. (+4 more)
Diffusion models have achieved remarkable progress in
video generation, but their controllability remains a major limitation. Key scene factors such as layout, lighting, and camera trajectory are often entangled or only weakly modeled, restricting their applicability in domains like filmmaking and virtual production where explicit scene control is essential. We present LiVER, a
diffusion-based framework for scene-controllable
video generation. To achieve this, we introduce a novel framework that conditions
video synthesis on explicit 3D scene properties, supported by a new large-scale
dataset with dense annotations of object layout, lighting, and camera parameters. Our method disentangles these properties by rendering control signals from a unified 3D representation. We propose a lightweight conditioning module and a progressive training strategy to integrate these signals into a foundational
video diffusion model, ensuring stable convergence and high fidelity. Our framework enables a wide range of applications, including image-to-
video and
video-to-
video synthesis where the underlying 3D scene is fully editable. To further enhance usability, we develop a scene
agent that automatically translates high-level user instructions into the required 3D control signals. Experiments show that LiVER achieves state-of-the-art photorealism and temporal consistency while enabling precise, disentangled control over scene factors, setting a new standard for controllable
video generation.
Read the full abstract ▾
All authors Ziqi Cai, Taoyu Yang, Zheng Chang, Si Li, Han Jiang, Shuchen Weng, Boxin Shi
Computer Vision
DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editing
Gyanendra Das and Sai Satyam Jena
Model editing aims to update knowledge to add new concepts and change relevant information without retraining. Lifelong editing is a challenging task, prone to disrupting previously learned concepts, especially for
Vision Language Models (VLMs), because sequential edits can lead to degraded
reasoning and cross modal misalignment. Existing VLM knowledge editing methods based on gated adapters, activation edits, and parameter merging techniques address catastrophic forgetting seen in full fine tuning; however, they still operate in the shared representation space of the VLM, where concepts are entangled, so edits interfere with other non relevant concepts. We hypothesize that this instability persists because current methods algorithmically control edits via optimization rather than structurally separating knowledge. We introduce Dynamic Subspace Concept
Alignment (DSCA) which by design mitigates this limitation by decomposing the representation space into a set of orthogonal semantic subspaces and proposing edits only in those transformed spaces. These subspaces are obtained through incremental clustering and PCA on joint
vision language representations. This process structurally isolates concepts, enabling precise, non interfering edits by turning isolation from a soft training objective into an architectural property. The
surgical edits are guided by a multi term loss function for maintaining task fidelity, edit locality, and cross modal
alignment. With the base model frozen, our method achieves 98 percent single edit success, remains over 95 percent after 1000 sequential edits, lowers
hallucination by 3 to 5 percent, and achieves the best backward transfer (BWT) scores on continual instruction tuning
benchmarks. Extensive experiments demonstrate DSCA state of the art stability and knowledge retention capability in continual lifelong editing across various
datasets and
benchmarks.
Read the full abstract ▾
Computer Vision
TOOLCAD: Exploring Tool-Using Large Language Models in Text-to-CAD Generation with Reinforcement Learning
Yifei Gong, Xing Wu, Wenda Liu et al. (+1 more)
Computer-Aided Design (CAD) is an expert-level task that relies on long-horizon
reasoning and coherent modeling actions. Large
Language Models (LLMs) have shown remarkable advancements in enabling
language agents to tackle real-world tasks. Notably, there has been no investigation into how tool-using LLMs optimally interact with CAD engines, hindering the emergence of LLM-based agentic text-to-CAD modeling systems. We propose ToolCAD, a novel agentic CAD framework deploying LLMs as tool-using
agents for text-to-CAD generation. Furthermore, we introduce an interactive CAD modeling gym to rollout
reasoning and tool-augmented interaction trajectories with the CAD engine, incorporating hybrid feedback and human supervision. Meanwhile, an end-to-end post-training strategy is presented to enable the LLM
agent to elicit refined CAD Modeling Chain of Thought (CAD-CoT) and evolve into proficient CAD tool-using
agents via online curriculum reinforcement learning. Our findings demonstrate ToolCAD fills the gap in adopting and training open-source LLMs for CAD tool-using
agents, enabling them to perform comparably to proprietary models, paving the way for more accessible and robust autonomous text-to-CAD modeling systems.
Read the full abstract ▾
All authors Yifei Gong, Xing Wu, Wenda Liu, Kang Tu
Computer Vision
ImVideoEdit: Image-learning Video Editing via 2D Spatial Difference Attention Blocks
Jiayang Xu, Fan Zhuo, Majun Zhang et al. (+6 more)
Current
video editing models often rely on expensive paired
video data, which limits their practical scalability. In essence, most
video editing tasks can be formulated as a decoupled spatiotemporal process, where the temporal dynamics of the pretrained model are preserved while spatial content is selectively and precisely modified. Based on this insight, we propose ImVideoEdit, an efficient framework that learns
video editing capabilities entirely from image pairs. By freezing the pre-trained 3D
attention modules and treating images as single-frame
videos, we decouple the 2D spatial learning process to help preserve the original temporal dynamics. The core of our approach is a Predict-Update Spatial Difference
Attention module that progressively extracts and injects spatial differences. Rather than relying on rigid external masks, we incorporate a Text-Guided Dynamic Semantic Gating mechanism for adaptive and implicit text-driven modifications. Despite training on only 13K image pairs for 5 epochs with exceptionally low computational overhead, ImVideoEdit achieves editing fidelity and temporal consistency comparable to larger models trained on extensive
video datasets.
Read the full abstract ▾
All authors Jiayang Xu, Fan Zhuo, Majun Zhang, Changhao Pan, Zehan Wang, Siyu Chen, Xiaoda Yang, Tao Jin, Zhou Zhao