Computer Vision
Factorized Learning for Temporally Grounded Video-Language Models
Wenzheng Zeng, Difei Gao, Mike Zheng Shou et al. (+1 more)
Recent
video-
language models have shown great potential for
video understanding, but still struggle with accurate temporal grounding for event-level perception. We observe that two main factors in
video understanding (i.e., temporal grounding and textual response) form a logical hierarchy: accurate temporal evidence grounding lays the foundation for reliable textual response. However, existing works typically handle these two tasks in a coupled manner without a clear logical structure, leading to sub-optimal objectives. We address this from a factorized learning perspective. We first propose D$^2$VLM, a framework that decouples the learning of these two tasks while also emphasizing their inherent dependency. We adopt a "grounding then answering with evidence referencing" paradigm and introduce evidence
tokens for evidence grounding, which emphasize event-level visual semantic capture beyond the focus on timestamp representation in existing works. To further facilitate the learning of these two tasks, we introduce a novel factorized preference optimization (FPO)
algorithm. Unlike standard preference optimization, FPO explicitly incorporates probabilistic temporal grounding modeling into the optimization objective, enabling preference learning for both temporal grounding and textual response. We also construct a synthetic
dataset to address the lack of suitable
datasets for factorized preference learning with explicit temporal grounding. Experiments on various tasks demonstrate the clear advantage of our approach. Our source code is available at https://github.com/nusnlp/d2vlm.
Read the full abstract ▾
All authors Wenzheng Zeng, Difei Gao, Mike Zheng Shou, Hwee Tou Ng
Computer Vision
RainFusion2.0: Temporal-Spatial Awareness and Hardware-Efficient Block-wise Sparse Attention
Aiyue Chen, Yaofu Liu, Junjian Huang et al. (+6 more)
In
video and image generation tasks,
Diffusion Transformer (DiT) models incur extremely high computational costs due to
attention mechanisms, which limits their practical applications. Furthermore, with hardware advancements, a wide range of devices besides graphics processing unit (GPU), such as application-specific integrated
circuit (ASIC), have been increasingly adopted for model inference. Sparse
attention, which leverages the inherent sparsity of
attention by skipping computations for insignificant
tokens, is an effective approach to mitigate computational costs. However, existing sparse
attention methods have two critical limitations: the overhead of sparse pattern prediction and the lack of hardware generality, as most of these methods are designed for GPU. To address these challenges, this study proposes RainFusion2.0, which aims to develop an online adaptive, hardware-efficient, and low-overhead sparse
attention mechanism to accelerate both
video and image generative models, with robust performance across diverse hardware platforms. Key technical insights include: (1) leveraging block-wise mean values as representative
tokens for sparse mask prediction; (2) implementing spatiotemporal-aware
token permutation; and (3) introducing a first-frame sink mechanism specifically designed for
video generation scenarios. Experimental results demonstrate that RainFusion2.0 can achieve 80% sparsity while achieving an end-to-end speedup of 1.5~1.8x without compromising
video quality. Moreover, RainFusion2.0 demonstrates effectiveness across various generative models and validates its generalization across diverse hardware platforms.
Read the full abstract ▾
All authors Aiyue Chen, Yaofu Liu, Junjian Huang, Guang Lian, Yiwu Yao, Wangli Lan, Jing Lin, Zhixin Ma, Tingting Zhou
Computer Vision
Balanced Hierarchical Contrastive Learning with Decoupled Queries for Fine-grained Object Detection in Remote Sensing Images
Jingzhou Chen, Dexin Chen, Fengchao Xiong et al. (+2 more)
Fine-grained remote sensing
datasets often use hierarchical label structures to differentiate objects in a coarse-to-fine manner, with each object annotated across multiple levels. However,
embedding this semantic hierarchy into the representation learning space to improve fine-grained detection performance remains challenging. Previous studies have applied supervised contrastive learning at different hierarchical levels to group objects under the same parent class while distinguishing sibling subcategories. Nevertheless, they overlook two critical issues: (1) imbalanced data distribution across the label hierarchy causes high-frequency classes to dominate the learning process, and (2) learning semantic relationships among categories interferes with class-agnostic localization. To address these issues, we propose a balanced hierarchical contrastive loss combined with a decoupled learning strategy within the detection
transformer (DETR) framework. The proposed loss introduces learnable class prototypes and equilibrates gradients contributed by different classes at each hierarchical level, ensuring that each hierarchical class contributes equally to the loss computation in every mini-batch. The decoupled strategy separates DETR's object queries into classification and localization sets, enabling task-specific feature extraction and optimization. Experiments on three fine-grained
datasets with hierarchical annotations demonstrate that our method outperforms state-of-the-art approaches.
Read the full abstract ▾
All authors Jingzhou Chen, Dexin Chen, Fengchao Xiong, Yuntao Qian, Liang Xiao
Computer Vision
Pathology Context Recalibration Network for Ocular Disease Recognition
Zunjie Xiao, Xiaoqing Zhang, Risa Higashita et al. (+1 more)
Pathology context and expert experience play significant roles in clinical ocular disease diagnosis. Although deep neural networks (DNNs) have good ocular disease recognition results, they often ignore exploring the clinical pathology context and expert experience priors to improve ocular disease recognition performance and decision-making interpretability. To this end, we first develop a novel Pathology Recalibration Module (PRM) to leverage the potential of pathology context prior via the combination of the well-designed pixel-wise context compression operator and pathology distribution concentration operator; then this paper applies a novel expert prior Guidance Adapter (EPGA) to further highlight significant pixel-wise representation regions by fully mining the expert experience prior. By incorporating PRM and EPGA into the modern DNN, the PCRNet is constructed for automated ocular disease recognition. Additionally, we introduce an Integrated Loss (IL) to boost the ocular disease recognition performance of PCRNet by considering the effects of sample-wise loss distributions and training label frequencies. The extensive experiments on three ocular disease
datasets demonstrate the superiority of PCRNet with IL over state-of-the-art
attention-based networks and advanced loss methods. Further visualization analysis explains the inherent behavior of PRM and EPGA that affects the decision-making process of DNNs.
Read the full abstract ▾
All authors Zunjie Xiao, Xiaoqing Zhang, Risa Higashita, Jiang Liu
Computer Vision
Neighbor-aware Instance Refining with Noisy Labels for Cross-Modal Retrieval
Yizhi Liu, Ruitao Pu, Shilin Xu et al. (+3 more)
In recent years, Cross-Modal
Retrieval (CMR) has made significant progress in the field of multi-modal analysis. However, since it is time-consuming and labor-intensive to collect large-scale and well-annotated data, the annotation of multi-modal data inevitably contains some noise. This will degrade the
retrieval performance of the model. To tackle the problem, numerous robust CMR methods have been developed, including robust learning paradigms, label calibration strategies, and instance selection mechanisms. Unfortunately, they often fail to simultaneously satisfy model performance ceilings, calibration reliability, and data utilization rate. To overcome the limitations, we propose a novel robust cross-modal learning framework, namely Neighbor-aware Instance Refining with Noisy Labels (NIRNL). Specifically, we first propose Cross-modal Margin Preserving (CMP) to adjust the relative distance between positive and negative pairs, thereby enhancing the discrimination between sample pairs. Then, we propose Neighbor-aware Instance Refining (NIR) to identify pure subset, hard subset, and noisy subset through cross-modal neighborhood consensus. Afterward, we construct different tailored optimization strategies for this fine-grained partitioning, thereby maximizing the utilization of all available data while mitigating error propagation. Extensive experiments on three
benchmark datasets demonstrate that NIRNL achieves state-of-the-art performance, exhibiting remarkable robustness, especially under high noise rates.
Read the full abstract ▾
All authors Yizhi Liu, Ruitao Pu, Shilin Xu, Yingke Chen, Quan-Hui Liu, Yuan Sun
Computer Vision
Reinforced Diffusion: Learning to Push the Limits of Anisotropic Diffusion for Image Denoising
Xinran Qin, Yuhui Quan, Ruotao Xu et al. (+1 more)
Image denoising is an important problem in low-level
vision and serves as a critical module for many image recovery tasks. Anisotropic
diffusion is a wide family of image denoising approaches with promising performance. However, traditional anisotropic
diffusion approaches use explicit
diffusion operators which are not well adapted to complex image structures. As a result, their performance is limited compared to recent learning-based approaches. In this work, we describe a trainable anisotropic
diffusion framework based on reinforcement learning. By modeling the denoising process as a series of naive
diffusion actions with order learned by deep Q-learning, we propose an effective
diffusion-based image denoiser. The
diffusion actions selected by deep Q-learning at different iterations indeed composite a
stochastic anisotropic
diffusion process with strong adaptivity to different image structures, which enjoys improvement over the traditional ones. The proposed denoiser is applied to removing three types of often-seen noise. The experiments show that it outperforms existing
diffusion-based methods and competes with the representative deep CNN-based methods.
Read the full abstract ▾
All authors Xinran Qin, Yuhui Quan, Ruotao Xu, Hui Ji
Computer Vision
PipeFlow: Pipelined Processing and Motion-Aware Frame Selection for Long-Form Video Editing
Mustafa Munir, Md Mostafijur Rahman, Kartikeya Bhardwaj et al. (+2 more)
Long-form
video editing poses unique challenges due to the exponential increase in the computational cost from joint editing and Denoising
Diffusion Implicit Models (DDIM) inversion across extended sequences. To address these limitations, we propose PipeFlow, a scalable, pipelined
video editing method that introduces three key innovations: First, based on a motion analysis using Structural Similarity Index Measure (SSIM) and Optical Flow, we identify and propose to skip editing of frames with low motion. Second, we propose a pipelined task scheduling
algorithm that splits a
video into multiple segments and performs DDIM inversion and joint editing in parallel based on available GPU
memory. Lastly, we leverage a neural network-based interpolation technique to smooth out the border frames between segments and interpolate the previously skipped frames. Our method uniquely scales to longer
videos by dividing them into smaller segments, allowing PipeFlow's editing time to increase linearly with
video length. In principle, this enables editing of infinitely long
videos without the growing per-frame computational overhead encountered by other methods. PipeFlow achieves up to a 9.6X speedup compared to TokenFlow and a 31.7X speedup over
Diffusion Motion Transfer (DMT).
Read the full abstract ▾
All authors Mustafa Munir, Md Mostafijur Rahman, Kartikeya Bhardwaj, Paul Whatmough, Radu Marculescu
Computer Vision
RSAgent: Learning to Reason and Act for Text-Guided Segmentation via Multi-Turn Tool Invocations
Xingqi He, Yujie Zhang, Shuyong Gao et al. (+6 more)
Text-guided object
segmentation requires both cross-modal
reasoning and pixel grounding abilities. Most recent methods treat text-guided
segmentation as one-shot grounding, where the model predicts pixel
prompts in a single forward pass to drive an external segmentor, which limits verification, refocusing and refinement when initial localization is wrong. To address this limitation, we propose RSAgent, an agentic
Multimodal Large
Language Model (MLLM) which interleaves
reasoning and action for
segmentation via multi-turn tool invocations. RSAgent queries a
segmentation toolbox, observes visual feedback, and revises its spatial hypothesis using historical observations to re-localize targets and iteratively refine masks. We further build a data pipeline to synthesize multi-turn
reasoning segmentation trajectories, and train RSAgent with a two-stage framework: cold-start supervised fine-tuning followed by agentic reinforcement learning with fine-grained, task-specific rewards. Extensive experiments show that RSAgent achieves a zero-shot performance of 66.5% gIoU on ReasonSeg test, improving over Seg-Zero-7B by 9%, and reaches 81.5% cIoU on RefCOCOg, demonstrating state-of-the-art performance on both in-domain and out-of-domain
benchmarks.
Read the full abstract ▾
All authors Xingqi He, Yujie Zhang, Shuyong Gao, Wenjie Li, Lingyi Hong, Mingxi Chen, Kaixun Jiang, Jiyuan Fu, Wenqiang Zhang
Computer Vision
FUSE-RSVLM: Feature Fusion Vision-Language Model for Remote Sensing
Yunkai Dang, Donghao Wang, Jiacheng Yang et al. (+7 more)
Large
vision-
language models (VLMs) exhibit strong performance across various tasks. However, these VLMs encounter significant challenges when applied to the remote sensing domain due to the inherent differences between remote sensing images and natural images. Existing remote sensing VLMs often fail to extract fine-grained visual features and suffer from visual forgetting during deep
language processing. To address this, we introduce MF-RSVLM, a Multi-Feature Fusion Remote Sensing
Vision–
Language Model that effectively extracts and fuses visual features for RS understanding. MF-RSVLM learns multi-scale visual representations and combines global context with local details, improving the capture of small and complex structures in RS scenes. A recurrent visual feature injection scheme ensures the
language model remains grounded in visual evidence and reduces visual forgetting during generation. Extensive experiments on diverse RS
benchmarks show that MF-RSVLM achieves state-of-the-art or highly competitive performance across remote sensing classification, image captioning, and VQA tasks. Our code is publicly available at https://github.com/Yunkaidang/RSVLM.
Read the full abstract ▾
All authors Yunkai Dang, Donghao Wang, Jiacheng Yang, Yifan Jiang, Meiyi Zhu, Yuekun Yang, Cong Wang, Qi Fan, Wenbin Li, Yang Gao
Computer Vision · Cross-listed from quant-ph
One-Shot Structured Pruning of Quantum Neural Networks via $q$-Group Engineering and Quantum Geometric Metrics
Haijian Shao, Wei Liu, Xing Deng et al. (+1 more)
Quantum neural networks (QNNs) suffer from severe gate-level redundancy, which hinders their deployment on noisy intermediate-scale
quantum (NISQ) devices. In this work, we propose q-iPrune, a one-shot structured
pruning framework grounded in the algebraic structure of $q$-deformed groups and task-conditioned
quantum geometry. Unlike prior heuristic or gradient-based
pruning methods, q-iPrune formulates redundancy directly at the gate level. Each gate is compared within an algebraically consistent subgroup using a task-conditioned $q$-overlap distance, which measures functional similarity through state overlaps on a task-relevant ensemble. A gate is removed only when its replacement by a subgroup representative provably induces a bounded deviation on all task observables. We establish three rigorous theoretical guarantees. First, we prove completeness of redundancy
pruning: no gate that violates the prescribed similarity threshold is removed. Second, we show that the pruned
circuit is functionally equivalent up to an explicit, task-conditioned error bound, with a closed-form dependence on the redundancy tolerance and the number of replaced gates. Third, we prove that the
pruning procedure is computationally feasible, requiring only
polynomial-time comparisons and avoiding exponential enumeration over the
Hilbert space. To adapt
pruning decisions to hardware imperfections, we introduce a noise-calibrated deformation parameter $\lambda$ that modulates the $q$-geometry and redundancy tolerance. Experiments on standard
quantum machine learning
benchmarks demonstrate that q-iPrune achieves substantial gate reduction while maintaining bounded task performance degradation, consistent with our theoretical guarantees.
Read the full abstract ▾
All authors Haijian Shao, Wei Liu, Xing Deng, Yingtao Jiang
Computer Vision
Structure-Guided Allocation of 2D Gaussians for Image Representation and Compression
Huanxiong Liang, Yunuo Chen, Yicheng Pan et al. (+4 more)
Recent advances in 2D Gaussian Splatting (2DGS) have demonstrated its potential as a compact image representation with millisecond-level decoding. However, existing 2DGS-based pipelines allocate representation capacity and parameter precision largely oblivious to image structure, limiting their rate-distortion (RD) efficiency at low bitrates. To address this, we propose a structure-guided allocation principle for 2DGS, which explicitly couples image structure with both representation capacity and quantization precision, while preserving native decoding speed. First, we introduce a structure-guided initialization that assigns 2D Gaussians according to spatial structural priors inherent in natural images, yielding a localized and semantically meaningful distribution. Second, during quantization-aware fine-tuning, we propose adaptive bitwidth quantization of covariance parameters, which grants higher precision to small-scale Gaussians in complex regions and lower precision elsewhere, enabling RD-aware optimization, thereby reducing redundancy without degrading edge quality. Third, we impose a geometry-consistent regularization that aligns Gaussian orientations with local gradient directions to better preserve structural details. Extensive experiments demonstrate that our approach substantially improves both the representational power and the RD performance of 2DGS while maintaining over 1000 FPS decoding. Compared with the baseline GSImage, we reduce BD-rate by 43.44% on Kodak and 29.91% on DIV2K.
Read the full abstract ▾
All authors Huanxiong Liang, Yunuo Chen, Yicheng Pan, Sixian Wang, Jincheng Dai, Guo Lu, Wenjun Zhang
Computer Vision
FitControler: Toward Fit-Aware Virtual Try-On
Lu Yang, Yicheng Liu, Letian Zhou et al. (+3 more)
Realistic virtual try-on (VTON) concerns not only faithful rendering of garment details but also coordination of the style. Prior art typically pursues the former, but neglects a key factor that shapes the holistic style – garment fit. Garment fit delineates how a garment aligns with the body of a wearer and is a fundamental element in fashion design. In this work, we introduce fit-aware VTON and present FitControler, a learnable plug-in that can seamlessly integrate into modern VTON models to enable customized fit control. To achieve this, we highlight two challenges: i) how to delineate layouts of different fits and ii) how to render the garment that matches the layout. FitControler first features a fit-aware layout generator to redraw the body-garment layout conditioned on a set of delicately processed garment-agnostic representations, and a multi-scale fit injector is then used to deliver layout cues to enable layout-driven VTON. In particular, we build a fit-aware VTON
dataset termed Fit4Men, including 13,000 body-garment pairs of different fits, covering both tops and bottoms, and featuring varying camera distances and body poses. Two fit consistency metrics are also introduced to assess the fitness of generations. Extensive experiments show that FitControler can work with various VTON models and achieve accurate fit control. Code and data will be released.
Read the full abstract ▾
All authors Lu Yang, Yicheng Liu, Letian Zhou, Yanan Li, Xiang Bai, Hao Lu
Computer Vision
On Exact Editing of Flow-Based Diffusion Models
Zixiang Li, Yue Song, Jianing Peng et al. (+7 more)
Recent methods in flow-based
diffusion editing have enabled direct transformations between source and target image distribution without explicit inversion. However, the latent trajectories in these methods often exhibit accumulated velocity errors, leading to semantic inconsistency and loss of structural fidelity. We propose Conditioned Velocity Correction (CVC), a principled framework that reformulates flow-based editing as a distribution transformation problem driven by a known source prior. CVC rethinks the role of velocity in inter-distribution transformation by introducing a dual-perspective velocity conversion mechanism. This mechanism explicitly decomposes the latent
evolution into two components: a structure-preserving branch that remains consistent with the source trajectory, and a semantically-guided branch that drives a controlled deviation toward the target distribution. The conditional velocity field exhibits an absolute velocity error relative to the true underlying distribution trajectory, which inherently introduces potential instability and trajectory drift in the latent space. To address this quantifiable deviation and maintain fidelity to the true flow, we apply a posterior-consistent update to the resulting conditional velocity field. This update is derived from Empirical Bayes Inference and Tweedie correction, which ensures a mathematically grounded error compensation over time. Our method yields stable and interpretable latent dynamics, achieving faithful reconstruction alongside smooth local semantic conversion. Comprehensive experiments demonstrate that CVC consistently achieves superior fidelity, better semantic
alignment, and more reliable editing behavior across diverse tasks.
Read the full abstract ▾
All authors Zixiang Li, Yue Song, Jianing Peng, Ting Liu, Jun Huang, Xiaochao Qu, Luoqi Liu, Wei Wang, Yao Zhao, Yunchao Wei
Computer Vision
Bridging the Perception-Cognition Gap:Re-engineering SAM2 with Hilbert-Mamba for Robust VLM-based Medical Diagnosis
Hao Wu, Hui Li and Yiyun Su
Recent studies suggest that Visual
Language Models (VLMs) hold great potential for tasks such as automated medical diagnosis. However, processing complex three-dimensional (3D)
multimodal medical images poses significant challenges - specifically, the effective integration of complementary information and the occasional oversight of subtle yet critical pathological features. To address these issues, we present a novel two-stage fusion framework termed
Hilbert-VLM. This framework leverages the HilbertMed-SAM module for precise lesion
segmentation, with the generated
multimodal enhanced
prompts then guiding the VLM toward accurate disease classification. Our key innovation lies in the systematic redesign of the Segment Anything Model 2 (SAM2) architecture: we incorporate
Hilbert space-filling curves into the scanning mechanism of the Mamba State Space Model (SSM) to maximize the preservation of spatial locality in 3D data, a property critical for medical image analysis. We also introduce a novel
Hilbert-Mamba Cross-
Attention (HMCA) mechanism and a scale-aware decoder to capture fine-grained details. Meanwhile, the
prompt enhancement module unifies
segmentation masks and their corresponding textual attributes into an information-dense
prompt to support VLM inference. Extensive experiments were conducted to validate the effectiveness of the
Hilbert-VLM model. On the BraTS2021
segmentation benchmark, it achieves a Dice score of 82.35 percent, with a diagnostic classification accuracy (ACC) of 78.85 percent. These results demonstrate that the proposed model offers substantial potential to improve the accuracy and reliability of medical VLM-based analysis.
Read the full abstract ▾
Computer Vision
Improved 3D Gaussian Splatting of Unknown Spacecraft Structure Using Space Environment Illumination Knowledge
Tae Ha Park and Simone D'Amico
This work presents a novel pipeline to recover the 3D structure of an unknown target spacecraft from a sequence of images captured during Rendezvous and Proximity Operations (RPO) in space. The target's geometry and appearance are represented as a 3D Gaussian Splatting (3DGS) model. However, learning 3DGS requires static scenes, an assumption in contrast to dynamic lighting conditions encountered in spaceborne imagery. The trained 3DGS model can also be used for camera pose estimation through photometric optimization. Therefore, in addition to recovering a geometrically accurate 3DGS model, the photometric accuracy of the rendered images is imperative to downstream pose estimation tasks during the RPO process. This work proposes to incorporate the prior knowledge of the Sun's position, estimated and maintained by the servicer spacecraft, into the training pipeline for improved photometric quality of 3DGS rasterization. Experimental studies demonstrate the effectiveness of the proposed solution, as 3DGS models trained on a sequence of images learn to adapt to rapidly changing illumination conditions in space and reflect global shadowing and self-occlusion.
Read the full abstract ▾
Computer Vision
MSSSeg: Learning Multi-Scale Structural Complexity for Self-Supervised Segmentation
Haotang Li, Zhenyu Qi, Hao Qin et al. (+4 more)
Self-supervised semantic
segmentation methods often suffer from structural errors, including merging distinct objects or fragmenting coherent regions, because they rely primarily on low-level appearance cues such as color and texture. These cues lack structural discriminability: they carry no information about the structural organization of a region, making it difficult to distinguish boundaries between similar-looking objects or maintain coherence within internally varying regions. Recent approaches attempt to address this by incorporating depth priors, yet remain limited by not explicitly modeling structural complexity that persists even when appearance cues are ambiguous. To bridge this gap, we present MSSSeg, a framework that explicitly learns multi-scale structural complexity from both semantic and depth domains, via three coupled components: (1) a Differentiable Box-Counting (DBC) module that captures and aligns multi-scale structural complexity features with semantic features; (2) a Learnable Structural Augmentation (StructAug) that corrupts pixel-intensity patterns, forcing the network to rely on structural complexity features from DBC; and (3) a Persistent Homology Loss (PHLoss) that directly supervises the structural complexity of predicted
segmentations. Extensive experiments demonstrate that MSSSeg achieves new state-of-the-art performance on COCO-Stuff-27, Cityscapes, and Potsdam without excessive computational overhead, validating that explicit structural complexity learning is crucial for self-supervised
segmentation.
Read the full abstract ▾
All authors Haotang Li, Zhenyu Qi, Hao Qin, Huanrui Yang, Kebin Peng, Qing Guo, Sen He
Computer Vision
GCA-ResUNet: Medical Image Segmentation Using Grouped Coordinate Attention
Jun Ding and Shang Gao
Accurate
segmentation of heterogeneous anatomical structures is pivotal for computer-aided diagnosis and subsequent clinical decision-making. Although U-Net based convolutional neural networks have achieved remarkable progress, their intrinsic locality and largely homogeneous
attention formulations often limit the modeling of long-range contextual dependencies, especially in multi-organ scenarios and low-contrast regions.
Transformer-based architectures mitigate this issue by leveraging global self-
attention, but they usually require higher computational resources and larger training data, which may impede deployment in resource-constrained clinical environments.In this paper, we propose GCA-ResUNet, an efficient medical image
segmentation framework equipped with a lightweight and plug-and-play Grouped Coordinate
Attention (GCA) module. The proposed GCA decouples channel-wise context modeling into multiple groups to explicitly account for semantic heterogeneity across channels, and integrates direction-aware coordinate encoding to capture structured spatial dependencies along horizontal and vertical axes. This design enhances global representation capability while preserving the efficiency advantages of CNN backbones. Extensive experiments on two widely used
benchmarks, Synapse and ACDC, demonstrate that GCA-ResUNet achieves Dice scores of 86.11% and 92.64%, respectively, outperforming a range of representative CNN and
Transformer-based methods, including Swin-UNet and TransUNet. In particular, GCA-ResUNet yields consistent improvements in delineating small anatomical structures with complex boundaries. These results indicate that the proposed approach provides a favorable trade-off between
segmentation accuracy and computational efficiency, offering a practical and scalable solution for clinical deployment.
Read the full abstract ▾
Computer Vision
Anomaly detection in satellite imagery through temporal inpainting
Bertrand Rouet-Leduc and Claudia Hulbert
Detecting surface changes from satellite imagery is critical for rapid disaster response and environmental monitoring, yet remains challenging due to the complex interplay between atmospheric noise, seasonal variations, and sensor artifacts. Here we show that deep learning can leverage the temporal redundancy of satellite time series to detect anomalies at unprecedented sensitivity, by learning to predict what the surface should look like in the absence of change. We train an inpainting model built upon the SATLAS foundation model to reconstruct the last frame of a Sentinel-2 time series from preceding acquisitions, using globally distributed training data spanning diverse
climate zones and land cover types. When applied to regions affected by sudden surface changes, the discrepancy between prediction and observation reveals anomalies that traditional change detection methods miss. We validate our approach on earthquake-triggered surface ruptures from the 2023 Turkey-Syria earthquake sequence, demonstrating detection of a rift feature in Tepehan with higher sensitivity and specificity than temporal median or Reed-Xiaoli anomaly detectors. Our method reaches detection thresholds approximately three times lower than baseline approaches, providing a path towards automated, global-scale monitoring of surface changes from freely available multi-spectral satellite data.
Read the full abstract ▾
Computer Vision
DriveExplorer: Images-Only Decoupled 4D Reconstruction with Progressive Restoration for Driving View Extrapolation
Yuang Jia, Jinlong Wang, Jiayi Zhao et al. (+3 more)
This paper presents an effective solution for view extrapolation in autonomous driving scenarios. Recent approaches focus on generating shifted novel view images from given viewpoints using
diffusion models. However, these methods heavily rely on priors such as
LiDAR point clouds, 3D bounding boxes, and lane annotations, which demand expensive sensors or labor-intensive labeling, limiting applicability in real-world deployment. In this work, with only images and optional camera poses, we first estimate a global static point cloud and per-frame dynamic point clouds, fusing them into a unified representation. We then employ a deformable 4D Gaussian framework to reconstruct the scene. The initially trained 4D Gaussian model renders degraded and pseudo-images to train a
video diffusion model. Subsequently, progressively shifted Gaussian renderings are iteratively refined by the
diffusion model,and the enhanced results are incorporated back as training data for 4DGS. This process continues until extrapolation reaches the target viewpoints. Compared with baselines, our method produces higher-quality images at novel extrapolated viewpoints.
Read the full abstract ▾
All authors Yuang Jia, Jinlong Wang, Jiayi Zhao, Chunlam Li, Shunzhou Wang, Wei Gao
Computer Vision
Changzhen Li, Yuecong Min, Jie Zhang et al. (+3 more)
The rapid
evolution of Text-to-
Video (T2V)
diffusion models has driven remarkable advancements in generating high-quality, temporally coherent
videos from natural
language descriptions. Despite these achievements, their vulnerability to
adversarial attacks remains largely unexplored. In this paper, we introduce T2VAttack, a comprehensive study of
adversarial attacks on T2V
diffusion models from both semantic and temporal perspectives. Considering the inherently dynamic nature of
video data, we propose two distinct attack objectives: a semantic objective to evaluate
video-text
alignment and a temporal objective to assess the temporal dynamics. To achieve an effective and efficient attack process, we propose two
adversarial attack methods: (i) T2VAttack-S, which identifies semantically or temporally critical words in
prompts and replaces them with synonyms via greedy search, and (ii) T2VAttack-I, which iteratively inserts optimized words with minimal perturbation to the
prompt. By combining these objectives and strategies, we conduct a comprehensive evaluation on the
adversarial robustness of several state-of-the-art T2V models, including ModelScope, CogVideoX, Open-Sora, and HunyuanVideo. Our experiments reveal that even minor
prompt modifications, such as the substitution or insertion of a single word, can cause substantial degradation in semantic fidelity and temporal dynamics, highlighting critical vulnerabilities in current T2V
diffusion models.
Read the full abstract ▾
All authors Changzhen Li, Yuecong Min, Jie Zhang, Zheng Yuan, Shiguang Shan, Xilin Chen
Computer Vision
U-Net-Like Spiking Neural Networks for Single Image Dehazing
Huibin Li, Haoran Liu, Mingzhe Liu et al. (+3 more)
Image dehazing is a critical challenge in computer
vision, essential for enhancing image clarity in hazy conditions. Traditional methods often rely on atmospheric scattering models, while recent deep learning techniques, specifically Convolutional Neural Networks (CNNs) and
Transformers, have improved performance by effectively analyzing image features. However, CNNs struggle with long-range dependencies, and
Transformers demand significant computational resources. To address these limitations, we propose DehazeSNN, an innovative architecture that integrates a U-Net-like design with Spiking Neural Networks (SNNs). DehazeSNN captures multi-scale image features while efficiently managing local and long-range dependencies. The introduction of the Orthogonal Leaky-Integrate-and-Fire Block (OLIFBlock) enhances cross-channel communication, resulting in superior dehazing performance with reduced computational burden. Our extensive experiments show that DehazeSNN is highly competitive to state-of-the-art methods on
benchmark datasets, delivering high-quality haze-free images with a smaller model size and less multiply-accumulate operations. The proposed dehazing method is publicly available at https://github.com/HaoranLiu507/DehazeSNN.
Read the full abstract ▾
All authors Huibin Li, Haoran Liu, Mingzhe Liu, Yulong Xiao, Peng Li, Guibin Zan
Computer Vision
Kinematic-Based Assessment of Surgical Actions in Microanastomosis
Yan Meng, Daniel Donoho, Marcelle Altshuler et al. (+1 more)
Proficiency in microanastomosis is a critical
surgical skill in neurosurgery, where the ability to precisely manipulate fine instruments is crucial to successful outcomes. These procedures require sustained
attention, coordinated hand movements, and highly refined motor skills, underscoring the need for objective and systematic methods to evaluate and enhance microsurgical training. Conventional assessment approaches typically rely on expert raters supervising the procedures or reviewing
surgical videos, which is an inherently subjective process prone to inter-rater variability, inconsistency, and significant time investment. These limitations highlight the necessity for automated and scalable solutions. To address this challenge, we introduce a novel AI-driven framework for automated action
segmentation and performance assessment in microanastomosis procedures, designed to operate efficiently on edge computing platforms. The proposed system comprises three main components: (1) an object tip tracking and localization module based on YOLO and DeepSORT; (2) an action
segmentation module leveraging self-similarity
matrix for action boundary detection and unsupervised clustering; and (3) a supervised classification module designed to evaluate
surgical gesture proficiency. Experimental validation on a
dataset of 58 expert-rated microanastomosis
videos demonstrates the effectiveness of our approach, achieving a frame-level action
segmentation accuracy of 92.4% and an overall skill classification accuracy of 85.5% in replicating expert evaluations. These findings demonstrate the potential of the proposed method to provide objective, real-time feedback in microsurgical education, thereby enabling more standardized, data-driven training protocols and advancing competency assessment in high-stakes
surgical environments.
Read the full abstract ▾
All authors Yan Meng, Daniel Donoho, Marcelle Altshuler, Omar Arnaout
Computer Vision
Learnable Query Aggregation with KV Routing for Cross-view Geo-localisation
Hualin Ye, Bingxi Liu, Jixiang Du et al. (+3 more)
Cross-view geo-localisation (CVGL) aims to estimate the geographic location of a query image by matching it with images from a large-scale database. However, the significant view-point discrepancies present considerable challenges for effective feature aggregation and
alignment. To address these challenges, we propose a novel CVGL system that incorporates three key improvements. Firstly, we leverage the DINOv2 backbone with a convolution adapter fine-tuning to enhance model adaptability to cross-view variations. Secondly, we propose a multi-scale channel reallocation module to strengthen the diversity and stability of spatial representations. Finally, we propose an improved aggregation module that integrates a Mixture-of-Experts (MoE) routing into the feature aggregation process. Specifically, the module dynamically selects expert subspaces for the keys and values in a cross-
attention framework, enabling adaptive processing of heterogeneous input domains. Extensive experiments on the University-1652 and SUES-200
datasets demonstrate that our method achieves competitive performance with fewer trained parameters.
Read the full abstract ▾
All authors Hualin Ye, Bingxi Liu, Jixiang Du, Yu Qin, Ziyi Chen, Hong Zhang
Computer Vision
MGML: A Plug-and-Play Meta-Guided Multi-Modal Learning Framework for Incomplete Multimodal Brain Tumor Segmentation
Yulong Zou, Bo Liu, Cun-Jing Zheng et al. (+6 more)
Leveraging
multimodal information from
Magnetic Resonance Imaging (MRI) plays a vital role in lesion
segmentation, especially for
brain tumors. However, in clinical practice,
multimodal MRI data are often incomplete, making it challenging to fully utilize the available information. Therefore, maximizing the utilization of this incomplete
multimodal information presents a crucial research challenge. We present a novel meta-guided multi-modal learning (MGML) framework that comprises two components: meta-parameterized adaptive modality fusion and consistency regularization module. The meta-parameterized adaptive modality fusion (Meta-AMF) enables the model to effectively integrate information from multiple modalities under varying input conditions. By generating adaptive soft-label supervision signals based on the available modalities, Meta-AMF explicitly promotes more coherent
multimodal fusion. In addition, the consistency regularization module enhances
segmentation performance and implicitly reinforces the robustness and generalization of the overall framework. Notably, our approach does not alter the original model architecture and can be conveniently integrated into the training pipeline for end-to-end model optimization. We conducted extensive experiments on the public BraTS2020 and BraTS2023
datasets. Compared to multiple state-of-the-art methods from previous years, our method achieved superior performance. On BraTS2020, for the average Dice scores across fifteen missing modality combinations, building upon the baseline, our method obtained scores of 87.55, 79.36, and 62.67 for the whole tumor (WT), the tumor core (TC), and the enhancing tumor (ET), respectively. We have made our source code publicly available at https://github.com/worldlikerr/MGML.
Read the full abstract ▾
All authors Yulong Zou, Bo Liu, Cun-Jing Zheng, Yuan-ming Geng, Siyue Li, Qiankun Zuo, Shuihua Wang, Yudong Zhang, Jin Hong
Computer Vision
Learning to learn skill assessment for fetal ultrasound scanning
Yipei Wang, Qianye Yang, Lior Drukker et al. (+3 more)
Traditionally, ultrasound skill assessment has relied on expert supervision and feedback, a process known for its subjectivity and time-intensive nature. Previous works on quantitative and automated skill assessment have predominantly employed supervised learning methods, often limiting the analysis to predetermined or assumed factors considered influential in determining skill levels. In this work, we propose a novel bi-level optimisation framework that assesses fetal ultrasound skills by how well a task is performed on the acquired fetal ultrasound images, without using manually predefined skill ratings. The framework consists of a clinical task predictor and a skill predictor, which are optimised jointly by refining the two networks simultaneously. We validate the proposed method on real-world clinical ultrasound
videos of scanning the fetal head. The results demonstrate the feasibility of predicting ultrasound skills by the proposed framework, which quantifies optimised task performance as a skill indicator.
Read the full abstract ▾
All authors Yipei Wang, Qianye Yang, Lior Drukker, Aris T. Papageorghiou, Yipeng Hu, J. Alison Noble
Computer Vision · Cross-listed from eess.SP
A multimodal Transformer for InSAR-based ground deformation forecasting with cross-site generalization across Europe
Wendong Yao, Binhua Huang and Soumyabrata Dev
Near-real-time regional-scale monitoring of ground deformation is increasingly required to support urban planning, critical infrastructure management, and natural hazard mitigation. While Interferometric Synthetic Aperture Radar (InSAR) and continental-scale services such as the European Ground Motion Service (EGMS) provide dense observations of past motion, predicting the next observation remains challenging due to the superposition of long-term trends, seasonal cycles, and occasional abrupt discontinuities (e.g., co-seismic steps), together with strong spatial heterogeneity. In this study we propose a
multimodal patch-based
Transformer for single-step, fixed-interval next-epoch nowcasting of displacement maps from EGMS time series (resampled to a 64x64 grid over 100 km x 100 km tiles). The model ingests recent displacement snapshots together with (i) static kinematic indicators (mean velocity, acceleration, seasonal amplitude) computed in a leakage-safe manner from the training window only, and (ii) harmonic day-of-year encodings. On the eastern Ireland tile (E32N34), the STGCN is strongest in the displacement-only setting, whereas the
multimodal Transformer clearly outperforms CNN-LSTM, CNN-LSTM+Attn, and
multimodal STGCN when all models receive the same
multimodal inputs, achieving RMSE = 0.90 mm and $R^2$ = 0.97 on the test set with the best threshold accuracies.
Read the full abstract ▾