Computer Vision
Minh Khoa Le, Kien Do, Duc Thanh Nguyen et al. (+1 more)
High-fidelity
video generation remains challenging for
diffusion models due to the difficulty of modeling complex spatio-temporal dynamics efficiently. Recent
video diffusion methods typically represent a
video as a sequence of spatio-temporal
tokens which can be modeled using
Diffusion Transformers (DiTs). However, this approach faces a trade-off between the strong but expensive Full 3D
Attention and the efficient but temporally limited Local Factorized
Attention. To resolve this trade-off, we propose
Matrix Attention, a frame-level temporal
attention mechanism that processes an entire frame as a
matrix and generates query, key, and value
matrices via
matrix-native operations. By attending across frames rather than
tokens,
Matrix Attention effectively preserves global spatio-temporal structure and adapts to significant motion. We build FrameDiT-G, a DiT architecture based on MatrixAttention, and further introduce FrameDiT-H, which integrates
Matrix Attention with Local Factorized
Attention to capture both large and small motion. Extensive experiments show that FrameDiT-H achieves state-of-the-art results across multiple
video generation
benchmarks, offering improved temporal coherence and
video quality while maintaining efficiency comparable to Local Factorized
Attention.
Read the full abstract ▾
All authors Minh Khoa Le, Kien Do, Duc Thanh Nguyen, Truyen Tran
Computer Vision
GSStream: 3D Gaussian Splatting based Volumetric Scene Streaming System
Zhiye Tang, Qiudan Zhang, Lei Zhang et al. (+3 more)
Recently, the 3D Gaussian splatting (3DGS) technique for real-time radiance field rendering has revolutionized the field of volumetric scene representation, providing users with an immersive experience. But in return, it also poses a large amount of data volume, which is extremely bandwidth-intensive. Cutting-edge researchers have tried to introduce different approaches and construct multiple variants for 3DGS to obtain a more compact scene representation, but it is still challenging for real-time distribution. In this paper, we propose GSStream, a novel volumetric scene streaming system to support 3DGS data format. Specifically, GSStream integrates a collaborative viewport prediction module to better predict users' future behaviors by learning collaborative priors and historical priors from multiple users and users' viewport sequences and a deep reinforcement learning (DRL)-based bitrate adaptation module to tackle the state and action space variability challenge of the bitrate adaptation problem, achieving efficient volumetric scene delivery. Besides, we first build a user viewport trajectory
dataset for volumetric scenes to support the training and streaming simulation. Extensive experiments prove that our proposed GSStream system outperforms existing representative volumetric scene streaming systems in visual quality and network usage. Demo
video: https://youtu.be/3WEe8PN8yvA.
Read the full abstract ▾
All authors Zhiye Tang, Qiudan Zhang, Lei Zhang, Junhui Hou, You Yang, Xu Wang
Computer Vision
ProGS: Towards Progressive Coding for 3D Gaussian Splatting
Zhiye Tang, Lingzhuo Liu, Shengjie Jiao et al. (+4 more)
Progressive transmission of 3D Gaussian Splatting (3DGS) requires each completed transmission stage to be decodable from received data and directly renderable. This work presents ProGS, a progressive codec that organizes anchor-based 3DGS as parent-closed octree prefixes. ProGS combines parent-causal
entropy coding, level-balanced anchor growth, bounded multi-prefix training, and lightweight parent-anchor refinement to improve early-prefix quality without altering the complete-model rendering path. One fixed-$\lambda$ training run yields five deployable rate–quality points from a single bitstream. Experiments on 17 scenes from three
datasets evaluate rate–distortion performance, rendering speed, and transmission efficiency against progressive and single-rate baselines. On one representative scene per
dataset, ProGS reaches a common quality target with 30.7 $\sim$ 60.7% fewer bytes than HAC-Rand and 22.6 $\sim$ 53.4% fewer bytes than HAC++-Rand. Across the three
dataset averages, ProGS-LR uses 4.7 $\sim$ 6.1% fewer bytes than HAC-high while improving SSIM by 0.002 $\sim$ 0.041 and reducing LPIPS by 4.8 $\sim$ 51.2%. The parent-closed syntax makes every prefix causally decodable and directly renderable without future
topology. ProGS-HR also yields higher endpoint SSIM and lower LPIPS than PCGS across all three
dataset averages, and all five prefixes render in real time. Code is available at https://github.com/ZhiyeTang/ProGS-Official
Read the full abstract ▾
All authors Zhiye Tang, Lingzhuo Liu, Shengjie Jiao, Qiudan Zhang, Junhui Hou, You Yang, Xu Wang
Computer Vision
TriFusion-SR: Joint Tri-Modal Medical Image Fusion and SR
Fayaz Ali Dharejo, Sharif S. M. A., Aiman Khalil et al. (+3 more)
Multimodal medical image fusion facilitates comprehensive diagnosis by aggregating complementary structural and functional information, but its effectiveness is limited by resolution degradation and modality discrepancies. Existing approaches typically perform image fusion and super-resolution (SR) in separate stages, leading to artifacts and degraded perceptual quality. These limitations are further amplified in tri-modal settings that combine anatomical modalities (e.g., MRI, CT) with functional scans (e.g., PET, SPECT) due to pronounced frequency domain imbalances. We propose TriFusionSR, a wavelet-guided conditional
diffusion framework for joint tri-modal fusion and SR. The framework explicitly decomposes
multimodal features into frequency bands using the 2D Discrete Wavelet Transform, enabling frequency-aware crossmodal interaction. We further introduce a Rectified Wavelet Features (RWF) strategy for latent coefficient calibration, followed by an Adaptive Spatial-Frequency Fusion (ASFF) module with gated channel-spatial
attention to enable structure-driven
multimodal refinement. Extensive experiments demonstrate state-of-the-art performance, achieving 4.8-12.4% PSNR improvement and substantial reductions in RMSE and LPIPS across multiple upsampling scales.
Read the full abstract ▾
All authors Fayaz Ali Dharejo, Sharif S. M. A., Aiman Khalil, Nachiket Chaudhary, Rizwan Ali Naqvi, Radu Timofte
Computer Vision
TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answering
Luca Carlini, Chiara Lena, Cesare Hassan et al. (+4 more)
Surgical Video Question Answering (VideoQA) requires accurate temporal grounding while remaining robust to natural variation in how clinicians phrase questions, where linguistic bias can arise. Standard Parameter Efficient Fine Tuning (PEFT) methods adapt pretrained projections without explicitly modeling frame-to-frame interactions within the adaptation pathway, limiting their ability to exploit sparse temporal evidence. We introduce TemporalDoRA, a
video-specific PEFT formulation that extends Weight-Decomposed Low-Rank Adaptation by (i) inserting lightweight temporal Multi-Head
Attention (MHA) inside the low-rank bottleneck of the
vision encoder and (ii) selectively applying weight decomposition only to the trainable low-rank branch rather than the full adapted weight. This design enables temporally-aware updates while preserving a frozen backbone and stable scaling. By mixing information across frames within the adaptation subspace, TemporalDoRA steers updates toward temporally consistent visual cues and improves robustness with minimal parameter overhead. To
benchmark this setting, we present REAL-Colon-VQA, a colonoscopy VideoQA
dataset with 6,424 clip–question pairs, including paired rephrased Out-of-Template questions to evaluate sensitivity to linguistic variation. TemporalDoRA improves Out-of-Template performance, and ablation studies confirm that temporal mixing inside the low-rank branch is the primary driver of these gains. We also validate on EndoVis18-VQA adapted to short clips and observe consistent improvements on the Out-of-Template split. Code and
dataset available at~\href{https://anonymous.4open.science/r/TemporalDoRA-BFC8/}{Anonymous GitHub}.
Read the full abstract ▾
All authors Luca Carlini, Chiara Lena, Cesare Hassan, Danail Stoyanov, Elena De Momi, Sophia Bano, Mobarak I. Hoque
Computer Vision · Cross-listed from cs.RO
DRIFT: Dual-Representation Inter-Fusion Transformer for Automated Driving Perception with 4D Radar Point Clouds
Siqi Pei, Andras Palffy and Dariu M. Gavrila
4D radars, which provide 3D point cloud data along with Doppler velocity, are attractive components of modern automated driving systems due to their low cost and robustness under adverse weather conditions. However, they provide a significantly lower point cloud density than
LiDAR sensors. This makes it important to exploit not only local but also global contextual scene information. This paper proposes DRIFT, a model that effectively captures and fuses both local and global contexts through a dual-path architecture. The model incorporates a point path to aggregate fine-grained local features and a pillar path to encode coarse-grained global features. These two parallel paths are intertwined via novel feature-sharing layers at multiple stages, enabling full utilization of both representations. DRIFT is evaluated on the widely used View-of-Delft (VoD)
dataset and a proprietary internal
dataset. It outperforms the baselines on the tasks of object detection and/or free road estimation. For example, DRIFT achieves a mean average precision (mAP) of 52.6% (compared to, say, 45.4% of CenterPoint) on the VoD
dataset.
Read the full abstract ▾
Computer Vision
AutoViVQA: A Large-Scale Automatically Constructed Dataset for Vietnamese Visual Question Answering
Nguyen Anh Tuong, Phan Ba Duc, Nguyen Trung Quoc et al. (+4 more)
Visual Question Answering (VQA) is a fundamental
multimodal task that requires models to jointly understand visual and textual information. Early VQA systems relied heavily on
language biases, motivating subsequent work to emphasize visual grounding and balanced
datasets. With the success of large-scale pre-trained
transformers for both text and
vision domains – such as PhoBERT for Vietnamese
language understanding and
Vision Transformers (ViT) for image representation learning –
multimodal fusion has achieved remarkable progress. For Vietnamese VQA, several
datasets have been introduced to promote research in low-resource
multimodal learning, including ViVQA, OpenViVQA, and the recently proposed ViTextVQA. These resources enable benchmarking of models that integrate linguistic and visual features in the Vietnamese context. Evaluation of VQA systems often employs automatic metrics originally designed for image captioning or machine translation, such as BLEU, METEOR, CIDEr, Recall, Precision, and F1-score. However, recent research suggests that large
language models can further improve the
alignment between automatic evaluation and human judgment in VQA tasks. In this work, we explore Vietnamese Visual Question Answering using
transformer-based architectures, leveraging both textual and visual pre-training while systematically comparing automatic evaluation metrics under
multilingual settings.
Read the full abstract ▾
All authors Nguyen Anh Tuong, Phan Ba Duc, Nguyen Trung Quoc, Tran Dac Thinh, Dang Duy Lan, Nguyen Quoc Thinh, Tung Le
Computer Vision
Improving 3D Foot Motion Reconstruction in Markerless Monocular Human Motion Capture
Tom Wehrbein and Bodo Rosenhahn
State-of-the-art methods can recover accurate overall 3D human body motion from in-the-wild
videos. However, they often fail to capture fine-grained articulations, especially in the feet, which are critical for applications such as gait analysis and animation. This limitation results from training
datasets with inaccurate foot annotations and limited foot motion diversity. We address this gap with FootMR, a Foot Motion Refinement method that refines foot motion estimated by an existing human recovery model through lifting 2D foot keypoint sequences to 3D. By avoiding direct image input, FootMR circumvents inaccurate image-3D annotation pairs and can instead leverage large-scale motion capture data. To resolve ambiguities of 2D-to-3D lifting, FootMR incorporates knee and foot motion as context and predicts only residual foot motion. Generalization to extreme foot poses is further improved by representing joints in global rather than parent-relative rotations and applying extensive data augmentation. To support evaluation of foot motion reconstruction, we introduce MOOF, a 2D
dataset of complex foot movements. Experiments on MOOF, MOYO, and RICH show that FootMR outperforms state-of-the-art methods, reducing ankle joint angle error on MOYO by up to 30% over the best
video-based approach.
Read the full abstract ▾
Computer Vision
VarSplat: Uncertainty-aware 3D Gaussian Splatting for Robust RGB-D SLAM
Anh Thuan Tran and Jana Kosecka
Simultaneous Localization and Mapping (SLAM) with 3D Gaussian Splatting (3DGS) enables fast, differentiable rendering and high-fidelity reconstruction across diverse real-world scenes. However, existing 3DGS-SLAM approaches handle measurement reliability implicitly, making pose estimation and global
alignment susceptible to drift in low-texture regions, transparent surfaces, or areas with complex reflectance properties. To this end, we introduce VarSplat, an uncertainty-aware 3DGS-SLAM system that explicitly learns per-splat appearance variance. By using the law of total variance with alpha compositing, we then render differentiable per-pixel uncertainty map via efficient, single-pass rasterization. This map guides tracking, submap registration, and loop detection toward focusing on reliable regions and contributes to more stable optimization. Experimental results on Replica (synthetic) and TUM-RGBD, ScanNet, and ScanNet++ (real-world) show that VarSplat improves robustness and achieves competitive or superior tracking, mapping, and novel view synthesis rendering compared to existing studies for dense RGB-D SLAM.
Read the full abstract ▾
Computer Vision
DiffWind: Physics-Informed Differentiable Modeling of Wind-Driven Object Dynamics
Yuanhang Lei, Boming Zhao, Zesong Yang et al. (+10 more)
Modeling wind-driven object dynamics from
video observations is highly challenging due to the invisibility and spatio-temporal variability of wind, as well as the complex deformations of objects. We present DiffWind, a physics-informed differentiable framework that unifies wind-object interaction modeling,
video-based reconstruction, and forward simulation. Specifically, we represent wind as a grid-based physical field and objects as particle systems derived from 3D Gaussian Splatting, with their interaction modeled by the Material Point Method (MPM). To recover wind-driven object dynamics, we introduce a reconstruction framework that jointly optimizes the spatio-temporal wind force field and object motion through differentiable rendering and simulation. To ensure physical validity, we incorporate the
Lattice Boltzmann Method (LBM) as a physics-informed constraint, enforcing compliance with fluid dynamics laws. Beyond reconstruction, our method naturally supports forward simulation under novel wind conditions and enables new applications such as wind retargeting. We further introduce WD-Objects, a
dataset of synthetic and real-world wind-driven scenes. Extensive experiments demonstrate that our method significantly outperforms prior dynamic scene modeling approaches in both reconstruction accuracy and simulation fidelity, opening a new avenue for
video-based wind-object interaction modeling.
Read the full abstract ▾
All authors Yuanhang Lei, Boming Zhao, Zesong Yang, Xingxuan Li, Tao Cheng, Haocheng Peng, Ru Zhang, Yang Yang, Siyuan Huang, Yujun Shen, Ruizhen Hu, Hujun Bao, Zhaopeng Cui
Computer Vision
Tianyi Zeng, Jincheng Gao, Tianyi Wang et al. (+8 more)
Maintaining background consistency while enhancing foreground quality remains a core challenge in
video editing. Injecting full-image information often leads to background artifacts, whereas rigid background locking severely constrains the model's capacity for foreground generation. To address this issue, we propose KV-Lock, a training-free framework tailored for DiT-based
video diffusion models. Our core insight is that the
hallucination metric (variance of denoising prediction) directly quantifies generation diversity, which is inherently linked to the classifier-free guidance (CFG) scale. Building upon this, KV-Lock leverages
diffusion hallucination detection to dynamically schedule two key components: the fusion ratio between cached background key-values (KVs) and newly generated KVs, and the CFG scale. When
hallucination risk is detected, KV-Lock strengthens background KV locking and simultaneously amplifies conditional guidance for foreground generation, thereby mitigating artifacts and improving generation fidelity. As a training-free, plug-and-play module, KV-Lock can be easily integrated into any pre-trained DiT-based models. Extensive experiments validate that our method outperforms existing approaches in improved foreground quality with high background fidelity across various
video editing tasks.
Read the full abstract ▾
All authors Tianyi Zeng, Jincheng Gao, Tianyi Wang, Zijie Meng, Miao Zhang, Jun Yin, Haoyuan Sun, Junfeng Jiao, Christian Claudel, Junbo Tan, Xueqian Wang
Computer Vision
OTPL-VIO: Robust Visual-Inertial Odometry with Optimal Transport Line Association and Adaptive Uncertainty
Zikun Chen, Wentao Zhao, Yihe Niu et al. (+2 more)
Robust stereo visual-inertial odometry (VIO) remains challenging in low-texture scenes and under abrupt illumination changes, where point features become sparse and unstable, leading to ambiguous association and under-constrained estimation. Line structures offer complementary
geometric cues, yet many efficient point-line systems still rely on point-guided line association, which can break down when point support is weak and may lead to biased constraints. We present a stereo point-line VIO system in which line segments are equipped with dedicated deep descriptors and matched using an
entropy-regularized optimal transport formulation, which performs global mass-transport assignment and supports unmatched observations under ambiguity, outliers, and partial observations. The proposed descriptor is training-free and is computed by sampling and pooling network feature maps. To improve estimation stability, we analyze the impact of line measurement noise and introduce reliability-adaptive weighting to regulate the influence of line constraints during optimization. Experiments on EuRoC and UMA-VI, together with real-world deployments in low-texture and illumination-challenging environments, demonstrate improved accuracy and robustness over representative baselines while maintaining real-time performance.
Read the full abstract ▾
All authors Zikun Chen, Wentao Zhao, Yihe Niu, Tianchen Deng, Jingchuan Wang
Computer Vision
X-GS: An Extensible Framework for Perceiving and Thinking with 3D Gaussian Splatting
Yueen Ma, Zenglin Xu and Irwin King
3D Gaussian Splatting (3DGS) has emerged as a powerful technique for novel view synthesis, subsequently extending into numerous spatial AI applications. However, most existing 3DGS methods operate in isolation, focusing on specific domains. In this paper, we introduce X-GS, an extensible framework that integrates previously isolated 3DGS methods into the perception module of a VLM for spatial tasks, with two major components: the $\textit{Perceiver}$ and the $\textit{Thinker}$. The $\textit{Perceiver}$ performs online 3DGS-based SLAM with semantic
distillation and outputs semantic Gaussians from unposed
video streams. It leverages recent
vision foundation models for stronger
geometric priors, and we introduce three novel optimizations to improve semantic
distillation efficiency. The $\textit{Thinker}$ interfaces diverse VLMs with these semantic Gaussians, unlocking spatial
multimodal capabilities such as 3D visual grounding and scene captioning. Experimental results on diverse
benchmarks demonstrate the efficiency and newly unlocked
multimodal capabilities of the X-GS framework.
Read the full abstract ▾
Computer Vision
Grounding Synthetic Data Generation With Vision and Language Models
Ümit Mert Çağlar and Alptekin Temizel
Deep learning models benefit from increasing data diversity and volume, motivating synthetic data augmentation to improve existing
datasets. However, existing evaluation metrics for synthetic data typically calculate latent feature similarity, which is difficult to interpret and does not always correlate with the contribution to downstream tasks. We propose a
vision-
language grounded framework for interpretable synthetic data augmentation and evaluation in remote sensing. Our approach combines generative models, semantic
segmentation and image captioning with
vision and
language models. Based on this framework, we introduce ARAS400k: A large-scale Remote sensing
dataset Augmented with Synthetic data for
segmentation and captioning, containing 100k real images and 300k synthetic images, each paired with
segmentation maps and descriptions. ARAS400k enables the automated evaluation of synthetic data by analyzing semantic composition, minimizing caption redundancy, and verifying cross-modal consistency between visual structures and
language descriptions. Experimental results indicate that while models trained exclusively on synthetic data reach competitive performance levels, those trained with augmented data (a combination of real and synthetic images) consistently outperform real-data baselines. Consequently, this work establishes a scalable
benchmark for remote sensing tasks, specifically in semantic
segmentation and image captioning. The
dataset is available at zenodo.org/records/18890661 and the code base at github.com/caglarmert/ARAS400k.
Read the full abstract ▾
Computer Vision
Decoder-Free Distillation for Quantized Image Restoration
S. M. A. Sharif, Abdur Rehman, Seongwan Kim et al. (+1 more)
Quantization-Aware Training (QAT), combined with Knowledge
Distillation (KD), holds immense promise for compressing models for edge deployment. However, joint optimization for precision-sensitive image restoration (IR) to recover visual quality from degraded images remains largely underexplored. Directly adapting QAT-KD to low-level
vision reveals three critical bottlenecks: teacher-student capacity mismatch, spatial error amplification during decoder
distillation, and an optimization "tug-of-war" between reconstruction and
distillation losses caused by quantization noise. To tackle these, we introduce Quantization-aware Distilled Restoration (QDR), a framework for edge-deployed IR. QDR eliminates capacity mismatch via FP32 self-
distillation and prevents error amplification through Decoder-Free
Distillation (DFD), which corrects quantization errors strictly at the network bottleneck. To stabilize the optimization tug-of-war, we propose a Learnable Magnitude Reweighting (LMR) that dynamically balances competing gradients. Finally, we design an Edge-Friendly Model (EFM) featuring a lightweight Learnable Degradation Gating (LDG) to dynamically modulate spatial degradation localization. Extensive experiments across four IR tasks demonstrate that our Int8 model recovers 96.5% of FP32 performance, achieves 442 frames per second (FPS) on an NVIDIA Jetson Orin, and boosts downstream object detection by 16.3 mAP
Read the full abstract ▾
All authors S. M. A. Sharif, Abdur Rehman, Seongwan Kim, Jaeho Lee
Computer Vision
Physics-Driven 3D Gaussian Rendering for Zero-Shot MRI Super-Resolution
Shuting Liu, Lei Zhang, Wei Huang et al. (+2 more)
High-resolution
Magnetic Resonance Imaging (MRI) is vital for clinical diagnosis but limited by long acquisition times and motion artifacts. Super-resolution (SR) reconstructs low-resolution scans into high-resolution images, yet existing methods are mutually constrained: paired-data methods achieve efficiency only by relying on costly aligned
datasets, while implicit neural representation approaches avoid such data needs at the expense of heavy computation. We propose a zero-shot MRI SR framework using explicit Gaussian representation to balance data requirements and efficiency. MRI-tailored Gaussian parameters embed tissue physical properties, reducing learnable parameters while preserving MR signal fidelity. A physics-grounded volume rendering strategy models MRI signal formation via normalized Gaussian aggregation. Additionally, a brick-based order-independent rasterization scheme enables highly parallel 3D computation, lowering training and inference costs. Experiments on two public MRI
datasets show superior reconstruction quality and efficiency, demonstrating the method's potential for clinical MRI SR.
Read the full abstract ▾
All authors Shuting Liu, Lei Zhang, Wei Huang, Zhao Zhang, Zizhou Wang
Computer Vision
A Saccade-inspired Approach to Image Classification using Vision Transformer Attention Maps
Matthis Dallain, Laurent Rodriguez, Laurent Udo Perrinet et al. (+1 more)
Human
vision achieves remarkable perceptual performance while operating under strict metabolic constraints. A key ingredient is the selective
attention mechanism, driven by rapid saccadic eye movements that constantly reposition the high-resolution fovea onto task-relevant locations, unlike conventional AI systems that process entire images with equal emphasis. Our work aims to draw inspiration from the human visual system to create smarter, more efficient image processing models. Using DINO, a self-supervised
Vision Transformer that produces
attention maps strikingly similar to human gaze patterns, we explore a saccade inspired method to focus the processing of information on key regions in visual space. To do so, we use the ImageNet
dataset in a standard classification task and measure how each successive saccade affects the model's class scores. This selective-processing strategy preserves most of the full-image classification performance and can even outperform it in certain cases. By benchmarking against established saliency models built for human gaze prediction, we demonstrate that DINO provides superior fixation guidance for selecting informative regions. These findings highlight
Vision Transformer attention as a promising basis for biologically inspired active
vision and open new directions for efficient, neuromorphic visual processing.
Read the full abstract ▾
All authors Matthis Dallain, Laurent Rodriguez, Laurent Udo Perrinet, Benoît Miramond
Computer Vision
ParTY: Part-Guidance for Expressive Text-to-Motion Synthesis
KunHo Heo, SuYeon Kim, Yonghyun Gwon et al. (+2 more)
Text-to-motion synthesis aims to generate natural and expressive human motions from textual descriptions. While existing approaches primarily focus on generating holistic motions from text descriptions, they struggle to accurately reflect actions involving specific body parts. Recent part-wise motion generation methods attempt to resolve this but face two critical limitations: (i) they lack explicit mechanisms for aligning textual semantics with individual body parts, and (ii) they often generate incoherent full-body motions due to integrating independently generated part motions. To overcome these issues and resolve the fundamental trade-off in existing methods, we propose ParTY, a novel framework that enhances part expressiveness while generating coherent full-body motions. ParTY comprises: (1) Part-Guided Network, which first generates part motions to obtain part guidance, then uses it to generate holistic motions; (2) Part-aware Text Grounding, which diversely transforms text
embeddings and appropriately aligns them with each body part; and (3) Holistic-Part Fusion, which adaptively fuses holistic motions and part motions. Extensive experiments, including part-level and coherence-level evaluations, demonstrate that ParTY achieves substantial improvements over previous methods.
Read the full abstract ▾
All authors KunHo Heo, SuYeon Kim, Yonghyun Gwon, Youngbin Kim, MyeongAh Cho
Computer Vision
Chaodong Xiao, Zhengqiang Zhang and Lei Zhang
Transformers have achieved widespread and remarkable success, while the computational complexity of their
attention modules remains a major bottleneck for
vision tasks. Existing methods mainly employ 8-bit or 4-bit quantization to balance efficiency and accuracy. In this paper, with theoretical justification, we indicate that binarization of
attention preserves the essential similarity relationships, and propose BinaryAttention, an effective method for fast and accurate 1-bit qk-
attention. Specifically, we retain only the sign of queries and keys in computing the
attention, and replace the floating dot products with bit-wise operations, significantly reducing the computational cost. We mitigate the inherent information loss under 1-bit quantization by incorporating a learnable bias, and enable end-to-end acceleration. To maintain the accuracy of
attention, we adopt quantization-aware training and self-
distillation techniques, mitigating quantization errors while ensuring sign-aligned similarity. BinaryAttention is more than 2x faster than FlashAttention2 on A100 GPUs. Extensive experiments on
vision transformer and
diffusion transformer benchmarks demonstrate that BinaryAttention matches or even exceeds full-precision
attention, validating its effectiveness. Our work provides a highly efficient and effective alternative to full-precision
attention, pushing the frontier of low-bit
vision and
diffusion transformers. The codes and models can be found at https://github.com/EdwardChasel/BinaryAttention.
Read the full abstract ▾
Computer Vision
More than the Sum: Panorama-Language Models for Adverse Omni-Scenes
Weijia Fan, Ruiping Liu, Jiale Wei et al. (+7 more)
Existing
vision-
language models (VLMs) are tailored for pinhole imagery, stitching multiple narrow field-of-view inputs to piece together a complete omni-scene understanding. Yet, such multi-view perception overlooks the holistic spatial and contextual relationships that a single panorama inherently preserves. In this work, we introduce the Panorama-
Language Modeling (PLM)paradigm, a unified $360^\circ$
vision-
language reasoning that is more than the sum of its pinhole counterparts. Besides, we present PanoVQA, a large-scale panoramic VQA
dataset that involves adverse omni-scenes, enabling comprehensive
reasoning under object occlusions and driving accidents. To establish a foundation for PLM, we develop a plug-and-play panoramic sparse
attention module that allows existing pinhole-based VLMs to process equirectangular panoramas without retraining. Extensive experiments demonstrate that our PLM achieves superior robustness and holistic
reasoning under challenging omni-scenes, yielding understanding greater than the sum of its narrow parts. Project page: https://github.com/InSAI-Lab/PanoVQA.
Read the full abstract ▾
All authors Weijia Fan, Ruiping Liu, Jiale Wei, Yufan Chen, Junwei Zheng, Zichao Zeng, Jiaming Zhang, Qiufu Li, Linlin Shen, Rainer Stiefelhagen
Computer Vision
GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning
Xiao Yang, Ronghao Fu, Zhuoran Duan et al. (+3 more)
Vision-
language pretraining models have made significant progress in bridging remote sensing imagery with natural
language. However, existing approaches often fail to effectively integrate multi-granular visual and textual information, relying primarily on global image-text
alignment. This limitation hinders the model's ability to accurately capture fine-grained details in images, thus restricting its performance in complex, fine-grained tasks. To address this, we propose GeoAlignCLIP, a unified framework that achieves fine-grained
alignment in remote sensing tasks by learning multi-granular semantic
alignments and incorporating intra-modal consistency, enabling more precise visual-semantic
alignment between image regions and text concepts. Additionally, we construct RSFG-100k, a fine-granular remote sensing
dataset containing scene descriptions, region-level annotations, and challenging hard-negative samples, providing hierarchical supervision for model training. Extensive experiments conducted on multiple public remote-sensing
benchmarks demonstrate that GeoAlignCLIP consistently outperforms existing RS-specific methods across diverse tasks, exhibiting more robust and accurate fine-grained
vision-
language alignment.
Read the full abstract ▾
All authors Xiao Yang, Ronghao Fu, Zhuoran Duan, Zhiwen Lin, Xueyan Liu, Bo Yang
Computer Vision
GeoSolver: Scaling Test-Time Reasoning in Remote Sensing with Fine-Grained Process Supervision
Lang Sun, Ronghao Fu, Zhuoran Duan et al. (+3 more)
While
Vision-
Language Models (VLMs) have significantly advanced remote sensing interpretation, enabling them to perform complex, step-by-step
reasoning remains highly challenging. Recent efforts to introduce Chain-of-Thought (CoT)
reasoning to this domain have shown promise, yet ensuring the visual faithfulness of these intermediate steps remains a critical bottleneck. To address this, we introduce GeoSolver, a novel framework that transitions remote sensing
reasoning toward verifiable, process-supervised reinforcement learning. We first construct Geo-PRM-2M, a large-scale,
token-level process supervision
dataset synthesized via
entropy-guided Monte Carlo Tree Search (MCTS) and targeted visual
hallucination injection. Building upon this
dataset, we train GeoPRM, a
token-level process reward model (PRM) that provides granular faithfulness feedback. To effectively leverage these verification signals, we propose Process-Aware Tree-GRPO, a reinforcement learning
algorithm that integrates tree-structured exploration with a faithfulness-weighted reward mechanism to precisely assign credit to intermediate steps. Extensive experiments demonstrate that our resulting model, GeoSolver-9B, achieves state-of-the-art performance across diverse remote sensing
benchmarks. Crucially, GeoPRM unlocks robust Test-Time Scaling (TTS). Serving as a universal geospatial verifier, it seamlessly scales the performance of GeoSolver-9B and directly enhances general-purpose VLMs, highlighting its remarkable cross-model generalization.
Read the full abstract ▾
All authors Lang Sun, Ronghao Fu, Zhuoran Duan, Haoran Liu, Xueyan Liu, Bo Yang
Computer Vision
A comprehensive study of time-of-flight non-line-of-sight imaging
Julio Marco, Adrian Jarabo, Ji Hyun Nam et al. (+3 more)
Time-of-Flight non-line-of-sight (ToF NLOS) imaging techniques provide state-of-the-art reconstructions of scenes hidden around corners by inverting the optical path of indirect
photons scattered by visible surfaces and measured by picosecond resolution sensors. The emergence of a wide range of ToF NLOS imaging methods with heterogeneous formulae and hardware implementations obscures the assessment of both their theoretical and experimental aspects. We present a comprehensive study of a representative set of ToF NLOS imaging methods by discussing their similarities and differences under common formulation and hardware. We first outline the problem statement under a common general forward model for ToF NLOS measurements, and the typical assumptions that yield tractable inverse models. We discuss the relationship of the resulting simplified forward and inverse models to a family of Radon transforms, and how migrating these to the frequency domain relates to recent phasor-based virtual line-of-sight imaging models for NLOS imaging that obey the constraints of conventional lens-based imaging systems. We then evaluate performance of the selected methods on hidden scenes captured under the same hardware setup and similar
photon counts. Our experiments show that existing methods share similar limitations on spatial resolution, visibility, and sensitivity to noise when operating under equal hardware constraints, with particular differences that stem from method-specific parameters. We expect our methodology to become a reference in future research on ToF NLOS imaging to obtain objective comparisons of existing and new methods.
Read the full abstract ▾
All authors Julio Marco, Adrian Jarabo, Ji Hyun Nam, Alberto Tosi, Diego Gutierrez, Andreas Velten
Computer Vision
Memory-Efficient Active Perception for Embodied Question Answering in Dynamic Human-Populated Scenes
Xin Lu, Rui Li, Xun Huang et al. (+5 more)
Embodied Question Answering (EQA) couples active visual perception,
vision-
language reasoning, and sequential decision making, and has traditionally been evaluated in temporally stable environments where visual evidence can be accumulated reliably. In realistic multimedia settings such as dynamic human-populated scenes, however, moving people and transient occlusions introduce human-induced perceptual variations: task-relevant
multimodal cues become transient, viewpoint-dependent, and incompletely captured in individual views. Conventional store-then-retrieve strategies generate redundant visual evidence and incur high inference cost, making them ill-suited to such variations. This exposes two core problems for
embodied agents: resolving viewpoint-dependent ambiguity, and maintaining compact evidence for efficient inference. To address these challenges, we present DIVRR (Dynamic-Informed View Refinement with Relevance-guided Rotation and adaptive
memory selection), a training-free active evidence-acquisition framework for EQA. DIVRR combines question-conditioned room guidance with relevance-triggered view augmentation to guide exploration and additional observation. Selective
memory admission uses the same relevance signal as view augmentation to retain high-scoring evidence, reducing stored entries while keeping the retrieved visual context compact. To evaluate EQA under human-induced perceptual variations, we introduce DynHuman-EQA, a dynamic human-populated EQA
dataset constructed in Habitat, comprising dynamic and static subsets under shared scene layouts. Experiments on DynHuman-EQA and HM-EQA show that DIVRR outperforms existing baselines in both accuracy and
memory efficiency. The gains are most pronounced in dynamic human-populated scenes.
Read the full abstract ▾
All authors Xin Lu, Rui Li, Xun Huang, Weixin Li, Jiayuan Li, Chuanqing Zhuang, Zhengda Lu, Jun Xiao
Computer Vision
Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization
Ming Nie, Chunwei Wang, Jianhua Han et al. (+2 more)
Unified
vision-
language models have made significant progress in
multimodal understanding and generation, yet they largely fall short in producing
multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual
reasoning. In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale
multimodal interleaved
datasets. We begin with a warm-up stage using a hybrid
dataset comprising curated interleaved sequences and limited data for
multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities. To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the
multimodal setting. Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text
alignment, and structural fidelity. Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex
multimodal tasks. Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of
multimodal interleaved generation.
Read the full abstract ▾
All authors Ming Nie, Chunwei Wang, Jianhua Han, Hang Xu, Li Zhang
Computer Vision · Cross-listed from q-bio.QM
Association of Progressive PPFE and Mortality in Lung Cancer Screening Cohorts
Shahab Aslani, Mehran Azimbagirad, Daryl Cheng et al. (+16 more)
Background: Pleuroparenchymal fibroelastosis (PPFE) is an upper lobe predominant fibrotic lung abnormality associated with increased mortality in established interstitial lung disease. However, the clinical significance of radiologic PPFE progression in lung cancer screening (LCS) populations remains unclear. Methods: We analysed longitudinal low-dose CT scans and clinical data from two LCS studies: National Lung Screening Trial (NLST; n=7,980); SUMMIT study (n=8,561). An automated
algorithm quantified PPFE volume on baseline and follow-up scans. Annualised change in PPFE was derived and dichotomised using a distribution-based threshold to define progressive PPFE. Associations between progressive PPFE and mortality were evaluated using Cox proportional hazards models adjusted for demographic and clinical variables. In SUMMIT cohort, associations between progressive PPFE and clinical outcomes were assessed using incidence rate ratios (IRR) and odds ratios (OR). Findings: Progressive PPFE independently associated with mortality in both LCS cohorts (NLST: Hazard Ratio (HR)=1.25, 95% Confidence Interval (CI): 1.01–1.56, p=0.042; SUMMIT: HR=3.14, 95% CI: 1.66–5.97, p<0.001). Within SUMMIT, progressive PPFE was strongly associated with higher respiratory admissions (IRR=2.79, p<0.001), increased antibiotic and steroid use (IRR=1.55, p=0.010), and showed a trend towards higher modified medical research council scores (OR=1.40, p=0.055). Interpretation: Radiologic PPFE progression independently associates with mortality across two large LCS cohorts, and associates with adverse clinical outcomes. Quantitative assessment of PPFE progression may provide a clinically relevant imaging biomarker to identify individuals at increased risk of respiratory morbidity within LCS programmes.
Read the full abstract ▾
All authors Shahab Aslani, Mehran Azimbagirad, Daryl Cheng, Daisuke Yamada, Ryoko Egashira, Adam Szmul, Justine Chan-Fook, Robert Chapman, Alfred Chung Pui So, Shanshan Wang, John McCabe, Tianqi Yang, Jose M Brenes, Eyjolfur Gudmundsson, The SUMMIT Consortium, Susan M. Astley, Daniel C. Alexander, Sam M. Janes, Joseph Jacob
Computer Vision
DCAU-Net: Differential Cross Attention and Channel-Spatial Feature Fusion for Medical Image Segmentation
Yanxin Li, Hui Wan and Libin Lan
Accurate medical image
segmentation requires effective modeling of both long-range dependencies and fine-grained boundary details. While
transformers mitigate the issue of insufficient semantic information arising from the limited receptive field inherent in convolutional neural networks, they introduce new challenges: standard self-
attention incurs quadratic computational complexity and often assigns non-negligible
attention weights to irrelevant regions, diluting focus on discriminative structures and ultimately compromising
segmentation accuracy. Existing
attention variants, although effective in reducing computational complexity, fail to suppress redundant computation and inadvertently impair global context modeling. Furthermore, conventional fusion strategies in encoder-decoder architectures, typically based on simple concatenation or summation, can not adaptively integrate high-level semantic information with low-level spatial details. To address these limitations, we propose DCAU-Net, a novel yet efficient
segmentation framework with two key ideas. First, a new Differential Cross
Attention (DCA) is designed to compute the difference between two independent softmax
attention maps to adaptively highlight discriminative structures. By replacing pixel-wise key and value
tokens with window-level summary
tokens, DCA dramatically reduces computational complexity without sacrificing precision. Second, a Channel-Spatial Feature Fusion (CSFF) strategy is introduced to adaptively recalibrate features from skip connections and up-sampling paths through using sequential channel and spatial
attention, effectively suppressing redundant information and amplifying salient cues. Experiments on two public
benchmarks demonstrate that DCAU-Net achieves competitive performance with enhanced
segmentation accuracy and robustness.
Read the full abstract ▾
Computer Vision
RESBev: Making BEV Perception More Robust
Lifeng Zhuo, Kefan Jin, Zhe Liu et al. (+1 more)
Bird's-eye-view (BEV) perception has emerged as a cornerstone of autonomous driving systems, providing a structured, ego-centric representation critical for downstream planning and control. However, real-world deployment faces challenges from sensor degradation and
adversarial attacks, which can cause severe perceptual anomalies and ultimately compromise the safety of autonomous driving systems. To address this, we propose a resilient and plug-and-play BEV perception method, RESBev, which can be easily applied to existing BEV perception methods to enhance their robustness to diverse disturbances. Specifically, we reframe perception robustness as a latent semantic prediction problem. A latent world model is constructed to extract spatiotemporal correlations across sequential BEV observations, thereby learning the underlying BEV state transitions to predict clean BEV features for reconstructing corrupted observations. The proposed framework operates at the semantic feature level of the Lift-Splat-Shoot pipeline, enabling recovery that generalizes across both natural disturbances and
adversarial attacks without modifying the underlying backbone. Extensive experiments on the nuScenes
dataset demonstrate that, with few-shot fine-tuning, RESBev significantly improves the robustness of existing BEV perception models against various external disturbances and
adversarial attacks.
Read the full abstract ▾
All authors Lifeng Zhuo, Kefan Jin, Zhe Liu, Hesheng Wang
Computer Vision
Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoning
Chun-Peng Chang, Chen-Yu Wang, Holger Caesar et al. (+1 more)
A reliable driving assistant should provide consistent responses based on temporally grounded
reasoning derived from observed information. In this work, we investigate whether
Vision-
Language Models (VLMs), when applied as driving assistants, can response consistantly and understand how present observations shape future outcomes, or whether their outputs merely reflect patterns memorized during training without temporally grounded
reasoning. While recent efforts have integrated VLMs into autonomous driving, prior studies typically emphasize scene understanding and instruction generation, implicitly assuming that strong visual interpretation naturally enables consistant future
reasoning and thus ensures reliable decision-making, a claim we critically examine. We focus on two major challenges limiting VLM reliability in this setting: response inconsistency, where minor input perturbations yield different answers or, in some cases, responses degenerate toward near-random guessing, and limited temporal
reasoning, in which models fail to reason and align sequential events from current observations, often resulting in incorrect or even contradictory responses. Moreover, we find that models with strong visual understanding do not necessarily perform best on tasks requiring temporal
reasoning, indicating a tendency to over-rely on pretrained patterns rather than modeling temporal dynamics. To address these issues, we adopt existing evaluation methods and introduce FutureVQA, a human-annotated
benchmark dataset specifically designed to assess future scene
reasoning. In addition, we propose a simple yet effective self-supervised tuning approach with chain-of-thought
reasoning that improves both consistency and temporal
reasoning without requiring temporal labels.
Read the full abstract ▾
All authors Chun-Peng Chang, Chen-Yu Wang, Holger Caesar, Alain Pagani
Computer Vision
Context-Nav: Context-Driven Exploration and Viewpoint-Aware 3D Spatial Reasoning for Instance Navigation
Won Shik Jang and Ue-Hwan Kim
Text-goal instance
navigation (TGIN) asks an
agent to resolve a single, free-form description into actions that reach the correct object instance among same-category distractors. We present Context-Nav, which elevates long, contextual captions from a local matching cue to a global exploration prior and verifies candidates through 3D spatial
reasoning. First, we compute dense text-image
alignments for a value map that ranks frontiers – guiding exploration toward regions consistent with the entire description rather than early detections. Second, upon observing a candidate, we perform a viewpoint-aware relation check: the
agent samples plausible observer poses, aligns local frames, and accepts a target only if the spatial relations can be satisfied from at least one viewpoint. The pipeline requires no task-specific training or fine-tuning; we attain state-of-the-art performance on InstanceNav and CoIN-Bench. Ablations show that (i) encoding full captions into the value map avoids wasted motion and (ii) explicit, viewpoint-aware 3D verification prevents semantically plausible but incorrect stops. This suggests that geometry-grounded spatial
reasoning is a scalable alternative to heavy policy training or human-in-the-loop interaction for fine-grained instance disambiguation in cluttered 3D scenes.
Read the full abstract ▾