Computer Vision
SUG-Occ: Explicit Semantics and Uncertainty Guided Sparse Learning for Efficient 3D Occupancy Prediction
Hanlin Wu, Pengfei Lin, Ehsan Javanmardi et al. (+4 more)
3D semantic occupancy prediction has emerged as a critical perception task for autonomous driving due to its ability to offer voxel-level semantic and
geometric understanding of the environment. However, such a refined representation for large-scale scenes incurs prohibitive computation, posing a significant challenge to practical real-time deployment. To address this, we propose SUGOcc, an explicit semantics and uncertainty guided sparse learning framework for efficient occupancy prediction, which exploits the inherent sparsity of 3D scenes to reduce redundant computation while maintaining
geometric and semantic integrity. Specifically, we first utilize semantic and uncertainty priors to suppress image projections from free space while employing explicit unsigned distance encoding to enhance
geometric consistency, thereby producing a structurally sparse representation. Secondly, we introduce a cascade sparse completion module to enable efficient coarse-to-fine
reasoning over the sparse representation via hyper cross sparse convolution, generative upsampling and adaptive
pruning. Finally, we propose an object contextual representation (OCR) based mask decoder that refines the voxel-wise predictions through lightweight query-context interactions, thereby avoiding expensive
attention operations over volumetric features. Extensive experiments on SemanticKITTI and Occ3D-Nuscenes
benchmark demonstrate that the proposed approach outperforms the baselines, achieving notable improvements in both accuracy and efficiency across
datasets.
Read the full abstract ▾
All authors Hanlin Wu, Pengfei Lin, Ehsan Javanmardi, Naren Bao, Bo Qian, Hao Si, Manabu Tsukada
Computer Vision
Heterogeneous Uncertainty-Guided Composed Image Retrieval with Fine-Grained Probabilistic Learning
Haomiao Tang, Jinpeng Wang, Minyi Zhao et al. (+4 more)
Composed Image
Retrieval (CIR) enables image search by combining a reference image with modification text. Intrinsic noise in CIR triplets incurs intrinsic uncertainty and threatens the model's robustness. Probabilistic learning approaches have shown promise in addressing such issues; however, they fall short for CIR due to their instance-level holistic modeling and homogeneous treatment of queries and targets. This paper introduces a Heterogeneous Uncertainty-Guided (HUG) paradigm to overcome these limitations. HUG utilizes a fine-grained probabilistic learning framework, where queries and targets are represented by Gaussian
embeddings that capture detailed concepts and uncertainties. We customize heterogeneous uncertainty estimations for multi-modal queries and uni-modal targets. Given a query, we capture uncertainties not only regarding uni-modal content quality but also multi-modal coordination, followed by a provable dynamic weighting mechanism to derive comprehensive query uncertainty. We further design uncertainty-guided objectives, including query-target holistic contrast and fine-grained contrasts with comprehensive negative sampling strategies, which effectively enhance discriminative learning. Experiments on
benchmarks demonstrate HUG's effectiveness beyond state-of-the-art baselines, with faithful analysis justifying the technical contributions.
Read the full abstract ▾
All authors Haomiao Tang, Jinpeng Wang, Minyi Zhao, Guanghao Meng, Ruisheng Luo, Long Chen, Shu-Tao Xia
Computer Vision
Think-Clip-Sample: Slow-Fast Frame Selection for Video Understanding
Wenhui Tan, Ruihua Song, Jiaze Li et al. (+2 more)
Recent progress in multi-modal large
language models (MLLMs) has significantly advanced
video understanding. However, their performance on long-form
videos remains limited by computational constraints and suboptimal frame selection. We present Think-Clip-Sample (TCS), a training-free framework that enhances long
video understanding through two key components: (i) Multi-Query
Reasoning, which generates multiple queries to capture complementary aspects of the question and
video; and (ii) Clip-level Slow-Fast Sampling, which adaptively balances dense local details and sparse global context. Extensive experiments on MLVU, LongVideoBench, and VideoMME demonstrate that TCS consistently improves performance across different MLLMs, boosting up to 6.9% accuracy, and is capable of achieving comparable accuracy with 50% fewer inference time cost, highlighting both efficiency and efficacy of TCS on long
video understanding.
Read the full abstract ▾
All authors Wenhui Tan, Ruihua Song, Jiaze Li, Jianzhong Ju, Zhenbo Luo
Computer Vision
Assessing Building Heat Resilience Using UAV and Street-View Imagery with Coupled Global Context Vision Transformer
Steffen Knoblauch, Ram Kumar Muthusamy, Hao Li et al. (+4 more)
Climate change is intensifying human heat exposure, particularly in densely built urban centers of the Global South. Low-cost construction materials and high thermal-mass surfaces further exacerbate this risk. Yet scalable methods for assessing such heat-relevant building attributes remain scarce. We propose a machine learning framework that fuses openly available unmanned aerial vehicle (
UAV) and street-view (SV) imagery via a coupled global context
vision transformer (CGCViT) to learn heat-relevant representations of urban structures. Thermal infrared (TIR) measurements from HotSat-1 are used to quantify the relationship between building attributes and heat-associated health risks. Our dual-modality cross-view learning approach outperforms the best single-modality models by up to $9.3\%$, demonstrating that
UAV and SV imagery provide valuable complementary perspectives on urban structures. The presence of vegetation surrounding buildings (versus no vegetation), brighter roofing (versus darker roofing), and roofing made of concrete, clay, or wood (versus metal or tarpaulin) are all significantly associated with lower HotSat-1 TIR values. Deployed across the city of Dar es Salaam, Tanzania, the proposed framework illustrates how household-level inequalities in heat exposure - often linked to socio-economic disadvantage and reflected in building materials - can be identified and addressed using machine learning. Our results point to the critical role of localized, data-driven risk assessment in shaping
climate adaptation strategies that deliver equitable outcomes.
Read the full abstract ▾
All authors Steffen Knoblauch, Ram Kumar Muthusamy, Hao Li, Iddy Chazua, Benedcto Adamu, Innocent Maholi, Alexander Zipf
Computer Vision
Beer-Lambert Autoencoder for Unsupervised Stain Representation Learning and Deconvolution in Multi-immunohistochemical Brightfield Histology Images
Mark Eastwood, Thomas McKee, Zedong Hu et al. (+2 more)
Separating the contributions of individual chromogenic stains in RGB histology whole slide images (WSIs) is essential for stain normalization, quantitative assessment of marker expression, and
cell-level readouts in immunohistochemistry (IHC). Classical Beer-Lambert (BL) color deconvolution is well-established for two- or three-stain settings, but becomes under-determined and unstable for multiplex IHC (mIHC) with K>3 chromogens. We present a simple, data-driven encoder-decoder architecture that learns cohort-specific stain characteristics for mIHC RGB WSIs and yields crisp, well-separated per-stain concentration maps. The encoder is a compact U-Net that predicts K nonnegative concentration channels; the decoder is a differentiable BL forward model with a learnable stain
matrix initialized from typical chromogen hues. Training is unsupervised with a perceptual reconstruction objective augmented by loss terms that discourage unnecessary stain mixing. On a colorectal mIHC panel comprising 5 stains (H, CDX2, MUC2, MUC5, CD8) we show excellent RGB reconstruction, and significantly reduced inter-channel bleed-through compared with
matrix-based deconvolution. Code and model are available at https://github.com/measty/StainQuant.git.
Read the full abstract ▾
All authors Mark Eastwood, Thomas McKee, Zedong Hu, Sabine Tejpar, Fayyaz Minhas
Computer Vision
Enhancing Vision Language Models with Logic Reasoning for Situational Awareness
Pavana Pradeep, Krishna Kant and Suya Yu
Vision-
Language Models (VLMs) offer the ability to generate high-level, interpretable descriptions of complex activities from images and
videos, making them valuable for situational awareness (SA) applications. In such settings, the focus is on identifying infrequent but significant events with high reliability and accuracy, while also extracting fine-grained details and assessing recognition quality. In this paper, we propose an approach that integrates VLMs with traditional computer
vision methods through explicit logic
reasoning to enhance SA in three key ways: (a) extracting fine-grained event details, (b) employing an intelligent fine-tuning (FT) strategy that achieves substantially higher accuracy than uninformed selection, and (c) generating justifications for VLM outputs during inference. We demonstrate that our intelligent FT mechanism improves the accuracy and provides a valuable means, during inferencing, to either confirm the validity of the VLM output or indicate why it may be questionable.
Read the full abstract ▾
Computer Vision
Antoine Carreaud, Elias Naha, Arthur Chansel et al. (+3 more)
Semantic ultra-high-resolution (UHR) image
segmentation is essential in remote sensing applications such as aerial mapping and environmental monitoring.
Transformer-based models remain challenging in this setting because
memory grows quadratically with the number of
tokens, limiting either spatial resolution or contextual scope. We introduce CASWiT (Context-Aware Stage-Wise
Transformer), a dual-branch Swin-based architecture that injects low-resolution contextual information into fine-grained high-resolution features through lightweight stage-wise cross-
attention. To strengthen cross-scale learning, we also propose a SimMIM-style pretraining strategy based on masked reconstruction of the high-resolution image. Extensive experiments on the large-scale FLAIR-HUB aerial
dataset demonstrate the effectiveness of CASWiT. Under our RGB-only UHR protocol, CASWiT reaches 66.37% mIoU with a SegFormer decoder, improving over strong RGB baselines while also improving boundary quality. On the URUR
benchmark, CASWiT reaches 49.2% mIoU under the official evaluation protocol, and it also transfers effectively to medical UHR
segmentation benchmarks. Code and pretrained models are available at https://huggingface.co/collections/heig-vd-geo/caswit
Read the full abstract ▾
All authors Antoine Carreaud, Elias Naha, Arthur Chansel, Nina Lahellec, Jan Skaloud, Adrien Gressin
Computer Vision
SAMannot: A Memory-Efficient, Local, Open-source Framework for Interactive Video Instance Segmentation based on SAM2
Gergely Dinya, András Gelencsér, Krisztina Kupán et al. (+3 more)
Current research workflows for precise
video segmentation are often forced into a compromise between labor-intensive manual curation, costly commercial platforms, and/or
privacy-compromising cloud-based services. The demand for high-fidelity
video instance
segmentation in research is often hindered by the bottleneck of manual annotation and the
privacy concerns of cloud-based tools. We present SAMannot, an open-source, local framework that integrates the Segment Anything Model 2 (SAM2) into a human-in-the-loop workflow. To address the high resource requirements of foundation models, we modified the SAM2 dependency and implemented a processing layer that minimizes computational overhead and maximizes throughput, ensuring a highly responsive user interface. Key features include persistent instance identity management, an automated “lock-and-refine” workflow with barrier frames, and a mask-skeletonization-based auto-prompting mechanism. SAMannot facilitates the generation of research-ready
datasets in YOLO and PNG formats alongside structured interaction logs. Verified through animal behavior tracking use-cases and subsets of the LVOS and DAVIS
benchmark datasets, the tool provides a scalable, private, and cost-effective alternative to commercial platforms for complex
video annotation tasks.
Read the full abstract ▾
All authors Gergely Dinya, András Gelencsér, Krisztina Kupán, Clemens Küpper, Kristóf Karacs, Anna Gelencsér-Horváth
Computer Vision
Efficient On-Board Processing of Oblique UAV Video for Rapid Flood Extent Mapping
Vishisht Sharma, Sam Leroux, Lisa Landuyt et al. (+2 more)
Effective disaster response relies on rapid disaster response, where oblique aerial
video is the primary modality for initial scouting due to its ability to maximize spatial coverage and situational awareness in limited flight time. However, the on-board processing of high-resolution oblique streams is severely bottlenecked by the strict Size, Weight, and Power (SWaP) constraints of Unmanned Aerial Vehicles (
UAVs). The computational density required to process these wide-field-of-view streams precludes low-latency inference on standard edge hardware. To address this, we propose Temporal
Token Reuse (TTR), an adaptive inference framework capable of accelerating
video segmentation on embedded devices. TTR exploits the intrinsic spatiotemporal redundancy of aerial
video by formulating image patches as
tokens; it utilizes a lightweight similarity metric to dynamically identify static regions and propagate their precomputed deep features, thereby bypassing redundant backbone computations. We validate the framework on standard
benchmarks and a newly curated Oblique Floodwater
Dataset designed for hydrological monitoring. Experimental results on edge-grade hardware demonstrate that TTR achieves a 30% reduction in inference latency with negligible degradation in
segmentation accuracy (< 0.5% mIoU). These findings confirm that TTR effectively shifts the operational Pareto frontier, enabling high-fidelity, real-time oblique
video understanding for time-critical remote sensing missions
Read the full abstract ▾
All authors Vishisht Sharma, Sam Leroux, Lisa Landuyt, Nick Witvrouwen, Pieter Simoens
Computer Vision
X-Distill: Cross-Architecture Vision Distillation for Visuomotor Learning
Maanping Shao, Feihong Zhang, Gu Zhang et al. (+3 more)
Visuomotor policies often leverage large pre-trained
Vision Transformers (ViTs) for their powerful generalization capabilities. However, their significant data requirements present a major challenge in the data-scarce context of most robotic learning settings, where compact CNNs with strong inductive biases can be more easily optimized. To address this trade-off, we introduce X-Distill, a simple yet highly effective method that synergizes the strengths of both architectures. Our approach involves an offline, cross-architecture knowledge
distillation, transferring the rich visual representations of a large, frozen DINOv2 teacher to a compact ResNet-18 student on the general-purpose ImageNet
dataset. This distilled encoder, now endowed with powerful visual priors, is then jointly fine-tuned with a
diffusion policy head on the target manipulation tasks. Extensive experiments on $34$ simulated
benchmarks and $5$ challenging real-world tasks demonstrate that our method consistently outperforms policies equipped with from-scratch ResNet or fine-tuned DINOv2 encoders. Notably, X-Distill also surpasses 3D encoders that utilize privileged point cloud observations or much larger
Vision-
Language Models. Our work highlights the efficacy of a simple, well-founded
distillation strategy for achieving state-of-the-art performance in data-efficient robotic manipulation.
Read the full abstract ▾
All authors Maanping Shao, Feihong Zhang, Gu Zhang, Baiye Cheng, Zhengrong Xue, Huazhe Xu
Computer Vision
FTDMamba: Frequency-Assisted Temporal Dilation Mamba for Unmanned Aerial Vehicle Video Anomaly Detection
Cheng-Zhuang Liu, Si-Bao Chen, Qing-Ling Shu et al. (+3 more)
Recent advances in
video anomaly detection (VAD) mainly focus on ground-based surveillance or unmanned aerial vehicle (
UAV)
videos with static backgrounds, whereas research on
UAV videos with dynamic backgrounds remains limited. Unlike static scenarios, dynamically captured
UAV videos exhibit multi-source motion coupling, where the motion of objects and
UAV-induced global motion are intricately intertwined. Consequently, existing methods may misclassify normal
UAV movements as anomalies or fail to capture true anomalies concealed within dynamic backgrounds. Moreover, many approaches do not adequately address the joint modeling of inter-frame continuity and local spatial correlations across diverse temporal scales. To overcome these limitations, we propose the Frequency-Assisted Temporal Dilation Mamba (FTDMamba) network for
UAV VAD, including two core components: (1) a Frequency Decoupled Spatiotemporal Correlation Module, which disentangles coupled motion patterns and models global spatiotemporal dependencies through frequency analysis; and (2) a Temporal Dilation Mamba Module, which leverages Mamba's sequence modeling capability to jointly learn fine-grained temporal dynamics and local spatial structures across multiple temporal receptive fields. Additionally, unlike existing
UAV VAD
datasets which focus on static backgrounds, we construct a large-scale Moving
UAV VAD
dataset (MUVAD), comprising 222,736 frames with 240 anomaly events across 12 anomaly types. Extensive experiments demonstrate that FTDMamba achieves state-of-the-art (SOTA) performance on two public static
benchmarks and the new MUVAD
dataset. The code and MUVAD
dataset will be available at: https://github.com/uavano/FTDMamba.
Read the full abstract ▾
All authors Cheng-Zhuang Liu, Si-Bao Chen, Qing-Ling Shu, Chris Ding, Jin Tang, Bin Luo
Computer Vision
Fangke Chen, Tianhao Dong, Sirry Chen et al. (+3 more)
Handwritten word
retrieval is vital for digital archives but remains challenging due to large handwriting variability and cross-lingual semantic gaps. While large
vision-
language models offer potential solutions, their prohibitive computational costs hinder practical edge deployment. To address this, we propose a lightweight asymmetric dual-encoder framework that learns unified, style-invariant visual
embeddings. By jointly optimizing instance-level
alignment and class-level semantic consistency, our approach anchors visual
embeddings to
language-agnostic semantic prototypes, enforcing invariance across scripts and writing styles. Experiments show that our method outperforms 28 baselines and achieves state-of-the-art accuracy on within-
language retrieval benchmarks. We further conduct explicit cross-lingual
retrieval, where the query
language differs from the target
language, to validate the effectiveness of the learned cross-lingual representations. Achieving strong performance with only a fraction of the parameters required by existing models, our framework enables accurate and resource-efficient cross-script handwriting
retrieval.
Read the full abstract ▾
All authors Fangke Chen, Tianhao Dong, Sirry Chen, Guobin Zhang, Yishu Zhang, Yining Chen
Computer Vision
Image-Text Knowledge Modeling for Unsupervised Multi-Scenario Person Re-Identification
Zhiqi Pang, Lingling Zhao, Yang Liu et al. (+2 more)
We propose unsupervised multi-scenario (UMS) person re-identification (ReID) as a new task that expands ReID across diverse scenarios (cross-resolution, clothing change, etc.) within a single coherent framework. To tackle UMS-ReID, we introduce image-text knowledge modeling (ITKM) – a three-stage framework that effectively exploits the representational power of
vision-
language models. We start with a pre-trained CLIP model with an image encoder and a text encoder. In Stage I, we introduce a scenario
embedding in the image encoder and fine-tune the encoder to adaptively leverage knowledge from multiple scenarios. In Stage II, we optimize a set of learned text
embeddings to associate with pseudo-labels from Stage I and introduce a multi-scenario separation loss to increase the divergence between inter-scenario text representations. In Stage III, we first introduce cluster-level and instance-level heterogeneous matching modules to obtain reliable heterogeneous positive pairs (e.g., a visible image and an infrared image of the same person) within each scenario. Next, we propose a dynamic text representation update strategy to maintain consistency between text and image supervision signals. Experimental results across multiple scenarios demonstrate the superiority and generalizability of ITKM; it not only outperforms existing scenario-specific methods but also enhances overall performance by integrating knowledge from multiple scenarios.
Read the full abstract ▾
All authors Zhiqi Pang, Lingling Zhao, Yang Liu, Chunyu Wang, Gaurav Sharma
Computer Vision
Bio-inspired fine-tuning for selective transfer learning in image classification
Ana Davila, Jacinto Colan and Yasuhisa Hasegawa
Deep learning has significantly advanced image analysis across diverse domains but often depends on large, annotated
datasets for success. Transfer learning addresses this challenge by utilizing pre-trained models to tackle new tasks with limited labeled data. However, discrepancies between source and target domains can hinder effective transfer learning. We introduce BioTune, a novel adaptive fine-tuning technique utilizing evolutionary optimization. BioTune enhances transfer learning by optimally choosing which layers to freeze and adjusting learning rates for unfrozen layers. Through extensive evaluation on nine image classification
datasets, spanning natural and specialized domains such as medical imaging, BioTune demonstrates superior accuracy and efficiency over state-of-the-art fine-tuning methods, including AutoRGN and LoRA, highlighting its adaptability to various data characteristics and distribution changes. Additionally, BioTune consistently achieves top performance across four different CNN architectures, underscoring its flexibility. Ablation studies provide valuable insights into the impact of BioTune's key components on overall performance. The source code is available at https://github.com/davilac/BioTune.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CR
VidLeaks: Membership Inference Attacks Against Text-to-Video Models
Li Wang, Wenyu Chen, Ning Yu et al. (+2 more)
The proliferation of powerful Text-to-
Video (T2V) models, trained on massive web-scale
datasets, raises urgent concerns about copyright and
privacy violations. Membership inference attacks (MIAs) provide a principled tool for
auditing such risks, yet existing techniques - designed for static data like images or text - fail to capture the spatio-temporal complexities of
video generation. In particular, they overlook the sparsity of memorization signals in keyframes and the instability introduced by
stochastic temporal dynamics. In this paper, we conduct the first systematic study of MIAs against T2V models and introduce a novel framework VidLeaks, which probes sparse-temporal memorization through two complementary signals: 1) Spatial Reconstruction Fidelity (SRF), using a Top-K similarity to amplify spatial memorization signals from sparsely memorized keyframes, and 2) Temporal Generative Stability (TGS), which measures semantic consistency across multiple queries to capture temporal leakage. We evaluate VidLeaks under three progressively restrictive black-box settings - supervised, reference-based, and query-only. Experiments on three representative T2V models reveal severe vulnerabilities: VidLeaks achieves AUC of 82.92% on AnimateDiff and 97.01% on InstructVideo even in the strict query-only setting, posing a realistic and exploitable
privacy risk. Our work provides the first concrete evidence that T2V models leak substantial membership information through both sparse and temporal memorization, establishing a foundation for
auditing video generation systems and motivating the development of new defenses. Code is available at: https://zenodo.org/records/17972831.
Read the full abstract ▾
All authors Li Wang, Wenyu Chen, Ning Yu, Zheng Li, Shanqing Guo
Computer Vision
ATATA: One Algorithm to Align Them All
Boyi Pang, Savva Ignatyev, Vladimir Ippolitov et al. (+8 more)
We suggest a new multi-modal
algorithm for joint inference of paired structurally aligned samples with Rectified Flow models. While some existing methods propose a codependent generation process, they do not view the problem of joint generation from a structural
alignment perspective. Recent work uses Score
Distillation Sampling to generate aligned 3D models, but SDS is known to be time-consuming, prone to mode collapse, and often provides cartoonish results. By contrast, our suggested approach relies on the joint transport of a segment in the sample space, yielding faster computation at inference time. Our approach can be built on top of an arbitrary Rectified Flow model operating on the structured latent space. We show the applicability of our method to the domains of image,
video, and 3D shape generation using state-of-the-art baselines and evaluate it against both editing-based and joint inference-based competing approaches. We demonstrate a high degree of structural
alignment for the sample pairs obtained with our method and a high visual quality of the samples. Our method improves the state-of-the-art for image and
video generation pipelines. For 3D generation, it is able to show comparable quality while working orders of magnitude faster.
Read the full abstract ▾
All authors Boyi Pang, Savva Ignatyev, Vladimir Ippolitov, Ramil Khafizov, Yurii Melnik, Oleg Voynov, Maksim Nakhodnov, Aibek Alanov, Xiaopeng Fan, Peter Wonka, Evgeny Burnaev
Computer Vision
Democratizing planetary-scale analysis: An ultra-lightweight Earth embedding database for accurate and flexible global land monitoring
Shuang Chen, Jie Wang, Shuai Yuan et al. (+15 more)
The rapid
evolution of satellite-borne Earth Observation (EO) systems has revolutionized terrestrial monitoring, yielding petabyte-scale archives. However, the immense computational and storage requirements for global-scale analysis often preclude widespread use, hindering planetary-scale studies. To address these barriers, we present Embedded Seamless Data (ESD), an ultra-lightweight, 30-m global Earth
embedding database spanning the 25-year period from 2000 to 2024. By transforming high-dimensional, multi-sensor observations from the Landsat series (5, 7, 8, and 9) and MODIS Terra into information-dense, quantized latent vectors, ESD distills essential geophysical and semantic features into a unified latent space. Utilizing the ESDNet architecture and Finite Scalar Quantization (FSQ), the
dataset achieves a transformative ~340-fold reduction in data volume compared to raw archives. This compression allows the entire global land surface for a single year to be encapsulated within approximately 2.4 TB, enabling decadal-scale global analysis on standard local workstations. Rigorous validation demonstrates high reconstructive fidelity (MAE: 0.0130; RMSE: 0.0179; CC: 0.8543). By condensing the annual phenological cycle into 12 temporal steps, the
embeddings provide inherent denoising and a semantically organized space that outperforms raw reflectance in land-cover classification, achieving 79.74% accuracy (vs. 76.92% for raw fusion). With robust few-shot learning capabilities and longitudinal consistency, ESD provides a versatile foundation for democratizing planetary-scale research and advancing next-generation geospatial artificial intelligence.
Read the full abstract ▾
All authors Shuang Chen, Jie Wang, Shuai Yuan, Jiayang Li, Yu Xia, Yuanhong Liao, Junbo Wei, Jincheng Yuan, Xiaoqing Xu, Xiaolin Zhu, Peng Zhu, Hongsheng Zhang, Yuyu Zhou, Haohuan Fu, Huabing Huang, Bin Chen, Fan Dai, Peng Gong
Computer Vision
SoLA-Vision: Fine-grained Layer-wise Linear Softmax Hybrid Attention
Ruibang Li, Guan Luo, Yiwei Zhang et al. (+3 more)
Standard softmax self-
attention excels in
vision tasks but incurs quadratic complexity O(N^2), limiting high-resolution deployment. Linear
attention reduces the cost to O(N), yet its compressed state representations can impair modeling capacity and accuracy. We present an analytical study that contrasts linear and softmax
attention for visual representation learning from a layer-stacking perspective. We further conduct systematic experiments on layer-wise hybridization patterns of linear and softmax
attention. Our results show that, compared with rigid intra-block hybrid designs, fine-grained layer-wise hybridization can match or surpass performance while requiring fewer softmax layers. Building on these findings, we propose SoLA-
Vision (Softmax-Linear
Attention Vision), a flexible layer-wise hybrid
attention backbone that enables fine-grained control over how linear and softmax
attention are integrated. By strategically inserting a small number of global softmax layers, SoLA-
Vision achieves a strong trade-off between accuracy and computational cost. On ImageNet-1K, SoLA-
Vision outperforms purely linear and other hybrid
attention models. On dense prediction tasks, it consistently surpasses strong baselines by a considerable margin. Code will be released.
Read the full abstract ▾
All authors Ruibang Li, Guan Luo, Yiwei Zhang, Jin Gao, Bing Li, Weiming Hu
Computer Vision · Cross-listed from cs.LG
GMM-COMET: Continual Source-Free Universal Domain Adaptation via a Mean Teacher and Gaussian Mixture Model-Based Pseudo-Labeling
Pascal Schlachter and Bin Yang
Unsupervised domain adaptation tackles the problem that domain shifts between training and test data impair the performance of neural networks in many real-world applications. Thereby, in realistic scenarios, the source data may no longer be available during adaptation, and the label space of the target domain may differ from the source label space. This setting, known as source-free universal domain adaptation (SF-UniDA), has recently gained
attention, but all existing approaches only assume a single domain shift from source to target. In this work, we present the first study on continual SF-UniDA, where the model must adapt sequentially to a stream of multiple different unlabeled target domains. Building upon our previous methods for online SF-UniDA, we combine their key ideas by integrating Gaussian mixture model-based pseudo-labeling within a mean teacher framework for improved stability over long adaptation sequences. Additionally, we introduce consistency losses for further robustness. The resulting method GMM-COMET provides a strong first baseline for continual SF-UniDA and is the only approach in our experiments to consistently improve upon the source-only model across all evaluated scenarios. Our code is available at https://github.com/pascalschlachter/GMM-COMET.
Read the full abstract ▾
Computer Vision
Shaofeng Yin, Jiaxin Ge, Zora Zhiruo Wang et al. (+6 more)
Vision-as-inverse-graphics, the concept of reconstructing images into editable programs, remains challenging for
Vision-
Language Models (VLMs), which inherently lack fine-grained spatial grounding in one-shot settings. To address this, we introduce VIGA (
Vision-as-Inverse-Graphics
Agent), an interleaved
multimodal reasoning framework where symbolic logic and visual perception actively cross-verify each other. VIGA operates through a tightly coupled code-render-inspect loop: synthesizing symbolic programs, projecting them into visual states, and inspecting discrepancies to guide iterative edits. Equipped with high-level semantic skills and an evolving
multimodal memory, VIGA sustains evidence-based modifications over long horizons. This training-free, task-agnostic framework seamlessly supports 2D document generation, 3D reconstruction, multi-step 3D editing, and 4D physical interaction. Finally, we introduce BlenderBench, a challenging visual-to-code
benchmark. Empirically, VIGA substantially improves accuracy compared with one-shot baselines in BlenderGym (35.32%), SlideBench (117.17%) and our proposed BlenderBench (124.70%).
Read the full abstract ▾
All authors Shaofeng Yin, Jiaxin Ge, Zora Zhiruo Wang, Chenyang Wang, Xiuyu Li, Michael J. Black, Trevor Darrell, Angjoo Kanazawa, Haiwen Feng
Computer Vision · Cross-listed from q-bio.NC
Simple Models, Rich Representations: Visual Decoding from Primate Intracortical Neural Signals
Matteo Ciferri, Matteo Ferrante and Nicola Toschi
Understanding how neural activity gives rise to perception is a central challenge in neuroscience. We address the problem of decoding visual information from high-density intracortical recordings in primates, using the THINGS Ventral Stream Spiking
Dataset. We systematically evaluate the effects of model architecture, training objectives, and data scaling on decoding performance. Results show that decoding accuracy is mainly driven by modeling temporal dynamics in neural signals, rather than architectural complexity. A simple model combining temporal
attention with a shallow MLP achieves up to 70% top-1 image
retrieval accuracy, outperforming linear baselines as well as recurrent and convolutional approaches. Scaling analyses reveal predictable diminishing returns with increasing input dimensionality and
dataset size. Building on these findings, we design a modular generative decoding pipeline that combines low-resolution latent reconstruction with semantically conditioned
diffusion, generating plausible images from 200 ms of
brain activity. This framework provides principles for
brain-computer interfaces and semantic neural decoding.
Read the full abstract ▾
Computer Vision
Graph Smoothing for Enhanced Local Geometry Learning in Point Cloud Analysis
Shangbo Yuan, Jie Xu, Ping Hu et al. (+2 more)
Graph-based methods have proven to be effective in capturing relationships among points for 3D point cloud analysis. However, these methods often suffer from suboptimal graph structures, particularly due to sparse connections at boundary points and noisy connections in junction areas. To address these challenges, we propose a novel method that integrates a graph smoothing module with an enhanced local geometry learning module. Specifically, we identify the limitations of conventional graph structures, particularly in handling boundary points and junction areas. In response, we introduce a graph smoothing module designed to optimize the graph structure and minimize the negative impact of unreliable sparse and noisy connections. Based on the optimized graph structure, we improve the feature extract function with local geometry information. These include shape features derived from adaptive
geometric descriptors based on eigenvectors and distribution features obtained through cylindrical coordinate transformation. Experimental results on real-world
datasets validate the effectiveness of our method in various point cloud learning tasks, i.e., classification, part
segmentation, and semantic
segmentation.
Read the full abstract ▾
All authors Shangbo Yuan, Jie Xu, Ping Hu, Xiaofeng Zhu, Na Zhao
Computer Vision
CoDance: An Unbind-Rebind Paradigm for Robust Multi-Subject Animation
Shuai Tan, Biao Gong, Ke Ma et al. (+5 more)
Character image animation is gaining significant importance across various domains, driven by the demand for robust and flexible multi-subject rendering. While existing methods excel in single-person animation, they struggle to handle arbitrary subject counts, diverse character types, and spatial misalignment between the reference image and the driving poses. We attribute these limitations to an overly rigid spatial binding that forces strict pixel-wise
alignment between the pose and reference, and an inability to consistently rebind motion to intended subjects. To address these challenges, we propose CoDance, a novel Unbind-Rebind framework that enables the animation of arbitrary subject counts, types, and spatial configurations conditioned on a single, potentially misaligned pose sequence. Specifically, the Unbind module employs a novel pose shift encoder to break the rigid spatial binding between the pose and the reference by introducing
stochastic perturbations to both poses and their latent features, thereby compelling the model to learn a location-agnostic motion representation. To ensure precise control and subject association, we then devise a Rebind module, leveraging semantic guidance from text
prompts and spatial guidance from subject masks to direct the learned motion to intended characters. Furthermore, to facilitate comprehensive evaluation, we introduce a new multi-subject CoDanceBench. Extensive experiments on CoDanceBench and existing
datasets show that CoDance achieves SOTA performance, exhibiting remarkable generalization across diverse subjects and spatial layouts. The code and weights will be open-sourced.
Read the full abstract ▾
All authors Shuai Tan, Biao Gong, Ke Ma, Yutong Feng, Qiyuan Zhang, Yan Wang, Yujun Shen, Hengshuang Zhao
Computer Vision
PhysRVG: Physics-Aware Unified Reinforcement Learning for Video Generative Models
Qiyuan Zhang, Biao Gong, Shuai Tan et al. (+7 more)
Physical principles are fundamental to realistic visual simulation, but remain a significant oversight in
transformer-based
video generation. This gap highlights a critical limitation in rendering rigid body motion, a core tenet of classical mechanics. While computer graphics and physics-based simulators can easily model such collisions using Newton formulas, modern pretrain-finetune paradigms discard the concept of object
rigidity during pixel-level global denoising. Even perfectly correct mathematical constraints are treated as suboptimal solutions (i.e., conditions) during model optimization in post-training, fundamentally limiting the physical realism of generated
videos. Motivated by these considerations, we introduce, for the first time, a physics-aware reinforcement learning paradigm for
video generation models that enforces physical collision rules directly in high-dimensional spaces, ensuring the physics knowledge is strictly applied rather than treated as conditions. Subsequently, we extend this paradigm to a unified framework, termed Mimicry-Discovery Cycle (MDcycle), which allows substantial fine-tuning while fully preserving the model's ability to leverage physics-grounded feedback. To validate our approach, we construct new
benchmark PhysRVGBench and perform extensive qualitative and quantitative experiments to thoroughly assess its effectiveness.
Read the full abstract ▾
All authors Qiyuan Zhang, Biao Gong, Shuai Tan, Zheng Zhang, Yujun Shen, Xing Zhu, Yuyuan Li, Kelu Yao, Chunhua Shen, Changqing Zou
Computer Vision · Cross-listed from eess.IV
Generation of Chest CT pulmonary Nodule Images by Latent Diffusion Models using the LIDC-IDRI Dataset
Kaito Urata, Maiko Nagao, Atsushi Teramoto et al. (+3 more)
Recently, computer-aided diagnosis systems have been developed to support diagnosis, but their performance depends heavily on the quality and quantity of training data. However, in clinical practice, it is difficult to collect the large amount of CT images for specific cases, such as small
cell carcinoma with low epidemiological incidence or benign tumors that are difficult to distinguish from malignant ones. This leads to the challenge of data imbalance. In this study, to address this issue, we proposed a method to automatically generate chest CT nodule images that capture target features using latent
diffusion models (LDM) and verified its effectiveness. Using the LIDC-IDRI
dataset, we created pairs of nodule images and finding-based text
prompts based on physician evaluations. For the image generation models, we used Stable
Diffusion version 1.5 (SDv1) and 2.0 (SDv2), which are types of LDM. Each model was fine-tuned using the created
dataset. During the generation process, we adjusted the guidance scale (GS), which indicates the fidelity to the input text. Both quantitative and subjective evaluations showed that SDv2 (GS = 5) achieved the best performance in terms of image quality, diversity, and text consistency. In the subjective evaluation, no statistically significant differences were observed between the generated images and real images, confirming that the quality was equivalent to real clinical images. We proposed a method for generating chest CT nodule images based on input text using LDM. Evaluation results demonstrated that the proposed method could generate high-quality images that successfully capture specific medical features.
Read the full abstract ▾
All authors Kaito Urata, Maiko Nagao, Atsushi Teramoto, Kazuyoshi Imaizumi, Masashi Kondo, Hiroshi Fujita
Computer Vision · Cross-listed from eess.IV
Visual question answering-based image-finding generation for pulmonary nodules on chest CT from structured annotations
Maiko Nagao, Kaito Urata, Atsushi Teramoto et al. (+3 more)
Interpretation of imaging findings based on morphological characteristics is important for diagnosing pulmonary nodules on chest computed
tomography (CT) images. In this study, we constructed a visual question answering (VQA)
dataset from structured data in an open
dataset and investigated an image-finding generation method for chest CT images, with the aim of enabling interactive diagnostic support that presents findings based on questions that reflect physicians' interests rather than fixed descriptions. In this study, chest CT images included in the Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI)
datasets were used. Regions of interest surrounding the pulmonary nodules were extracted from these images, and image findings and questions were defined based on morphological characteristics recorded in the database. A
dataset comprising pairs of cropped images, corresponding questions, and image findings was constructed, and the VQA model was fine-tuned on it.
Language evaluation metrics such as BLEU were used to evaluate the generated image findings. The VQA
dataset constructed using the proposed method contained image findings with natural expressions as radiological descriptions. In addition, the generated image findings showed a high CIDEr score of 3.896, and a high agreement with the reference findings was obtained through evaluation based on morphological characteristics. We constructed a VQA
dataset for chest CT images using structured information on the morphological characteristics from the LIDC-IDRI
dataset. Methods for generating image findings in response to these questions have also been investigated. Based on the generated results and evaluation metric scores, the proposed method was effective as an interactive diagnostic support system that can present image findings according to physicians' interests.
Read the full abstract ▾
All authors Maiko Nagao, Kaito Urata, Atsushi Teramoto, Kazuyoshi Imaizumi, Masashi Kondo, Hiroshi Fujita
Computer Vision · Cross-listed from cs.RO
EmboTeam: Asymmetric State Grounding for Heterogeneous Multi-Robot Collaboration
Haishan Zeng, Mengna Wang and Peng Li
Heterogeneous multi-
robot collaboration over long horizons requires both global task planning and robust coordination during execution. Existing plan- ners using large
language models (LLMs) can generate task decompositions or symbolic plans, but often execute them as rigid action sequences, making them fragile to local state mismatches and synchronization failures. We propose Em- boTeam, a neuro-symbolic framework for asymmetric state grounding. An LLM generates structured state expectations, which are verified and coordinated through symbolic search using Planning Domain Definition
Language (PDDL) and then grounded into a team-level parallel behavior tree.
Robot-specific subtrees per- form precondition checking, skill execution, effect validation, local replanning, and blackboard-based synchronization. We further introduce MACE-THOR, con- taining 42 heterogeneous multi-
robot tasks across 8 indoor layouts. EmboTeam improves task success rate from 12% to 55% and goal condition recall from 32% to 75% over LaMMA-P, demonstrating the effectiveness of synchronized reactive grounding for collaboration over long horizons.
Read the full abstract ▾
Computer Vision
M3DDM+: An improved video outpainting by a modified masking strategy
Takuya Murakawa, Takumi Fukuzawa, Ning Ding et al. (+1 more)
M3DDM provides a computationally efficient framework for
video outpainting via latent
diffusion modeling. However, it exhibits significant quality degradation – manifested as spatial blur and temporal inconsistency – under challenging scenarios characterized by limited camera motion or large outpainting regions, where inter-frame information is limited. We identify the cause as a training-inference mismatch in the masking strategy: M3DDM's training applies random mask directions and widths across frames, whereas inference requires consistent directional outpainting throughout the
video. To address this, we propose M3DDM+, which applies uniform mask direction and width across all frames during training, followed by fine-tuning of the pretrained M3DDM model. Experiments demonstrate that M3DDM+ substantially improves visual fidelity and temporal coherence in information-limited scenarios while maintaining computational efficiency. The code is available at https://github.com/tamaki-lab/M3DDM-Plus.
Read the full abstract ▾
All authors Takuya Murakawa, Takumi Fukuzawa, Ning Ding, Toru Tamaki
Computer Vision · Cross-listed from eess.IV
Convolutions Need Registers Too: HVS-Inspired Dynamic Attention for Video Quality Assessment
Mayesha Maliha R. Mithila and Mylene C.Q. Farias
No-reference
video quality assessment (NR-VQA) estimates perceptual quality without a reference
video, which is often challenging. While recent techniques leverage saliency or
transformer attention, they merely address global context of the
video signal by using static maps as auxiliary inputs rather than
embedding context fundamentally within feature extraction of the
video sequence. We present Dynamic
Attention with Global Registers for
Video Quality Assessment (DAGR-VQA), the first framework integrating register-
token directly into a convolutional backbone for spatio-temporal, dynamic saliency prediction. By
embedding learnable register
tokens as global context carriers, our model enables dynamic, HVS-inspired
attention, producing temporally adaptive saliency maps that track salient regions over time without explicit motion estimation. Our model integrates dynamic saliency maps with RGB inputs, capturing spatial data and analyzing it through a temporal
transformer to deliver a perceptually consistent
video quality assessment. Comprehensive tests conducted on the LSVQ, KonVid-1k, LIVE-VQC, and YouTube-UGC
datasets show that the performance is highly competitive, surpassing the majority of top baselines. Research on ablation studies demonstrates that the integration of register
tokens promotes the development of stable and temporally consistent
attention mechanisms. Achieving an efficiency of 387.7 FPS at 1080p, DAGR-VQA demonstrates computational performance suitable for real-time applications like multimedia streaming systems.
Read the full abstract ▾
Computer Vision
Your One-Stop Solution for AI-Generated Video Detection
Long Ma, Zihao Xue, Yan Wang et al. (+6 more)
Recent advances in generative modeling can create remarkably realistic synthetic
videos, making it increasingly difficult for humans to distinguish them from real ones and necessitating reliable detection methods. However, two key limitations hinder the development of this field. From the
dataset perspective, existing
datasets are often limited in scale and constructed using outdated or narrowly scoped generative models, making it difficult to capture the diversity and rapid
evolution of modern generative techniques. Moreover, the
dataset construction process frequently prioritizes quantity over quality, neglecting essential aspects such as semantic diversity, scenario coverage, and technological representativeness. From the
benchmark perspective, current
benchmarks largely remain at the stage of
dataset creation, leaving many fundamental issues and in-depth analysis yet to be systematically explored. Addressing this gap, we propose AIGVDBench, a
benchmark designed to be comprehensive and representative, covering 31 state-of-the-art generation models and over 440,000
videos. By executing more than 1,500 evaluations on 33 existing detectors belonging to four distinct categories. This work presents 8 in-depth analyses from multiple perspectives and identifies 4 novel findings that offer valuable insights for future research. We hope this work provides a solid foundation for advancing the field of AI-generated
video detection. Our
benchmark is open-sourced at https://github.com/LongMa-2025/AIGVDBench.
Read the full abstract ▾
All authors Long Ma, Zihao Xue, Yan Wang, Zhiyuan Yan, Jin Xu, Xiaorui Jiang, Haiyang Yu, Yong Liao, Zhen Bi