Computer Vision · Cross-listed from cs.AI
Linguistic Context Recodes Visual Representations in Vision-Language Models
Brian Song, Michael A. Lepori and Ellie Pavlick
Goal-directed visual processing is a hallmark of human visual intelligence, resulting in representations that support downstream tasks such as categorization or search. Though
vision-
language models (VLMs) are often faced with these same tasks, their ability to recode visual representations when presented with goal-directed
language remains poorly characterized. Indeed, prior work largely treats visual representations in VLMs as static repositories of visual information that are manipulated by
language representations. In the present work, we provide evidence for two concrete instances of
language-induced recoding of visual representations. First, we identify an abstract reference representation that denotes which objects are goal-relevant under a natural
language prompt. We extract contrastive steering vectors corresponding to this reference representation and demonstrate that they are causally implicated in model predictions. These reference representations are abstract in that they generalize to different objects, different task contexts, and even from synthetic to naturalistic images. Second, we demonstrate
language-induced attribute modulation: later layers selectively amplify goal-relevant attributes in visual representations of objects. We demonstrate this phenomenon across a range of different
prompts. Finally, we provide a causal intervention that demonstrates that attribute modulation mediates a VLM's response distribution. Together, our results support a more dynamic account of cross-modality processing in VLMs – rather than
vision tokens serving as static repositories of information, they are modulated to support queries articulated in
language.
Read the full abstract ▾
Computer Vision
An Interactive Vision Language Platform for Cognitive Remediation in Schizophrenia
Nassira Ait Mehdi, Milissa Temmam and Slimane Larabi
Cognitive remediation tasks often require patients to perform structured actions involving object manipulation and sequential
reasoning. For patients diagnosed with schizophrenia, these tasks are crucial for addressing severe cognitive deficits. However, evaluating the correctness of these physical actions generally relies on manual observation by clinicians, which introduces subjectivity and limits the scalability of therapeutic interventions. In this paper, we propose an automated framework based on
Vision-
Language Models for action verification in cognitive remediation tasks tailored for schizophrenia rehabilitation. The proposed system relies on a camera-monitored tabletop environment composed of structured miniature scenes including roads, a roundabout, a park, and toy vehicles. Patients receive audio instructions describing goal-oriented spatial actions to perform by manipulating a toy vehicle. These interactive physical activities are specifically designed to stimulate targeted cognitive functions, such as sustained
attention, motor coordination, spatial
navigation, and cognitive flexibility. To verify the correctness of the performed actions without requiring continuous clinical oversight, the system analyzes the
video feed tracking the patient's hand and toy movements. A fine-tuned
Vision-
Language Model interprets the recorded
video sequences and generates semantic descriptions of the observed activities, enabling high-level verification of the executed actions with respect to the initial textual instructions. A dedicated
dataset of 4634 tabletop cognitive remediation
video scenarios was collected to evaluate the proposed approach. Experimental results demonstrate that our specialized framework effectively bridges low-level physical telemetry with high-level clinical feedback, presenting a scalable and objective solution for advanced cognitive rehabilitation.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Kiran Nair, Smriti Regmi and Rodrigue Rizk
Existing adaptive inference methods for Large
Language Models rely on observational heuristics, such as hidden-state similarity or activation magnitudes, to drop redundant modules. However, these correlation-based metrics often fail to capture subtle, non-linear structural computations vital for semantic accuracy. We introduce CausalGate, an intervention-guided framework for compute-efficient
transformer inference. During a calibration phase, CausalGate isolates individual
Attention and MLP sub-layers, zeros out their respective outputs, and measures the exact semantic damage via the Kullback-Leibler divergence of the final logit distribution. To eliminate runtime routing overhead, this structural importance hierarchy is distilled into a global set of static, lightweight scalar gates using an Exponential Moving Average smoothing objective paired with a differentiable pairwise ranking loss. Evaluated on TinyLlama-1.1B, Qwen2.5-3B, and Llama-3.1-8B across
language modeling and commonsense
reasoning benchmarks, CausalGate consistently outperforms prominent dynamic routing and layer-skipping baselines, translating theoretical compute savings into concrete hardware latency reductions with zero operational overhead.
Read the full abstract ▾
Computer Vision
$\gamma$-Bridge: A Look-Parametric Diffusion Bridge
Xuran Hu, Yujie Zhu, Tengxi Wang et al. (+2 more)
Multiplicative Gamma noise is a signal-dependent degradation in coherent imaging; synthetic aperture radar (SAR) despeckling is its most prominent real-world instance. Existing
diffusion denoisers parameterize their forward process by abstract signal-to-noise schedules rather than by the physical look number $L$, so different deployment scenarios typically require separately trained models, and transfer from synthetic Gamma training to real SAR remains challenging without clean ground truth. We introduce $\gamma$-Bridge, a look-parametric bridge whose schedule $L(t)$ connects the noisy observation at $L_{obs}$ to the clean limit through exact multiplicative Gamma marginals. Its closed-form Gamma–Lévy reverse posterior admits both
stochastic and deterministic processes, while observation conditioning and a two-step consistency loss stabilize multi-step inference in the low-SNR single-look regime. Because bridge time directly represents $L$, one conditioned network can smart-start from any admissible input look and stop at a target look number. These two orthogonal controls enable zero-shot restoration over the full admissible grid after training only at $L_{obs} = 1$ on natural images with synthetic Gamma corruption. Combined with a homogeneous-patch look
estimator, $\gamma$-Bridge processes data from six spaceborne and airborne SAR sensors without sensor-specific fine-tuning, achieving leading results on standard synthetic
benchmarks while providing physically interpretable input and output controls absent from prior denoisers. Codes are released \href{https://github.com/Teriri1999/GammaBridge}{here}.
Read the full abstract ▾
All authors Xuran Hu, Yujie Zhu, Tengxi Wang, Jilong Li, Wufan Zhao
Computer Vision
DAMamba-UNet3D: A Parameter-Efficient Mamba State Space U-Net with Dynamic Adaptive Scan for 3D Medical Image Segmentation
Mohammad Arafat Hussain, Ellen Grant and Yangming Ou
We propose parameter-efficient SSM-based U-Net architectures for 3D medical image
segmentation. Convolutional U-Nets afford O(n) local mixing per layer but lack explicit global context;
transformers provide global
reasoning at O(n^2) cost in sequence length $n$. State-space models (SSMs), such as Mamba, offer $O(n)$ global propagation per block. Yet, existing medical SSM segmenters rely on fixed scan patterns and large parameter budgets. Dynamic Adaptive Scan (DAS), which learns data-dependent reordering before selective scan, has not been applied to medical imaging or extended to 3D volumes. We propose DAMamba-UNet3D, a hybrid encoder-decoder that integrates tri-plane 3D-DAS blocks at encoder stages E2-E4 while retaining convolutions elsewhere (~5.3M parameters). On BraTS 2020 five-fold cross-validation, DAMamba-UNet3D achieves mean Dice 0.815+/-0.013 (full-volume per-case evaluation) at ~13x lower parameter cost than SegMamba (0.824+\-0.014, ~70M). At comparable scale, DAMamba-L (~70M), a wide DAS-native variant with encoder-only DAMamba and a convolutional bottleneck, reaches 0.829+\-0.012, surpassing retrained SegMamba by 0.5pt. Component ablations show that encoder-only DAS placement is critical as bottleneck and decoder SSM blocks lower Dice. Together, the results suggest that learned tri-plane DAS in a hybrid U-Net is competitive with, and under our large-scale design may improve upon, SegMamba's fixed Tri-orientated Mamba (ToM) scanning on BraTS 2020. Code: https://github.com/marafathussain/DAMamba-UNet3D.
Read the full abstract ▾
Computer Vision
TOM-GS: Editable Video Representation via Temporal Opacity Modulation of Static 3D Gaussians
Marek Lisowski, Łukasz Smoliński, Kornel Howil et al. (+3 more)
While Implicit Neural Representations (INRs) and dynamic 3D Gaussian Splatting (3DGS) achieve impressive results in
video processing, they often fall short of producing representations that are easily editable. Recent methods address this by introducing complex spatial deformations or folded distributions, which constrain optimization and reduce flexibility for downstream editing. In this paper, we introduce TOM-GS, an editable
video representation that forgoes complex deformations in favor of regular 3D Gaussians equipped with a continuous temporal opacity formulation. By assigning a learnable temporal mean and scale to the opacity of each Gaussian, our model enables static 3D spatial components to fade smoothly in and out of the scene. Grounded by robust, off-the-shelf pose estimation, our approach maintains a static spatial geometry that naturally supports a wide range of manual and physics-based edits. TOM-GS outperforms prior editable
video representations in visual fidelity, while its reliance on standard 3D Gaussians ensures seamless compatibility with established 3D editing tools.
Read the full abstract ▾
All authors Marek Lisowski, Łukasz Smoliński, Kornel Howil, Piotr Biliński, Marcin Mazur, Przemysław Spurek
Computer Vision
Visual Token Compression Enhances Robustness of MLLMs
Shishen Gu, Jiequan Cui, Wenbo Hu et al. (+3 more)
In this paper, we show for the first time that visual
token pruning enhances the robustness of
Multimodal Large
Language Models (MLLMs), mitigating vulnerabilities such as jailbreak attacks and
hallucinations. Given that
vision and
language modalities cannot be perfectly aligned, the misaligned visual
tokens might act as out-of-distribution (OOD) inputs, leading to unpredictable outputs and introducing potential vulnerabilities. Building on this insight, we aim to enhance model robustness against jailbreaks and
hallucinations by reducing OOD visual
tokens at robust-
pruning layers, while also reducing inference cost as a side benefit. Specifically, we measure the distance between each visual
token and the
language feature space. Then, visual
tokens with large distances are identified as OOD
tokens, which can be iteratively pruned. To demonstrate the effectiveness of our method, we evaluate it on seven diverse popular
benchmarks. Notably, our method yields an average improvement of 13.29% in defending jailbreak attacks, consistently achieves competitive performance in mitigating
hallucinations, and maintains strong results on general
datasets like MME.
Read the full abstract ▾
All authors Shishen Gu, Jiequan Cui, Wenbo Hu, Zenglin Shi, Zhenzhen Hu, Richang Hong
Computer Vision
Child-Oriented AIGC Video Risk Reviewing: A Benchmark and Knowledge-Supported Iterative Reasoning Framework
Lewen Mi, Manyi Li, Yuling Sun et al. (+5 more)
The rapid growth of Artificial Intelligence-generated content (AIGC) is reshaping
video production and circulation, exposing children to an increasing volume of AIGC
videos. Unlike traditionally produced
videos, AIGC
videos often exhibit greater uncertainty in visual details, narrative coherence, and content expression, which may introduce developmentally inappropriate risks for children. However, existing
video safety research is largely designed for general violation detection from an adult perspective and remains insufficient for identifying the fine-grained, implicit, and context-dependent risks that children may encounter when viewing AIGC
videos. To address this gap, we study child-oriented AIGC
video reviewing, making three contributions. First, we construct CAVSR, a
benchmark of 605 real-world
videos collected from multiple platforms, and develop a hierarchical risk taxonomy comprising 6 top-level categories and 26 fine-grained labels to support systematic evaluation of children's viewing risks. Second, we propose QVRS-E, a knowledge- and experience-augmented
video reviewing framework that combines multi-
agent collaboration with expert and experiential knowledge to support targeted evidence acquisition and fact-grounded reviewing decisions. Third, extensive experiments demonstrate that our method significantly enhances the reviewing of child-related risks integrated with
vision-
language models, and yields more robust review reports.
Read the full abstract ▾
All authors Lewen Mi, Manyi Li, Yuling Sun, Yufan Zhang, Yuxin Shi, Yulong Bian, Xiangxian Li, Juan Liu
Computer Vision
Real-Time Semantic Segmentation with Optimized RetinaNet Architectures for Embedded Automotive Systems
Sai Sidharth D
Real-time perception is a foundational requirement for advanced driver assistance systems (ADAS) and autonomous vehicles, yet embedded automotive platforms impose severe constraints on compute,
memory, and power. This paper presents an optimized semantic
segmentation architecture derived from the RetinaNet detection framework, adapted for dense pixel-wise prediction and tailored for deployment on resource-constrained embedded hardware. The proposed architecture, termed Opt-RetinaSeg, replaces the standard ResNet-50 backbone with a hybrid lightweight feature extractor, restructures the Feature Pyramid Network (FPN) to reduce redundant multi-scale computation, and introduces a compact
segmentation head guided by focal-loss-inspired class balancing to address the severe foreground-background imbalance common in road scenes. We further apply a three-stage optimization pipeline consisting of structured channel
pruning, post-training INT8 quantization, and knowledge
distillation from a high-capacity teacher network. Evaluated on the Cityscapes and BDD100K
datasets and deployed on an NVIDIA Jetson Xavier NX and a Qualcomm QCS610 automotive SoC, the proposed model achieves 73.9% mIoU at 70.4 FPS, representing a 7.4x inference speedup and a 4x reduction in model size relative to the ResNet-50 baseline, with less than 3% accuracy degradation. These results indicate that RetinaNet-derived architectures, when systematically optimized, are viable candidates for real-time semantic
segmentation in embedded automotive perception pipelines
Read the full abstract ▾
Computer Vision
scMIR: a vision-language foundation model for single-cell light microscopy image representation
Yifan Shang, Jiahui Tan, Xiangxiang Zeng et al. (+1 more)
Single-
cell light microscopy images have become an important data source for characterizing
cell phenotypes, but their complexity and heterogeneity pose challenges to high-throughput automated analysis. Existing representation learning methods mostly rely on task-oriented modeling, which is limited by specific
datasets and predefined tasks, making them difficult to generalize across different
cell types and microscopy modalities, and experimental conditions. Although general-purpose methods have improved the generalization ability of image representation in recent years, their limited utilization of experimental background and biological context information still poses challenges in complex phenotypic analysis. Here, we propose scMIR, a
vision-
language foundation model for single-
cell light microscopy image representation. By synergistically combining self-supervised image reconstruction with text-guided cross-modal
alignment, scMIR can simultaneously encode morphological and biological semantic information in a unified representation space. scMIR is pre-trained on 207,957 image-text pairs, covering various
cell types, microscopy modalities, and perturbation conditions. scMIR outperforms existing general models and task-oriented methods as systematically evaluated on various complex tasks using 16
benchmark datasets, including
cell classification, clustering, phenotype inference, and batch effect correction tasks. Furthermore, scMIR shows a strong generalization ability across various tasks without requiring task-specific fine-tuning. With its unique advantages, we envision scMIR may promote the standardization and automation of high-throughput phenotyping workflows through supporting various downstream analysis tasks.
Read the full abstract ▾
All authors Yifan Shang, Jiahui Tan, Xiangxiang Zeng, Renjie Zhou
Computer Vision
Pathologist Attention-Aligned Report Generation for Prostate Histopathology
Ruoyu Xue, Suryakant Singh, Souradeep Chakraborty et al. (+12 more)
The allocation of visual
attention by pathologists during cancer diagnosis is a highly selective process that critically shapes the information extracted from whole-slide images (WSIs). Human
attention helps medical imaging tasks such as classification and
segmentation, and becomes a strong semantic cue for identifying diagnostically informative regions for report generation. In this paper, we introduce human
attention into the training of pathologist report generation models. To this end, we collected a
multimodal human-
attention dataset of 121 prostate WSIs annotated with pathologists' multi-scale viewport trajectories synchronized with the pathologists' verbal descriptions and cursor movements for five clinically relevant components (e.g., Gleason patterns). Using this
dataset, we finetune two report generation models with an
attention-
alignment loss that regularizes the model
attention over image patches to match the distribution of pathologist
attention. We evaluate our approach on prostate cancer report generation and visual question answering using two models with different internal
attention mechanisms (i.e., how image
tokens are integrated into the
language decoder). Experiments show average gains of 10.9% on NLP-based metrics and 19.3% in accuracy across five clinically relevant report components. Further, model
attention maps extracted at inference time, with minimal computational overhead, align more closely with pathologist
attention, providing stronger visual support for the generated reports by highlighting the regions that most influence the output.
Read the full abstract ▾
All authors Ruoyu Xue, Suryakant Singh, Souradeep Chakraborty, Pierre Marza, Oksana Yaskiv, Constantin Friedman, Natallia Sheuka, Paul Friedman, Bharat Ramlal, Beatrice Knudsen, Rajarsi Gupta, Joel Saltz, Prateek Prasanna, Gregory Zelinsky, Dimitris Samaras
Computer Vision
EGRNet: A Lightweight Semantic Segmentation Network with Edge-Gated Refinement and Adversarial Sensing
Bareera Qaseem, Mohsin Kamal and Muhammad Naveed Aman
As autonomous systems and smart cities continue to evolve, the demand for efficient and robust scene understanding becomes increasingly critical. Semantic
segmentation plays a key role in enabling autonomous vehicles to comprehend complex urban environments. However, achieving high accuracy with minimal computational cost remains a significant challenge. In this paper, we present Edge-Gated Refinement Network (EGRNet), a lightweight and efficient deep learning model designed for real-time semantic
segmentation in urban scenarios. The model incorporates depthwise separable convolutions to reduce computational complexity and dilated residual blocks for capturing rich multi-scale contextual information. Additionally, we introduce a novel Edge-Gated Refinement (EGR) module, which adaptively fuses original and refined features through a learnable gating mechanism, enhancing boundary preservation and edge-sensitive regions. To further improve feature representation, Squeeze-and-Excitation (SE)
attention is applied across the network. With only 0.46M parameters, EGRNet achieves state-of-the-art performance while maintaining low computational overhead. When evaluated on the Cityscapes
dataset, the model attains a mean Intersection over Union (mIoU) of 65.28%, demonstrating strong accuracy with minimal resource consumption. Moreover, we introduce a lightweight
adversarial attack detection strategy, ensuring robustness against
adversarial inputs without compromising real-time performance. By combining efficiency, accuracy, and resilience, EGRNet is well-suited for deployment on edge devices in safety-critical real-time applications.
Read the full abstract ▾
Computer Vision · Cross-listed from stat.ME
Deep Shape Regression for Planar Curves with Multimodal Covariates
Manuel Pfeuffer, Roshan Prakash Rane, Hadya Yassin et al. (+2 more)
The shape of a planar curve is the
geometric information that remains once translation, rotation, scale and reparametrisation are removed and is of interest in many health applications, e.g. in neuroimaging. We propose a deep shape regression model for open planar curves that admits
multimodal and high-dimensional
covariates. Representing curves as complex-valued functions, we show that the conditional full Procrustes mean is the leading eigenfunction of the conditional covariance. To estimate this covariance surface, we propose a novel deep conditional covariance smoother with modality-specific encoders - e.g. splines for scalar
covariates and convolutional networks for images, which classical spline smoothers cannot accommodate. Our model is by construction invariant to the translation, rotation and scaling of the input curves and handles sparsely and irregularly sampled curves. We further provide an
algorithm for elastic mean estimation that also removes parametrisation by iterating covariance smoothing, rotational
alignment and parametrisation
alignment. We illustrate the method on simulated outlines with known conditional mean and
multimodal covariates, and give a first application to hippocampal outlines from the ADNI cohort, recovering
covariate effects consistent with the literature. Code is available at https://github.com/mpff/dnn-shapes.
Read the full abstract ▾
All authors Manuel Pfeuffer, Roshan Prakash Rane, Hadya Yassin, Kerstin Ritter, Sonja Greven
Computer Vision
VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers
Xianghong Fang, Yuan Yuan, Dehan Kong et al. (+1 more)
Vector Quantization (VQ) underpins modern discrete visual tokenization. However, training quantization modules for state-of-the-art VQ-based models requires significant computational resources which, in practice, all but prevents the development of novel, cutting-edge VQ techniques under resource constraints. To address this limitation, we propose VQ-Transplant, a simple framework that enables plug-and-play integration of new VQ modules into frozen, pre-trained tokenizers by replacing their native VQ modules. Crucially, the proposed transplantation process preserves all encoder-decoder parameters, obviating the need for costly end-to-end retraining when modifying the quantization method. To mitigate decoder-quantization mismatch, we introduce a lightweight decoder adaptation strategy (trained for only 5 epochs on ImageNet-1k) to align feature priors with the new quantization space. In our empirical evaluation, we find that VQ-Transplant allows obtaining near state-of-the-art reconstruction fidelity for industry-level models like VAR while reducing the training cost by 95%. VQ-Transplant democratizes quantization research by enabling resource-efficient integration of novel VQ techniques while matching industry-level reconstruction performance.
Read the full abstract ▾
All authors Xianghong Fang, Yuan Yuan, Dehan Kong, Tim G. J. Rudner
Computer Vision
ChronoStitch: Training-Free Composition of Visual KV Memories for Long-Horizon Temporal Reasoning
Santiram Tiwari, Nishant Sinha and Kunal Kislay
Long-
video question answering requires a model to preserve visual evidence over time without repeatedly reprocessing the same
video. A practical approach is to store the
vision-
language model's internal key-value (KV) cache for each
video chunk and retrieve that state at query time. However, independently cached
video chunks do not compose correctly: every chunk is prefilled from local rotary position zero, so naive concatenation collides temporal phases and removes the global order required for questions about what happened first, how often events occurred, or what changed across the
video. This paper presents ChronoStitch, a training-free method for composing independently stored visual KV
memories. The method first re-bases stored post-rotary keys onto a global three-axis
multimodal RoPE coordinate system that preserves time, height, and width structure. We show why a one-dimensional scalar re-indexing is geometrically inconsistent for visual
tokens because it turns spatial order within a frame into false temporal displacement. We then address the residual content gap left by positional repair: later chunks were originally encoded without attending to earlier chunks. ChronoStitch therefore selectively recomputes a small fraction of high-deviation later-chunk visual
tokens while allowing them to attend over the composed cache. On Qwen2.5-VL-3B and the temporal split of TempCompass, ChronoStitch outperforms naive composition and position-only variants, improving event-ordering accuracy while running 3.3x faster than full joint re-prefilling.
Read the full abstract ▾
Computer Vision
Synthetic and Derived Training Images for Campus Waste Detection: A Multi-Seed Evaluation with YOLOv8n
Ali Behbahani, Newsha Javanmardi, Shahriar Ahmed et al. (+1 more)
Incorrect disposal can contaminate campus recycling streams, and a bin-mounted camera could provide feedback as an item is discarded. We evaluated whether synthetic and derived images improve a YOLOv8n detector for this view. The real
dataset contained 148 campus photographs: 86 for training, 31 for validation, and 31 for testing. Twelve joint-training configurations varied the amount and source of added images. We repeated seven principal settings with four matched seeds and computed bootstrap percentile intervals over those seeds. The real-only model reached a mean mAP@0.5 of 0.691 [0.665, 0.722]. Background replacement reduced the mean to 0.560 [0.499, 0.619], isolated-object images gave 0.680 [0.644, 0.724], and the full augmentation pool gave 0.487 [0.438, 0.537]. We also tested hand-and-forearm composites because every real photo showed a held object. Two cutouts in the initial composite set came from test photographs, so we discarded that experiment, rebuilt the set with training-split cutouts, and reran all four seeds. The corrected paired difference was +0.034 [-0.063, 0.199], which does not support a reliable hand-composite effect. Single-seed transfer experiments produced source-dependent rankings between joint mixing and sequential pretraining. None of the evaluated configurations exceeded the real-only baseline. The reported intervals quantify seed variation; the 31-photo test set remains too small for strong class-specific conclusions.
Read the full abstract ▾
All authors Ali Behbahani, Newsha Javanmardi, Shahriar Ahmed, Phouvadeth Vathana
Computer Vision
D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models
Heesang Han, A. Lynn Abbott and Abhijit Sarkar
Recent advances in
Multimodal Large
Language Models (MLLMs) have triggered the development of end-to-end MLLMs for autonomous driving. However, the main emphasis to date has been for MLLMs using 2D images and
videos. In contrast, this paper considers MLLM effectiveness using 3D sensors, particularly
LiDAR and stereo cameras.
LiDAR presents unique challenges to integration within an MLLM, largely because of data sparsity and lack of a grid structure for the data. For similar reasons, fusion of camera and
LiDAR data within an MLLM pipeline is also uncommon. However, most autonomous systems rely on
LiDAR-based sensing, and incorporating 3D data has been proven to improve performance in traditional 3D scene perception tasks. This paper presents D3VL, a novel MLLM framework that integrates 2D and 3D time-series data in a single but simple architecture. The model aims to answer questions involving traffic scene understanding and safety. D3VL shows an 11% improvement in the KITTI Question-Answering (QA)
dataset compared to baseline methods in processing 2D and 3D time-series data. This paper further introduces the Waymo QA
dataset extension, which assesses models' capabilities in processing 3D and time-series data under diverse driving conditions. D3VL implementation code and WaymoQA extension can be found on our supplemental website: https://automotivesafety-lvlm.github.io
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Geospatial Diffusion-based Evolution Synthesis (GeoDES) for Storm-Centered Weather Augmentation
Sonia Cromp, Satya Sai Srinath Namburi GNVV, Youran Wang et al. (+4 more)
While machine learning-based weather models hold significant promise, they struggle to predict the detailed structure of large-scale weather systems such as cyclonic storms. Regional models are constrained by limited historical records within fixed geographic boundaries, while global models are computationally expensive and often operate at resolutions too coarse to capture fine-grained storm dynamics. To bridge this gap, we introduce the Geospatial
Diffusion-based
Evolution Synthesis (GeoDES) model, a custom image-to-
video diffusion model. By focusing generation strictly on the evolving storm structure, GeoDES synthesizes physically consistent, high-fidelity weather events suitable for stress-testing forecast models and expanding meteorological
datasets. Evaluations demonstrate that GeoDES outperforms prior methods on key metrics, achieving $52\%$ lower Peak Vorticity Error and $8\%$ higher Anomaly Correlation Coefficient than the next strongest methods on the North Atlantic test set.
Read the full abstract ▾
All authors Sonia Cromp, Satya Sai Srinath Namburi GNVV, Youran Wang, Grace Kisslinger, Frederic Sala, James Booth, Allegra LeGrande
Computer Vision
Crowd4D: Scene-Aware Monocular 4D Crowd Reconstruction
Hongbo Kang, Tianyi Zhou, Qingyang Yang et al. (+4 more)
Recovering scene-consistent 4D crowd motion from
monocular video in large-scale scenes remains challenging due to severe depth ambiguity and complex scene geometry. Existing
monocular crowd reconstruction methods typically rely on single-plane assumptions, leading to unreliable metric scale and spatial drift under complex terrain. We propose Crowd4D, the first scene-aware 4D crowd reconstruction framework that jointly optimizes the crowd and scene from a
monocular RGB
video in large-scale scenes. Crowd4D explicitly incorporates scene geometry and ensures consistency across image and scene spaces via a multi-stage optimization strategy. A key bottleneck of this task lies in accurate human-scene
alignment, particularly in scale and position. However, human and scene reconstructions are typically decoupled. To address this, we introduce the Human-Scene Interaction Proxy, abbreviated as HSIP, as an intermediate representation derived from Scene Interaction Point Clouds and a Scene Interaction Surface, abbreviated as SIPC and SIS. These representations encode explicit scene-aware
geometric priors and redefine the optimization space for large-scale
monocular 4D crowd reconstruction. To further improve temporal stability under occlusions, we introduce Crowd Structural Coherence Regularization, abbreviated as CSCR, which leverages HSIP-based spatial priors to impose soft temporal consistency on pairwise relative displacements and directions within local crowd neighborhoods. Extensive experiments demonstrate that Crowd4D consistently outperforms existing state-of-the-art methods and enables robust
monocular 4D crowd reconstruction in complex, large-scale real-world scenes.
Read the full abstract ▾
All authors Hongbo Kang, Tianyi Zhou, Qingyang Yang, Hongwei Wen, Jing Huang, Yu-Kun Lai, Kun Li
Computer Vision · Cross-listed from eess.IV
BLUE: Semantics-Preserving Video Compression for Efficient Vision-Language Surveillance Analytics
Shubham Baid, Akash James, Sahil Chachra et al. (+2 more)
Continuous surveillance
video creates a growing storage, transmission, and inference burden for enterprise
video analytics systems. While modern codecs such as H.265 reduce bitrate for human-viewable
video, aggressive compression can degrade downstream computer-
vision performance and does not necessarily reduce the number of
vision-
language model (VLM) inference calls required for semantic
video understanding. This paper evaluates BLUE, a fixed-camera surveillance compression approach that suppresses static-background redundancy while preserving foreground activity, for its effect on VLM-based event and anomaly understanding. We compare raw H.265 and BLUE-compressed H.265
video on two surveillance
datasets: VIRAT, comprising 227 paired event samples from 106 clips, and CHAD, comprising 54 human-activity anomaly clips. For each pair, the same frame index is evaluated using a VLM captioning pipeline, and outputs are scored against annotation-derived ground truth using a blind judging protocol. The results show no measurable degradation in semantic inference quality. On VIRAT, the mean VLM score remains effectively unchanged between raw H.265 and BLUE, with a mean difference of approximately -0.01 on a 0-10 scale. On CHAD, raw H.265 and BLUE obtain near-equivalent mean scores of 4.31 and 4.26, respectively. Compression saving is also uncorrelated with VLM score change on VIRAT (r = 0.004), indicating that higher BLUE compression does not predict semantic quality loss. Beyond storage reduction, BLUE increases the share of skip-heavy P-frames on CHAD from 1.4% to 53.2%, enabling an estimated 53% reduction in VLM calls through packet-size-based frame skipping. These findings suggest that BLUE functions as a machine-centric compression layer for surveillance
video, reducing bandwidth and inference cost while preserving VLM semantic performance.
Read the full abstract ▾
All authors Shubham Baid, Akash James, Sahil Chachra, Nishant Sinha, Kunal Kislay
Computer Vision
Detect Early, Escalate Rarely: Anytime Detection of AI-Generated Video from the Compressed Bitstream
Mert Onur Cakiroglu, Mehmet Dalkilic and Hasan Kurban
Detectors for AI-generated
video are evaluated offline. A clip is decoded to pixels and scored once, increasingly by a large
vision-
language model. Detection, however, is deployed online. We recast the task as streaming perception and score the motion field the codec already wrote into the bitstream. Reading that field is a parse, not a pixel-domain forward pass. Because the running aggregate is monotone, one end-calibrated threshold is anytime-valid at the data-dependent decision time. Recalibrating at each prefix is not. Escalation is priced in closed form. A compute budget maps to a deferral window, on a frontier monotone exactly where the deferral condition holds. On matched GenVidBench the codec stage reaches full-length AUC 0.64 at five orders of magnitude less compute than a pixel CNN, on CPU. Its gate holds the stopping-time false-positive rate at target while the real data match its calibration, and drifts above it under distribution shift. Deferring 15% of clips lifts accuracy from 0.75 to 0.78 at $7\times$ less compute (paired: McNemar $p<10^{-6}$). The stage-1 ordering replicates on AIGVDBench. We introduce no new detector. The contribution is the reframing, two guarantees, and the measured frontiers. Code, configurations, and evaluation splits: https://github.com/KurbanIntelligenceLab/streamdet.
Read the full abstract ▾
Computer Vision · Cross-listed from astro-ph.IM
Strong Gravitational Lensing Posterior Sampling in Pixel-Space Using Diffusion Models and Recurrent Inference Machines
Guillaume Payeur, Laurence Perreault-Levasseur, Gabriel Missael Barco et al. (+1 more)
Modeling
galaxy-
galaxy strong gravitational lenses to infer the brightness of the source
galaxy and the mass distribution of the foreground
galaxy is computationally challenging, particularly for high-resolution, high signal-to-noise ratio observations. In this regime, high-dimensional representations of both the source and the foreground mass distribution are necessary to model the data down to the noise level. This inference problem has been challenging for both traditional and machine learning-based methods because of its high dimensionality and its non-linearity in the foreground mass distribution. We present a method to generate joint posterior samples of the source
galaxy and foreground mass distribution as pixelated images conditioned on observations. The method combines
diffusion-based generative modeling and recurrent inference machines. It can model realistic gravitational
lensing simulations with background and foreground
galaxies drawn from cosmological hydrodynamical simulations down to the noise level.
Read the full abstract ▾
All authors Guillaume Payeur, Laurence Perreault-Levasseur, Gabriel Missael Barco, Yashar Hezaveh
Computer Vision · Cross-listed from eess.IV
Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling
Shaokang Wang, Jinchang Xu, Peidong Jia et al. (+9 more)
Most existing
video compression
algorithms follow a paradigm of transformation and quantization, optimizing the trade-off between distortion and bitrate. However, extremely low-bitrate compression remains an underexplored frontier where perceptual quality optimization under severely constrained coding resources has not been adequately addressed. In this paper, we propose a unified generative framework that leverages pre-trained
Diffusion Transformer (DiT) priors to achieve high perceptual quality at extremely low bitrates. We first introduce a flexible Group-of-Latents (GoL) strategy within the latent space of a causal tokenizer, explicitly partitioning the latent stream into intra $I$-latents and inter $P$-latents. The Deep Compression Module (I-DCM) then encodes key $I$-latents to preserve perceptual anchors with minimal overhead. Building upon these anchors, the DiT-based Unified Latent Denoising Module (U-LDM) refines intra-frame textures and synthesizes $P$-latents from noise, reconstructing temporal dynamics at zero additional bitrate cost. Extensive experiments demonstrate that our method uniquely operates in the extreme-low-bitrate regime (e.g., (<0.005) bpp), achieving state-of-the-art perceptual fidelity with rich spatial details and robust temporal consistency. The code will be made publicly available.
Read the full abstract ▾
All authors Shaokang Wang, Jinchang Xu, Peidong Jia, Zhijian Hao, Siyuan Qian, Fei Zhao, Rui Ma, Xiaozhu Ju, Jian Tang, Xiaodong Xie, Shanghang Zhang, Huizhu Jia
Computer Vision
Rahul Sajnani, Yulia Gryaditskaya, Radomír Měch et al. (+2 more)
Controllable image generation remains challenging for creative professionals, who often require precise regional control over materials, object identities, and spatial arrangements that cannot be reliably achieved through text prompting alone.
Diffusion Transformers (DiTs) can natively ingest heterogeneous
tokens stemming from texts and images, but they lack mechanisms for determining where and how these
tokens should influence the output. We introduce appearance pointers, compact
tokens that guide DiTs toward the correct appearance cues at the correct spatial locations by aligning text or image inputs with user-specified masks. Appearance pointers are produced by a region correspondence network and refined through a spatial aggregation mechanism, enabling the model to handle multiple regional descriptions without significantly increasing
token load. Our approach introduces the first modality-agnostic interface for localized
multimodal control in a DiT without retraining the base model from scratch. Across a range of metrics, our single model reaches or surpasses the performance of modality-specific state of the art methods, offering a simple and extensible path toward precise, region-aware,
multimodal guidance in generative image synthesis.
Read the full abstract ▾
All authors Rahul Sajnani, Yulia Gryaditskaya, Radomír Měch, Srinath Sridhar, Matheus Gadelha
Computer Vision
Masked Visual Actions for Unified World Modeling
Hadi Alzayer, Wenlong Huang, Haonan Chen et al. (+8 more)
Video models absorb rich priors over how the visual world moves, interacts, and responds to contact, making them promising substrates for robotic world modeling. The central challenge is how to communicate action to such models in a form aligned with the visual space in which they learned these interaction priors, yet still grounded in physical manipulation. We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a
video. Revealing
robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level
robot actions, while revealing desired object motion makes the same model recover
robot behavior consistent with that outcome. Finetuned with only 15 hours of masked examples from real
videos and simulation, a single checkpoint achieves strong visual fidelity and controllability across diverse scenes and multiple embodiments. In downstream manipulation settings, the model produces imagined rollouts whose outcomes correlate with real-world execution for policy evaluation, improves decision making by ranking candidate futures in model-based planning, and supports inverse modeling by synthesizing
robot motion from desired object motion.
Read the full abstract ▾
All authors Hadi Alzayer, Wenlong Huang, Haonan Chen, Christopher Luey, Lvmin Zhang, Maneesh Agrawala, Gordon Wetzstein, Li Fei-Fei, Yilun Du, Jiajun Wu, Jia-Bin Huang
Computer Vision
ExpertVerse: A General-Purpose Benchmark for Expert-Level Reasoning in Knowledge-Intensive Visual Synthesis
Yuan Wang, Yongchao Du, Mengting Chen et al. (+4 more)
Recent advances in
multimodal generative models have enabled instruction-based image generation to move beyond semantic manipulation to knowledge-driven visual
reasoning. However, these methods focus on explicit commonsense
reasoning, shallow causal understanding, and direct knowledge recall, failing at knowledge-intensive generation. We develop ExpertVerse, a capability-centric
benchmark to evaluate generative models via knowledge-intensive lens. ExpertVerse stratifies
reasoning generation across an orthogonal taxonomy of 9 cognitive capabilities and 8 expert disciplines, yielding 58 sub-disciplines. We curate 1,611 expert-annotated instances covering single-image editing, multi-image composition, and text-to-image generation. We further develop an automated workflow to produce ExpertVerse-100K, a large-scale
dataset with
reasoning traces and knowledge-anchored rationale annotations. Based on this, we train KnowThinker with RL fine-tuning, a VLM
reasoning engine with world knowledge that jointly generates thinking processes and refined instructions. Towards the cross-modal credit misalignment and multi-objective gradient conflicts in multi-reward optimization, we propose a tailored Bootstrapped Pareto Policy Optimization (BPPO), which synergizes Bootstrapping Reward Rectification (BRR) and Conflict-Aware Pareto Advantage Fusion (CPAF). Extensive results of both open-source and proprietary models exposes critical
reasoning deficits, highlighting imperative for knowledge-intensive
benchmarks towards next-generation visual generation.
Read the full abstract ▾
All authors Yuan Wang, Yongchao Du, Mengting Chen, Jinsong Lan, Jiaxuan Luo, Xuetao Feng, Xiaoyong Zhu
Computer Vision
OmniReasoner: Thinking with Long Audio-Video via Native Tool Use
Yu Chen, Caorui Li, Ziyu Xiong et al. (+8 more)
Long audio-
video reasoning is difficult for omnimodal LLMs because the decisive evidence is often sparse, cross-modal, and too expensive to preserve with uniformly high-fidelity inputs. We introduce OmniReasoner, a tool-use post-training framework for Thinking with Long Audio-
Video: omni-modal LLMs learn, via supervised fine-tuning and reinforcement learning, to decide whether and where to call a zoom-in tool before answering. OmniReasoner first builds a low-cost global preview of the full stream and then, when needed, calls the zoom-in tool with a requested temporal interval for higher-fidelity visual and audio inspection before answering. Because the model observes different sampling granularities before and after this call – a sparse global preview and a denser local clip – we introduce TimeAnchor, which keeps the tool's temporal argument valid and round-trip-consistent across these granularities, rather than tied to frame indices from a particular sampling rate. To make this tool-use behavior trainable without expensive manual interval annotation, we build a Temporal Augmented Data Engine that synthesizes tool-use post-training trajectories by
video editing and composition. Experiments across omnimodal and
video benchmarks show that OmniReasoner improves both answer accuracy and temporal grounding while concentrating high-fidelity computation on informative regions. Code is available at https://github.com/RockyChen0205/OmniReasoner.
Read the full abstract ▾
All authors Yu Chen, Caorui Li, Ziyu Xiong, Yidong Wang, Mingqi Gao, Shuman Liu, Biao Liu, Chunfeng Yang, Anxiang Zeng, Haibo Zhang, Chaofan Chen
Computer Vision · Cross-listed from cs.LG
ROMS-IMLE: A Minimalist Approach to Competitive Single-Step Generative Modelling
Chirag Vashist and Ke Li
Generative models have undergone many generations of
evolution, from VAEs/GANs to
diffusion/flow matching. Along the way, the underlying techniques have become more complicated and various beliefs about what drives strong empirical performance have taken hold. Due to the success of
diffusion models and flow matching, one of the more common beliefs is the importance of transforming the noise distribution to the data distribution gradually through many small transformations. We ask whether this is truly necessary, and take a minimalist approach to designing a competitive generative model. We start with the bare-bones essentials, namely just a training objective and a model. We purposefully make both simple. For the training objective, we choose Implicit Maximum Likelihood Estimation (IMLE), and eschew more complicated alternatives such as variational inference,
adversarial training and numerical integration. For the model, we eschew
transformers and instead choose a moderately sized convolutional network. Then we judiciously added elements that are truly essential, which surprisingly do not include iterative denoising. The result is a single-step parameter-efficient generative model that produces high quality samples at fast speed: it achieves an FID of 2.56 on ImageNet 256 and simultaneously attains good precision and recall.
Read the full abstract ▾
Computer Vision
InstructMixup: Instruction-Guided Salient Patch Editing for Robust Data Augmentation
Khawar Islam, Arif Mahmood, Xin Jin et al. (+1 more)
In image and
video technologies, data augmentation is widely used to improve the generalization of deep visual models, and mixup-based strategies that interpolate between samples have become the dominant approach. However, computing informative mixing regions adds substantial overhead, and blending content across different images frequently disrupts the semantic integrity of the resulting sample. We propose \our{}, a data augmentation method that constructs challenging yet label-consistent training samples entirely within a single visual sample. \our{} first extracts multi-scale salient patches from the sample using a lightweight saliency detector, refines each patch with an instruction-guided generative model, and blends the edited patch back into the non-salient regions of the same sample; because the generative edits are computed once and cached offline, this step adds negligible training cost. To further diversify the learned representation, \our{} injects self-similar fractal structure into the same salient regions at an adaptive ratio, so each training sample carries both fractal and non-fractal structure. We derive a second-order approximation of the resulting vicinal risk, showing that the method simultaneously enforces invariance to the generative edit and suppresses
curvature along the perturbed salient directions, and we verify both predictions empirically. We evaluate on small to large backbones for instance Convolutional Neural Networks (CNNs),
Vision Transformers (ViTs) and
Vision-
Language Foundational Models (VLMs) across seven
benchmarks covering coarse- and fine-grained classification, robustness to corruption and occlusion, calibration, and transfer and self-supervised learning, InstructMixup outperforms nine competing augmentation methods, surpassing the strongest baseline across all
benchmarks.
Read the full abstract ▾
All authors Khawar Islam, Arif Mahmood, Xin Jin, Naveed Akhtar
Computer Vision
ERank in Latent Space as an Image-Complexity and Richness Measure
Maksim Smirnov, Grigory Kononov, Anastasiia Linich et al. (+2 more)
We propose the effective rank (ERank) of the channel covariance of an image's deep feature map as a per-sample, label-free measure of visual richness, computed from a single forward pass through a frozen pretrained encoder. ERank counts how many decorrelated channel directions an image activates, and we characterize its properties, including its behavior under noise. Empirically, ERank orders images from plain to visually rich, correlates with codec bitrate, sharpness, and edge density, and correlates with human complexity annotations on IC9600 with $r = 0.72$. As a data-selection criterion, removing low-ERank samples improves super-resolution and removing high-ERank samples improves OCR, in both pretraining and finetuning, while selection does not help classification,
segmentation, or denoising. ERank is thus a cheap richness signal, useful exactly when task difficulty is governed by input richness.
Read the full abstract ▾
All authors Maksim Smirnov, Grigory Kononov, Anastasiia Linich, Egor Surkov, Egor Shvetsov