Computer Vision · Cross-listed from physics.optics
Hybrid deep learning-based phase diversity method for wavefront reconstruction
Y. Rodimkov, A. Kotov, K. Burdonov et al. (+4 more)
The efficiency of high-power
laser systems is limited by wavefront distortions in the beam, particularly non-common path aberrations, which reduce the peak intensity at the focal plane. Compensating for these aberrations requires the calibration of the adaptive optics system. Conventional calibration methods rely on a time-consuming iterative optimization that is highly sensitive to initial conditions. While deep learning-based models offer high speed, they often demonstrate insufficient accuracy. In this work, we present a hybrid wavefront reconstruction method that combines a convolutional neural network to generate an initial estimate of the wavefront distortions, with the L-BFGS (Limited-
memory Broyden-Fletcher-Goldfarb-Shanno)
algorithm for its subsequent refinement. In numerical simulations, the method achieved an efficiency of $\sim 0.99$ in 80% of the cases for a root-mean-square (RMS) of wavefront distortions ranging from 0 to $1.3\lambda$. In a physical experiment, for initial wavefront distortions with RMS values from 0.15 to $0.6\lambda$, the method achieved an efficiency of $\sim 0.75$. As a result, focusing with a Strehl ratio of $0.96 \pm 0.02$ was attained within 2 to 4 iterations of the
algorithm, confirming the applicability of the method for the fast and accurate calibration of adaptive optics systems under real experimental conditions.
Read the full abstract ▾
All authors Y. Rodimkov, A. Kotov, K. Burdonov, S. Perevalov, V. Volokitin, I. Meyerov, A. Soloviev
Computer Vision
Naturalness Predicts but Does Not Cause Transferability in Image Encodings of Real-World Streams
Faruk Alpay and Baris Basaran
A common practice converts a one-dimensional signal into an image so that a
vision backbone pretrained on natural photographs can be reused for recognition, yet the encoded image is rarely examined. We ask how the visual naturalness of an encoded image relates to its transfer accuracy under a frozen backbone. We build WorldStream, a corpus of 299 heterogeneous current-value series from key-free public APIs (weather, air quality, earthquakes, gold and oil, equities, crypto, foreign exchange, web activity and space weather), with a nine-way source-recognition task over 3143 temporally split windows. Across seven encodings and six frozen backbones, the Frechet distance of an encoding to natural images (FID) predicts its accuracy: Spearman $\rho=-0.72$. Two controlled interventions show this is not causal in the spectrum. Our invertible encoder has a single adjustable part, a spectral exponent $\beta$ (power $\propto |f|^{-\beta}$); varying $\beta$ moves the image toward or away from the natural-image
manifold at fixed content. FID is lowest near the natural value $\beta \approx 2$, but frozen accuracy stays flat and far below the structured baselines (19.2% vs. 73.0%), and FID and accuracy are only weakly related over the sweep (Pearson $-0.32$). A second intervention, phase scrambling, holds the power spectrum exactly fixed while removing local structure; now FID and accuracy fall together (Pearson $-0.89$). The cross-encoding correlation is thus mediated by local structure, not spectral naturalness: FID predicts accuracy because Inception reads the same structure the backbones do. Full fine-tuning does not close the gap (27% vs. 67%), so the deficit is structural. The encoder is exactly invertible, recovering the signal from the 8-bit image at 72.9 dB, so the image doubles as a lossless record of the data.
Read the full abstract ▾
Computer Vision
Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphs
Agnese Taluzzi, Riccardo Santambrogio, Simone Mentasti et al. (+2 more)
Existing multi-modal large
language models (MLLMs) face significant challenges in processing long
video sequences due to strict input
token limitations. As a result, current
video understanding approaches, especially in
egocentric settings characterized by complex dynamics, frequent state changes, and moving cameras, are forced to massively subsample frames. This leads to severe loss of temporal and contextual information, constraining their ability to perform fine-grained
video reasoning. In this work, we introduce a framework for
egocentric video question answering (VQA) that overcomes these input constraints through
Egocentric Scene Graphs (EgoSGs), i.e., temporally grounded, structured representations that capture objects, attributes, spatial relations, and interactions over time. By representing
videos as compact, text-based scene graphs, our method preserves the essential visual and temporal information of the original
video in a symbolic form that drastically reduces input length while maintaining semantic richness. Crucially, this enables MLLMs to reason efficiently over entire
video sequences within their
token budget. On HD-EPIC VQA, our method achieves state-of-the-art results, outperforming strong
video-based baselines on multiple models and suggesting that structured, temporally grounded representations like EgoSGs can bridge long-form
egocentric video understanding and the context limitations of today's MLLMs.
Read the full abstract ▾
All authors Agnese Taluzzi, Riccardo Santambrogio, Simone Mentasti, Chiara Plizzari, Matteo Matteucci
Computer Vision
Edges Before Embeddings: A Confidence-Aware Blur Gate for Vision-Language Pipelines
Duy Tran Thanh
Production
vision pipelines silently degrade on blurry input, wasting compute on downstream OCR,
retrieval, and
vision-
language model (VLM) calls that cannot recover a usable output. We present MagikaDocumentFromPixel, a lightweight, CPU-friendly image quality gate that classifies a single image as sharp, blurred, or uncertain in roughly 7 ms on a single CPU core. The contributions are (i) a recipe selected from a 46-configuration, 8-sweep empirical search that isolates input resolution as the dominant lever and shows architecture capacity only pays off at >= 384 px; (ii) a confidence-aware routing formalism grounded in classical selective prediction; (iii) the Edge Prior Module (EPM), a
Laplacian-magnitude auxiliary input channel that gives the network direct access to the spectral evidence that classical blur heuristics rely on and that lifts test F1 by +1.3 points in a matched-env comparison; and (iv) an observation that the gate is one instance of a recurring design pattern that appears independently in Magika content-type detection, risk-controlled OCR with VLMs, and DocVLM. The final recipe MobileNetV3-Large with the EPM trained at 384x384 on paired GoPro Large frames, evaluated with 5-scale test-time augmentation reaches F1 = 0.9803 (AUC 0.9989) with a 17 MB ONNX artifact, improving over our fixed-scale baseline on the same hardware (F1 = 0.9672) by +1.31 points. We are explicit about limitations: results are on a single motion-blur distribution, numbers are from a single seed, and calibration is qualitative rather than measured.
Read the full abstract ▾
Computer Vision
Shift Variant Image Degradation and Restoration Using Singular Value Decomposition
Arun D. Kulkarni
Shift-variant image degradation is frequently encountered in practical imaging systems where the point spread function (PSF) varies across the image field due to motion, optical aberrations, atmospheric
turbulence, or sensor-related effects. Unlike shift-invariant, shift-variant degradation presents significant challenges for image restoration because the degradation process cannot be represented by a single convolution kernel. This paper proposes a singular value decomposition (SVD)-based framework for restoring images degraded by shift-variant motion blur. The proposed approach determines the contribution of small singular values using a singular-value energy retention criterion. Specifically, the number of small singular values is selected based on a specified percentage of cumulative singular-value energy, providing a systematic approach for controlling noise amplification while preserving useful image information. The degradation model is formulated using a position-dependent PSF represented by a shift-variant imaging operator. Three representative one dimensional shift-variant motion PSFs are considered: bidirectional linear motion, Gaussian motion, and simple harmonic motion. The image degradation process is modeled as a linear system, and SVD is employed to analyze and invert the corresponding degradation operator. The singular-value representation provides insight into the ill-conditioned nature of the restoration problem and enables the development of stable inversion techniques. The proposed SVD-based restoration
algorithm is applied to three degraded images. Experimental results demonstrate the effectiveness of the proposed approach in recovering image details and reducing blur artifacts under different motion models.
Read the full abstract ▾
Computer Vision
$S^{2}$-FracMix: Label-Preserving Self-Saliency Mixup Augmentation
Khawar Islam, Arif Mahmood, Xin Jin et al. (+1 more)
Data augmentation is known to improve generalization of deep visual models. Recent methods favor mixup strategies that generate interpolated samples to improve model performance. However, these techniques not only incur significant computational overhead, they also lead to semantic disruption of augmentation data due to cross-sample mixing. We first propose Self-Saliency ($S^2$) Mixup, which constructs challenging yet label-consistent samples by extracting multi-scale salient patches and reinserting them into non-salient regions of the same image. This promotes scale-invariant feature learning while avoiding cross-sample interference. To further enhance model robustness, we introduce FracMix, a mixing scheme that injects self-similarity patterns into salient regions using adaptive ratios. Collectively, our unified framework, $S^{2}$-FracMix, enables simultaneous learning from fractal and non-fractal structures within a single image, yielding a targeted and structurally coherent augmentation strategy. We theoretically analyze the advantage of our technique, and empirically establish its superiority over the existing methods by achieving state-of-the-art performance in extensive evaluation with seven
benchmarks across classification (coarse and fine-grained), robustness, calibration, object detection, and transfer learning tasks. Project page is available at \href{https://fracmix-data-augmentation.github.io/}{fracmix-data-augmentation.github.io}
Read the full abstract ▾
All authors Khawar Islam, Arif Mahmood, Xin Jin, Naveed Akhtar
Computer Vision · Cross-listed from cs.LG
Re-mixing Embeddings for Patient Augmentation in Data Scarce Multiple Instance Learning
Muhammed Furkan Dasdelen, Fatih Ozlugedik, Anastasia Litinetskaya et al. (+3 more)
Data scarcity is a major bottleneck in medical Multiple Instance Learning (MIL), especially for rare diseases or expensive modalities. We introduce a statistically grounded patient augmentation approach that generates realistic patients directly in
embedding space. Using Gaussian Mixture Models as a probabilistic clustering approach on pooled instance
embeddings from all patients, our method learns disease-specific "recipes"-statistical distributions of instances across unsupervised clusters. New patients are then generated by sampling
embeddings from clusters based on learned recipes. Unlike existing methods that require examples from all categories, our method can generate patients offline by re-mixing pooled
embeddings. Generated patients are further selected based on uncertainty quantification to improve MIL performance. We evaluate our method across three clinically relevant scarcity scenarios: (i) cross-
dataset transfer, where an entirely missing "healthy" class is generated using statistics from an external cohort; (ii) low-data regimes, where class sizes are extremely limited; and (iii) small-cohort non-image tasks, including single-
cell RNA-seq and flow cytometry. Across all experiments, our method improves performance over baseline, often outperforming other bag-mixing strategies. Notably, in the missing-class scenario, a performance comparable to full-
dataset training is achieved, demonstrating its potential for rare disease diagnostic and
privacy-preserving patient augmentation. The code is available at https://github.com/marrlab/RECIPE
Read the full abstract ▾
All authors Muhammed Furkan Dasdelen, Fatih Ozlugedik, Anastasia Litinetskaya, Nassir Navab, Carsten Marr, Ario Sadafi
Computer Vision
ShutterMuse: Capture-Time Photography Guidance with MLLMs
Jiayu Li, Yixiao Fang, Tianyu Hu et al. (+5 more)
Real-world photography requires capture-time guidance for both camera framing and subject pose. Yet existing aesthetic cropping
benchmarks mainly evaluate post-hoc crop prediction and overlook subject-side recommendations, leaving the capture-time guidance capabilities of
multimodal large
language models (MLLMs) underexplored. To address this gap, we introduce CaptureGuide-Bench, a
benchmark with two complementary tasks: photographer-side composition decision and refinement, and subject-side scene-conditioned pose recommendation. Our evaluation reveals limitations: general-purpose MLLMs can make composition decisions but lack precise refinement localization, while specialized aesthetic cropping models localize crops effectively but are limited to refinement; neither provides actionable pose guidance. To support model development, we further construct CaptureGuide-
Dataset, comprising 130K samples with textual rationales and structured visual annotations, and develop ShutterMuse, a unified MLLM trained with supervised and reinforcement fine-tuning. Experiments on CaptureGuide-Bench show that ShutterMuse achieves the best overall photographer-side performance among evaluated baselines and competitive subject-side pose recommendation with substantially lower inference cost, demonstrating the potential of MLLMs as interactive assistants for photography during image capture.
Read the full abstract ▾
All authors Jiayu Li, Yixiao Fang, Tianyu Hu, Wei Cheng, Ping Huang, Zheheng Fan, Gang Yu, Xingjun Ma
Computer Vision · Cross-listed from cs.LG
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
Divake Kumar, Sina Tayebati, Devashri Naik et al. (+5 more)
Computer-use
agents turn
vision-
language model (VLM) predictions into executable GUI clicks, so reliable uncertainty estimates are essential for rejection, calibration, miss-severity ranking, and spatial safety regions. Yet evidence on post-hoc uncertainty quantification (UQ) for these
agents is fragmented across isolated model and
dataset pairs, leaving it unclear whether UQ rankings stay stable when the
agent,
benchmark, or observable interface changes. We present Argus, a cross-regime
benchmark for post-hoc UQ in single-step executable GUI grounding: a 27-method open-weight
matrix over 4 VLM
agents and 4
datasets, plus an 8-method closed-source
matrix across 3 frontier vendors where logits, hidden states, and
attention maps are unavailable. Evaluated methods span logit-based scores, sampling and consistency measures, hidden-state and density
estimators (Mahalanobis, SAPLMA),
attention-based scores, P(True) and verbalised-confidence prompting, and split-conformal prediction. The main finding is selective transfer: UQ rankings are stable across
datasets for a fixed model, but degrade across model classes and observable interfaces. Hidden-state and density methods are the most stable open-weight family, while CoCoA-1MCA, Focus, sampling-based scores, and verbalised self-assessment win in specific regimes. Within-model ranking transfer is strong (Spearman rho up to 0.969), but cross-tier transfer to closed-source vendors averages only +0.08, so closed-source UQ should be reranked on the target rather than extrapolated. Conformal click regions show score-level discrimination is not enough for deployment: locally weighted disks shrink radii by 40-60% when the plug-in UQ is calibrated, but coverage degrades under calibration-test or interface mismatch. We release per-item records, calibration/test splits, UQ scores, and analysis scripts for regime-aware UQ selection in GUI
agents.
Read the full abstract ▾
All authors Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, Amit Ranjan Trivedi
Computer Vision
Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs
Wenjie Zhu, Yabin Zhang, Liang Xu et al. (+3 more)
While test-time adaptation (TTA) empowers
vision-
language models to adapt without costly retraining, it remains highly vulnerable to out-of-distribution (OOD) outliers prevalent in real-world applications. This discrepancy motivates Noisy TTA (NTTA), an online task to filter noisy OOD samples on the fly while maximizing in-distribution (ID) classification accuracy. Existing zero-shot NTTA approaches typically rely on test-time discriminative training, leading to overconfident misclassifications and significantly degraded inference efficiency. To address these limitations, we propose a novel framework named Dual Distribution Estimation (DDE), shifting the zero-shot NTTA paradigm from instance-level learning to training-free Gaussian distribution modeling. DDE incorporates two novel modules: Positive Feature Distribution Estimation (PFDE) and Negative Label Distribution Estimation (NLDE). PFDE explicitly models class-wise inclusion and exclusion Gaussian distributions to formulate a calibrated contrastive score, robustly enhancing ID accuracy. In parallel, NLDE improves OOD identification by explicitly modeling the negative label distribution to mine highly discriminative labels, effectively mitigating spurious correlations. Extensive experiments show that on the large-scale ImageNet
benchmark, DDE achieves an improvement of 3.70% in harmonic mean accuracy and reduces the FPR95 for OOD detection by 6.20%, while ensuring highly scalable and efficient online inference. Furthermore, DDE is zero-shot and training-free, demonstrating remarkable robustness in data-scarce scenarios. Codes are available at https://github.com/ZhuWenjie98/DDE.
Read the full abstract ▾
All authors Wenjie Zhu, Yabin Zhang, Liang Xu, Xin Jin, Wenjun Zeng, Lei Zhang
Computer Vision
Point Cloud Diffusion with Global and Local Reconstruction for Instance-Level 3D Anomaly Detection
Linchun Wu, Qin Zou, Jiwen Lu et al. (+1 more)
3D anomaly detection in point clouds is critical for high-precision industrial manufacturing. Reconstruction-based methods have laid a strong foundation by detecting 3D anomalies through comparisons between defective inputs and their reconstructed normal counterparts. However, existing methods still suffer from two challenges: 1) the foreground weak defective regions such as scratches are hard to reconstruct and detect, where the anomaly deviations in normalized point clouds can be as small as $10^{-3}$; 2) the background non-defective regions are prone to get positional bias in reconstruction, which leads to false positives. To address these challenges, we propose PCDiff, a point cloud
diffusion framework for instance-level 3D anomaly generation and detection. In the generation phase, an instance-level multi-modal
attention is embedded into the generation framework, where anomalies are conditioned with texture gradient, image patch, text and mask. The instance-level condition enables the high-quality generation of weak-defective anomalies. In the detection phase, a joint local-global reconstruction
algorithm is introduced to ensure local anomaly restoration and global
geometric consistency, which preserves background normal structure while restoring the foreground defect. Extensive experiments demonstrate that the proposed PCDiff significantly outperforms state-of-the-art methods in both 3D anomaly generation fidelity and reconstruction quality, leading to substantial improvements in anomaly detection accuracy.
Read the full abstract ▾
All authors Linchun Wu, Qin Zou, Jiwen Lu, Qingquan Li
Computer Vision
UniTeD: Unified Temporal Diffusion for Joint Perception and Planning in Autonomous Driving
Bo Zhao, Xinting Zhao, Naifan Li et al. (+2 more)
Diffusion models have shown strong potential for multi-modal planning in end-to-end autonomous driving. However, most existing methods confine
diffusion to the planning module, conditioning on fixed outputs from separate discriminative perception networks. This decoupled design propagates perception errors to the planner, increasing optimization difficulty and reducing robustness. To overcome these limitations, we propose UniTeD, a Unified Temporal
Diffusion framework that jointly models perception and planning through iterative denoising in a shared generative space. By enabling bidirectional information exchange, the framework facilitates mutual refinement between tasks and improves robustness via noise-conditioned multi-task training. We further extend this unified
diffusion paradigm to a streaming setting by incorporating temporal context. A Temporal Transition Module (TTM) is introduced to resolve the noise-level mismatch between historical and current frames. In addition, we propose an Anchor Refresh Strategy (ARS) to alleviate the training-inference distribution shift commonly observed in sparse
diffusion-based end-to-end driving frameworks. Without bells and whistles, UniTeD achieves state-of-the-art performance across multiple
benchmarks, surpassing both recent discriminative end-to-end methods and
diffusion-based planning approaches.
Read the full abstract ▾
All authors Bo Zhao, Xinting Zhao, Naifan Li, Erkang Cheng, Haibin Ling
Computer Vision
Efficient Real-World Dehazing via Physics-Inspired Global-Local Decoupling
Yifei Qu, Ru Li, Junjie Chen et al. (+1 more)
Real-world single image dehazing is highly ill-posed due to spatially and spectrally varying scattering, while practical deployment demands lightweight and low-latency models. Existing approaches either rely on fragile physical inversion under simplified assumptions or adopt heavy blind architectures unsuitable for edge deployment. To overcome these limitations, we propose PGL-Net (Physics-Inspired Global-Local Decoupling Network), a lightweight framework that incorporates physical inductive biases via operator-level emulation, avoiding explicit parameter estimation. It decouples dehazing into global distribution rectification and local structural refinement. A Physics-Inspired Affine Fusion (PAF) module performs globally conditioned
alignment across hierarchical skip connections to compensate for haze-induced bias, while a compact Degradation-Aware Modulation (DAM) block adaptively restores spatially and spectrally variant details through dynamic feature modulation. Extensive experiments on multiple real-world
benchmarks demonstrate that PGL-Net achieves state-of-the-art restoration quality with significantly reduced complexity. Compared with the recent SOTA SGDN, the Tiny variant (PGL-Net-T) improves PSNR by up to 2.6dB and consistently enhances downstream object detection accuracy, while achieving over a 10x reduction in inference latency. Code is available at https://github.com/sc-30-bit/PGL-Net}{github.com/sc-30-bit/PGL-Net.
Read the full abstract ▾
All authors Yifei Qu, Ru Li, Junjie Chen, Jinyuan Wu
Computer Vision
What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment
Salini Yadav, Taveena Lotey, Pravendra Singh et al. (+1 more)
Zero-shot visual decoding from electroencephalography (
EEG) aims to infer visual semantics from non-invasive neural recordings, but remains challenging due to the low signal-to-noise ratio, non-stationarity, and limited spatial resolution of
EEG. Existing
EEG-
vision alignment methods often rely on holistic
EEG embeddings, which can obscure the complementary temporal, spectral, and spatial structure underlying visual perception. We introduce a unified multiview
EEG representation learning framework for aligning
brain responses with visual semantic
embeddings. Our method builds an
EEG encoder that jointly models three complementary views: input-conditioned state-space temporal dynamics, learnable wavelet-based spectral decomposition for sample-adaptive frequency modeling, and
attention-modulated graph learning for structured electrode interactions. The resulting multiview
EEG embeddings are fused and aligned with pretrained visual representations in a shared semantic space using contrastive learning with
EEG-specific regularization, enabling 200-way zero-shot visual classification. Experiments on THINGS-
EEG benchmark show that our method achieves state-of-the-art performance, with 54.8% Top-1 and 85.6% Top-5 accuracy in the within-subject setting and 15.3% Top-1 and 45.4% Top-5 accuracy in the cross-subject setting. We further present the first systematic cross-session
EEG-image decoding evaluation, achieving 40.8% Top-1 and 78.0% Top-5 accuracy. These results suggest that explicitly modeling multiview neural structure improves both semantic
alignment and generalization in
EEG-based visual decoding.
Read the full abstract ▾
All authors Salini Yadav, Taveena Lotey, Pravendra Singh, Partha Pratim Roy
Computer Vision
Falcon: Functional Assembly and Language for Compositional Reasoning in X-ray
Yonathan Michael, Mohamad Alansari, Natnael Takele et al. (+2 more)
Conventional
vision-
language models are largely object-centric, focusing on detecting and describing individual entities. In safety-critical X-ray baggage screening, however, threat often emerges not from a single object but from the functional compatibility of spatially dispersed components, such as batteries, detonators, and explosive charges. We formalize this setting as compositional threat
reasoning, where risk is modeled as a relational property of grounded regions rather than an independent detection outcome. We introduce Falcon, a
multimodal framework that abstracts
segmentation-aware region features into a structured safety state capturing component presence, pairwise functional compatibility, and scene-level risk. This structured representation is injected into the
language model as an explicit intermediate interface, encouraging relationally consistent and safety-aware
reasoning. To evaluate this problem, we present Falcon-X, a
benchmark that unifies dense grounding with structured supervision over component completeness and risk inference in cluttered X-ray imagery. Experiments show that while existing
multimodal models adapt to appearance, they struggle with compositional safety
reasoning. Falcon improves functional grounding and produces more coherent threat assessments, establishing compositional safety
reasoning as a distinct evaluation paradigm for
multimodal systems.
Read the full abstract ▾
All authors Yonathan Michael, Mohamad Alansari, Natnael Takele, Andreas Henschel, Naoufel Werghi
Computer Vision
Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding
Baiyang Song, Yuli Lin, Qiong Wu et al. (+5 more)
Currently, streaming
video understanding is still a daunting task for existing
multimodal large
language models (MLLMs). Its difficulties not only lie in handling the ever-increasing
video frames, but also in the unpredictability of future
video content and input instructions. In this paper, we study this task from the perspective of constructing a dynamic but fixed-budget
memory bank, and propose a novel and training-free approach termed \emph{CausalMem}. CausalMem is dedicated to constructing a dynamic visual
memory update mechanism, thereby maximizing the amount of information in streaming
video within a limited
memory space, much like the human
brain. In practice, CausalMem estimates the redundancy of visual
tokens and updates the
memory bank via an online semantic basis, which models the principal semantics of the observed
video stream. To validate CausalMem, we apply it to two representative MLLMs, namely LLaVA-OneVision and Qwen2.5-VL respectively, and conduct extensive experiments on both streaming and offline
video understanding
benchmarks. The experimental results not only show the great advantages than existing methods under both streaming and offline settings, e.g., $+3.2\%$ and $+3.0\%$ average accuracy gains respectively, but also witness the superior semantic preservation for streaming
videos, e.g., using 12$k$
token budgets to memorize hour-long streaming
videos, which achieves more than \textbf{20$\times$} visual
token compression ratio and only occupies about 82 MB storage. Our code is given in \href{https://github.com/hktk07/CausalMem}{CausalMem}.
Read the full abstract ▾
All authors Baiyang Song, Yuli Lin, Qiong Wu, Tao Chen, Jun Peng, Xiao Chen, Yiyi Zhou, Rongrong Ji
Computer Vision
Steering Vision-Language Models with Joint Sparse Autoencoders
Huizhen Shu, Xuying Li, Hongxu Lin et al. (+2 more)
Sparse Autoencoders (SAEs) have shown promise for analyzing
language models, but applying them to
vision-
language models (VLMs) often yields representations that are difficult to use as controllable cross-modal steering directions. We introduce the Joint Sparse Autoencoder (JSAE), which uses an explicit
alignment constraint to jointly factorize sequence-pooled
vision and
language activations into shared, interpretable image/caption-level features. Applied to LLaVA, JSAE recovers cross-modal features for recognizable concepts (e.g., food and animals). Through bidirectional interventions (additive steering and suppression), we observe a layer-dependent asymmetry under our protocol: additive steering peaks at mid-to-late (pre-output) layers and weakens at both ends, whereas suppression scores remain within a comparable range across all probed layers within statistical noise. Experiments on three VLMs, namely LLaVA-v1.6-Mistral-7B, Llama3-LLaVA-8B, and the MoE-based Qwen3-VL-30B, show related layer-localized effects across architectures. Together, these results suggest that explicitly aligned sparse representations support more controllable intervention-based analysis of
multimodal features, within an identifiable layer range, than the unconstrained alternatives tested here.
Read the full abstract ▾
All authors Huizhen Shu, Xuying Li, Hongxu Lin, Wenjie Sun, Hui Li
Computer Vision
Auto-Labelling-Based Domain Transfer for 3D Object Detection on a Bicycle-Mounted LiDAR Platform
Mario Finkbeiner, Max A. Buettner, Kanak Mazumder et al. (+1 more)
Reliable 3D perception of vulnerable road users (VRUs) such as cyclists and pedestrians is essential for their safety in urban traffic and a core requirement for autonomous driving (AD). Alongside advances in vehicle-based perception, research increasingly equips bicycles with sensors to study traffic from a perspective native to VRUs. Such platforms still rely on
LiDAR detectors originally trained on vehicle data, yet annotated 3D data from a cyclist's perspective is scarce. How well these detectors generalise to this setting has not been evaluated. We present a 3D object detection
benchmark of 1,027 annotated
LiDAR keyframes (over 18,000 3D bounding boxes) from the FUSE-Bike platform in urban Munich. We evaluate four nuScenes-pre-trained detectors against 1,854 human-verified ground-truth (GT) boxes both in their original form and after finetuning on training labels produced by a VRU-dedicated auto-labelling pipeline that requires no manual annotation. The zero-shot domain gap is concentrated on the VRU classes. Finetuning recovers most of it, improving mean average precision (mAP) by up to 23.4 points with the largest gains on pedestrians and cyclists, and the adapted detectors even surpass the quality of the auto-labels they were trained on. The
benchmark provides a reproducible baseline for VRU-centric 3D detection and shows that auto-labels are a viable substitute for manual annotation when adapting vehicle-trained detectors to a cyclist platform.
Read the full abstract ▾
All authors Mario Finkbeiner, Max A. Buettner, Kanak Mazumder, Fabian B. Flohr
Computer Vision · Cross-listed from cs.RO
Calousel: Extrinsic Calibration of Non-overlapping Multi-camera Systems from Pure Rotation
Gwanhyeong Song, Chaehyeon Song and Ayoung Kim
Extrinsic calibration of multi-camera systems with non-overlapping FOVs has been a challenging problem in the robotics literature. Conventional target-based methods impose substantial target setup overhead, either deploying large calibration targets or requiring pre-measured multi-target poses. Motion-based approaches instead suffer from drift error, scale ambiguity, and motion degeneracy. Securing both accuracy and usability, we propose a novel calibration method that leverages pure rotational motion, requiring only a single static calibration board. The key idea is to make all cameras sequentially observe the same target under a shared
geometric reference, even without overlapping views. To integrate these time-separated observations, we formulate the problem using a latent turntable frame and a 3D error on SE(3) within a global optimization framework. We validate the proposed method on both a controlled camera rig and a full-scale vehicle platform with heterogeneous cameras, and analyze robustness under non-ideal turntable motion. Extensive experiments show that our approach maintains competitive accuracy without specialized precision hardware, proving its strong suitability for realistic on-site deployments. Our code is publicly available here.
Read the full abstract ▾
Computer Vision
SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessity
Tianchen Guo, Chen Liu, Ling Chen et al. (+1 more)
Multimodal Large
Language Models (MLLMs) have shown remarkable progress in single-image perception, yet their ability to reason about complex cross-view human-centric scenes remains largely unverified. Current multi-view
benchmarks evaluate models using a fixed "bag of frames" and thus conflate a model's robustness to visual distraction with its genuine ability to fuse fragmented cross-view evidence. To address this issue, we introduce SSMNBench, a diagnostic
benchmark comprising 3,300 curated QA pairs for cross-view human and human-object understanding. SSMNBench uniquely categorizes tasks into Single-View Sufficiency (SVS) and Multi-View Necessity (MVN). By systematically perturbing view availability across 17 state-of-the-art MLLMs, critical limitations are revealed: models suffer from severe "distraction degradation" when presented with redundant views (SVS), and fail to integrate fragmented
geometric evidence across cameras (MVN). Our evaluations demonstrate that modern MLLMs rely on multiple single-image semantic averaging and view preference rather than genuine cross-view synthesis. By exposing these fundamental vulnerabilities, SSMNBench provides a rigorous diagnostic framework to drive the advancement of future cross-view-aware
multimodal architectures. The code is available at: $ \href{https://github.com/gtc-gh/SSMNBench}{\text{SSMNBench}} $
Read the full abstract ▾
All authors Tianchen Guo, Chen Liu, Ling Chen, Xin Yu
Computer Vision · Cross-listed from cs.RO
1000 Rallies: An Event-Camera Dataset and Real-Time Learned Ball-State Estimation for Robotic Table Tennis
Raphaela Kreiser, Asude Aydin, Yin Bi et al. (+3 more)
Robotic table tennis has emerged as a compelling
benchmark for real-time robotic perception due to its fast ball dynamics and stringent timing requirements. Accurate, high-frequency, and low-latency ball state estimation is critical for reliable trajectory prediction and timely control. Traditional frame-based cameras face an inherent trade-off: low frame rates leave temporal blind spots that miss fast-moving objects and high frame rates raise data and computational cost. Event cameras instead offer microsecond temporal resolution and, under sufficient illumination, remain largely free of motion blur even at high ball speeds. However, the community lacks large-scale
datasets to develop and
benchmark event-based perception in realistic sports scenarios. We address this gap by introducing the first large-scale event-camera
dataset for table tennis, comprising over 1000 rallies from a diverse group of players ranging from amateurs to elite-level athletes. Each recording captures the event stream alongside 14 synchronized high-speed frame-based cameras at 200 FPS, which we use to produce 1 kHz pseudo ground-truth labels for ball position, velocity, and
spin. Building on this
dataset, we train a convolutional neural network robust to background player motion that jointly estimates the ball's position and velocity in the image-plane from events. Treating the predicted velocity as an additional measurement in the Kalman filter reduces bounce-point prediction error by 36% relative to a position-only baseline. Finally, we close the perception-action loop by integrating the event-based system with a Stäubli robotic arm, enabling the first real-time human-
robot table tennis rallies driven by event-based perception.
Read the full abstract ▾
All authors Raphaela Kreiser, Asude Aydin, Yin Bi, Claudio Fanconi, Peter Dürr, Naoya Takahashi
Computer Vision
ScaleHP: Scale-Mediated Optimization of Coupled Errors for Metric-Space Hand Pose Estimation
Ruitao Jing, Xingyu Chen, Hongyang Li et al. (+3 more)
In this paper, we present ScaleHP, a unified framework that explicitly represents per-instance metric scale to resolve the coupled errors in calibrated camera-space hand pose estimation. Under the common root-relative-to-global paradigm, camera-space accuracy depends jointly on relative geometry, root localization, and their scale-dependent composition. ScaleHP treats scale as the shared interface among these terms rather than optimizing them in isolation. Its metric-aware 2D-3D
Transformer decoder introduces a dedicated scale
token that exchanges information with sparse, semantically indexed 2D/3D joint queries, allowing global metric
reasoning and local joint geometry to mutually refine one another. A parameter-free calibrated module then recovers root translation from the predicted unit-scale joints, metric scale, and known camera intrinsics. ScaleHP demonstrates state-of-the-art CS-MPJPE on FreiHand (35.8 mm) and, under
benchmark-specific training protocols, state-of-the-art PA-MPJPE on FreiHand, DexYCB, and HO3Dv3 (5.0/4.6/5.9 mm), showing that improved global metric localization is attained together with improved local articulation. Ablations further show that scale-pose interaction improves both camera-space accuracy and relative depth geometry.
Read the full abstract ▾
All authors Ruitao Jing, Xingyu Chen, Hongyang Li, Qing Jiang, Yukai Shi, Lei Zhang
Computer Vision
Expresso-AI: Explainable Video-Based Deep Learning Models for Depression Diagnosis
Felipe Moreno, Sharifa Alghowinem, Hae Won Park et al. (+1 more)
Given the widespread prevalence of depression and its consequential impact on individuals and society, it is crucial to obtain objective measures for early diagnosis and intervention. As a multidisciplinary topic, these objective measures should be interpretable and accessible to health care professionals, ensuring effective collaboration and treatment planning in the realm of mental health care. Even though current automated depression diagnosis approaches improved over the last decade, a critical gap exists as they often lack affect-specificity and interpretability, limiting their practical application and potential impact on mental health care. In particular, interpretability from temporal activities from
videos when deep models are used is not fully explored. In this study, we present a novel framework for analyzing Deep Neural Networks' decisions when trained on facial
videos, specifically focusing on automatic depression severity diagnosis. By fine-tuning Deep Convolutional Neural Networks (DCNN) pre-trained on Action Recognition
datasets on depression severity facial
videos from AVEC depression
dataset, our framework is able to interpret the model's saliency maps by examining face regions and temporal expression semantics. Our approach generates both visual and quantitative explanations for the model's decisions, providing greater insight into its
reasoning. In addition to this interpretability, our
video-based modeling has improved upon previous single-face
benchmarks for visual depression diagnosis, resulting in enhanced predictive performance. Overall, our work demonstrates the successful development of a framework capable of generating hypotheses from a facial model's decisions while simultaneously improving depression's predictive capabilities.
Read the full abstract ▾
All authors Felipe Moreno, Sharifa Alghowinem, Hae Won Park, Cynthia Breazeal
Computer Vision
VPA-Guard: Defending and Benchmarking Image-to-Video Generation Against Visual Prompt Attacks
Yining Sun, Haoyu Kang, Jiajun Wu et al. (+7 more)
Recent advancements in Image-to-
Video (I2V) generation have transformed input images from simple appearance references into interactive control interfaces where visual cues such as arrows, sketches, and emojis orchestrate complex
video dynamics with unprecedented controllability. However, these seemingly innocuous static cues can be interpreted by models as executable temporal instructions, unfolding into harmful actions in the generated
videos. Despite the severity of this threat, existing safety
benchmarks remain predominantly focused on text-based and content-only image-based jailbreaks, leaving implicit visual
prompt attacks insufficiently explored. To bridge this gap, we present VVA-Bench, the first systematic
benchmark for evaluating
video generation safety under categorized
vision-centric
prompt attacks. Extensive experiments on VVA-Bench demonstrate that state-of-the-art models are highly susceptible to such attacks, with Attack Success Rates (ASR) reaching 100.0% on Wan 2.7 and 74.8% on Veo 3.1. To mitigate these risks, we propose VPA-Guard, a
retrieval-augmented and self-evolving defense framework. By leveraging few-shot
reasoning to identify latent malicious intents, our method reduces the attack ASR by 44.2% and the harmfulness score by 73.4% on average, while maintaining the model's utility for legitimate user edits. Our work provides both a rigorous
benchmark and an effective defense strategy to advance safe and socially responsible
multimodal generation.
Read the full abstract ▾
All authors Yining Sun, Haoyu Kang, Jiajun Wu, Heng Zhang, Danyang Zhang, Zhenjun Zhao, Haochen Han, Fangming Liu, Wai Kin Victor Chan, Alex Jinpeng Wang
Computer Vision
FeVOS: Foresight Expression Video Object Segmentation
Kehan Lan, Kaining Ying and Henghui Ding
Existing Referring
Video Object
Segmentation tasks focus on referring expressions describing events, actions or appearances of relevant objects within the observed frames, lacking evaluation in scenarios that require pre-decisive spatio-temporal
reasoning, thereby limiting their applicability. To address this, we propose Foresight Expression
Video Object
Segmentation, a task that queries future events in upcoming
video segments and requires masks of the objects in the observed frames as visual answers. For example, in ego-centric scenes, the question "What tool will be used?" demands
reasoning over spatio-temporal cues to predict the masks of the next tool to be used, which helps with the understanding of future actions and decisions. To support this task, we introduce FeVOS, a
dataset with 968
video clips, 14,525 foresight expressions, and 2,904 chain-of-thought annotations to provide explicit and interpretable
reasoning steps. We further develop FeVOS-R1, an MLLM-based model trained on our
dataset via a two-stage pipeline of supervised fine-tuning and reinforcement learning. FeVOS-R1 not only achieves state-of-the-art performance on FeVOS, but also demonstrates strong generalization to existing RVOS
benchmarks. We hope this work can inspire more research on predictive
reasoning in
video perception.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Cross-Attention Multimodal Learning for Predicting Response to Neoadjuvant Imatinib in Gastrointestinal Stromal Tumors: A Multicenter Retrospective Study
Fariba Tohidinezhad, Douwe J. Spaanderman, Natalia Oviedo Acosta et al. (+14 more)
Background: Response to neoadjuvant imatinib in gastrointestinal stromal tumors (GISTs) is highly variable and cannot be reliably predicted using current clinical or molecular markers. This study developed and evaluated an explainable
multimodal deep learning framework integrating computed
tomography (CT) imaging and clinical variables to predict treatment response. Methods: Patients from four tertiary centers were retrospectively included between 2000-2023 in independent pretraining (n=935) and prediction (n=213) cohorts. A cross-
attention framework integrating clinical variables and tumor-centered CT imaging was developed to predict response to neoadjuvant imatinib. Two training strategies were evaluated: (1) self-supervised pretraining with low-rank adaptation and (2) training from scratch. Hyperparameters were optimized using SMAC3. Performance was assessed through internal cross-validation and external testing. Ablation analyses and
attention-based explanations were used to quantify modality contributions. Results: Among 213 patients (54.5% responders), responders had larger tumors (112 vs. 89 mm, P=0.026), higher mitotic index (3 vs. 0, P<0.001), and more frequent KIT mutations (69.0% vs. 56.7%, P=0.019). Cross-
attention models achieved the highest internal performance (AUC up to 0.99) but lower external performance (AUC 0.60-0.63). Clinical-only performance was moderate (AUC 0.66), whereas imaging-only models showed limited generalizability (AUC 0.56-0.66). Explainability analyses identified significant differences in feature importance between responders and non-responders, including CD117, BRAF, PDGFRA, age, sex, disease status, and comorbidities (FDR-adjusted P<=0.036). Conclusion: The cross-
attention framework shows potential for improving imatinib response prediction in GIST while providing interpretable insights into
multimodal determinants of treatment response.
Read the full abstract ▾
All authors Fariba Tohidinezhad, Douwe J. Spaanderman, Natalia Oviedo Acosta, Kaouther Mouheb, Karthik Prathaban, David F. Hanff, Dirk J. Grünhagen, Cornelis Verhoef, Joris M. van Sabben, Evelyne Roets, Jette J. Slettenhaar, Hans Gelderblom, Ingrid M.E. Desar, Anna K.L. Reyners, Neeltje Steeghs, Stefan Klein, Martijn P.A. Starmans
Computer Vision
H-Adapter: Pose-Robust Hairstyle Transfer via Attention-Derived, Source-Aligned Hair Masks
Seulgi Jeong, Yunseong Cho and Sanghun Park
Hairstyle transfer has practical applications such as virtual try-on, yet remains challenging when the source and reference exhibit large head-pose discrepancies. We propose H-Adapter, which improves pose robustness by training with a region-specific loss that disentangles hair and non-hair objectives and thereby induces spatially disentangled cross-
attention, from which a source-aligned hair edit mask is derived to guide
diffusion-based inpainting. Experiments on pose-agnostic and pose-different subsets demonstrate strong quantitative results, including the best FID, $\mathrm{FID}_{\mathrm{CLIP}}$, and CLIP-I under pose differences, while maintaining competitive non-hair preservation and improving qualitative fidelity to fine-grained reference hairstyle details. Beyond source-conditioned transfer, H-Adapter supports practical extensions including text-to-image generation, auxiliary
prompt-based hair color control, and compatibility with an identity-preserving IP-Adapter variant. We also introduce a VLM-as-a-judge protocol and observe consistent gains in hairstyle faithfulness, non-hair preservation, and artifact quality.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AR
Energy-Efficient CNN Acceleration with MSDF Digit-Serial Arithmetic on FPGA
Muhammad Usman, Yousef Sadegheih and Dorit Merhof
This paper presents an energy-efficient hardware acceleration of the convolutional layers in the U-Net architecture for image
segmentation, implemented on FPGA. While digit-serial arithmetic, particularly most-significant-digit-first (MSDF) techniques, offers a compact hardware footprint, it suffers from initial latency before producing the first output digit. This delay accumulates in cascaded operations like multiplication followed by addition, where each unit introduces its own startup overhead. To overcome this, we propose a merged multiply-add (MMA) architecture that fuses these operations into a unified pipeline. Instead of incurring separate delays, the MMA introduces a single streamlined latency per iteration, shorter than the combined latency of conventional cascaded units, resulting in enhanced throughput and efficiency. The MMA units are designed to process spatial input depths in parallel, achieving significantly higher performance than both standalone MSDF-based and conventional designs. We evaluate the proposed design using U-Net as a target application. Despite operating at a lower frequency than a CPU, the FPGA-based accelerator achieves up to an order of magnitude higher energy efficiency, delivering up to $15.14$ GOPS/W compared to $1.93$ GOPS/W for CPU-based inference. The design also shows approximately $9\times$ reduction in energy consumption compared to MSDF-based FPGA implementations. These results highlight the efficacy of the merged arithmetic approach for resource-constrained, latency-sensitive edge applications in medical imaging and computer
vision.
Read the full abstract ▾
Computer Vision
Concept Removal for Frontier Image Generative Models
Aditya Kumar, Pierre Joly, Adam Dziedzic et al. (+1 more)
Image generative models are trained on massive, largely uncurated internet-scale
datasets that contain undesirable visual concepts. Efficiently removing such concepts from the model generations without degrading the quality of output images remains challenging. We introduce a novel concept removal method for frontier
diffusion and image autoregressive models, such as SD3.5, Flux, and Infinity. Our intervention replaces the internal bottleneck layer present in all these modern models with a transcoder that is trained to replicate the original layer while structuring it into distinct activation features. This in-place substitution creates an integrated filter through which concept-specific signals can be selectively disabled while preserving the rest of the model's behavior. Since the intervention modifies the model backbone rather than attaching an external component, it remains persistent under white-box access. Empirically, the approach achieves state-of-the-art concept removal performance across modern
diffusion and autoregressive models, maintains visual generation quality, provides robustness against
adversarial prompts, and supports sequential removal of diverse concepts. This positions our method as a practical approach for concept removal in frontier image generative models.
Read the full abstract ▾
All authors Aditya Kumar, Pierre Joly, Adam Dziedzic, Franziska Boenisch
Computer Vision
Efficient Cross-Scale Invertible Hiding Network with Spatial-Frequency Collaboration and Non-Invertible Mechanism
Junxue Yang and Xin Liao
Image hiding aims to conceal image-level messages within cover images at the same resolution. Invertible neural networks (INN)-based image hiding has emerged as an important branch. It treats concealing and revealing as a pair of inverse problems on image domain transformation and uses INN's forward and backward processes to address them. Due to architectural constraints, existing INN-based methods suffer from single-scale and single-domain feature extraction and limited nonlinear representation capability, resulting in inferior image quality. To mitigate these limitations, we propose an efficient cross-scale invertible hiding network with the spatial-frequency collaboration and the non-invertible mechanism, termed CrosInv. CrosInv exploits cross-scale and spatial-frequency collaborative features while enhancing nonlinear representation. Specifically, we introduce a cross-scale invertible module that bijectively maps inputs to cross-scale representations. To effectively integrate spatial and frequency information, the cross-scale invertible module employs pixel shuffle, Haar wavelet transformation, and their inverse operations for scale transformation. Furthermore, a non-invertible cross dense module is integrated to enhance the nonlinearity. Comprehensive experiments verify the effectiveness and superiority of the proposed CrosInv.
Read the full abstract ▾