Computer Vision · Cross-listed from cs.CL
Modality-Gated Deep Adapters: Adding a Modality to a Frozen Embedding Model with Exact Preservation
Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md. Shahrier Islam Arham et al. (+1 more)
Multimodal embedding models are deployed at scale:
retrieval indices,
benchmark results, and behavioral audits all depend on the base model's exact outputs. Extending such a model to a new modality with existing parameter-efficient methods silently changes those outputs; LoRA-style adaptation rewrites the text path whether or not the weights are merged, invalidating every stored
embedding. We propose modality-gated deep adapters: bottleneck adapters attached to every decoder layer of a frozen
multimodal embedding LLM, grouped into per-modality packs that execute only while their own modality is being encoded. The result is a modality added with zero change to existing outputs: inputs no pack claims traverse the base model's own computation graph, bit-for-bit unchanged, and co-loaded packs compose with an exact-zero isolation
matrix. Both properties are stated as propositions, hold after arbitrary training rather than only at initialization, require no task labels or routing metadata at inference, and are verified by exact-equality tests on the released checkpoints. On one frozen 2B base, the audio pack (injected as connector
tokens) improves audio-to-text R@10 by +3.4 to +5.4 points over an identically trained control, positive at every seed and reproduced at eleven times the data; the thermal pack, reusing the base's own frozen
vision path, clears its pre-registered acceptance gate roughly sevenfold at every seed and lifts thermal-to-text R@10 from 0.224 to 0.785. An encoder swap locates the missing capacity: an external audio encoder that outranks Whisper-family encoders in CLAP-style comparisons loses by 16 R@10 points inside the frozen LLM, so the capacity belongs in the layers, exactly where the gated adapters place it. We release the audio model, the thermal pack, and the training, evaluation and invariance suites: models at huggingface.co/EximiusLabs, code on GitHub.
Read the full abstract ▾
All authors Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md. Shahrier Islam Arham, Arman Luthra
Computer Vision
On Cross-Validation for Hyperparameter Optimization of Deep Learning Image Classifiers
Ljubomir Buturovic
Hyperparameter optimization (HPO) can materially affect the performance of deep learning (DL) image classifiers, but there is little empirical guidance on how to derive the validation signal that drives it, especially for the small sample sizes common in fields such as medical imaging. We compared three HPO protocols in terms of absolute performance-estimation error (AEE; the absolute difference between the winning configuration's validation AUROC and its test AUROC): fixed holdout (F), reshuffled holdout (R), and 5-fold cross-validation (C). The search space, sampler, training procedure, architecture, and test set were held identical across protocols. We evaluated the protocols on three public
datasets spanning two regimes: binary medical imaging (RSNA pneumonia radiographs and binarized HAM10000 skin lesions) and 200-class natural imaging (Tiny ImageNet), across a range of development set sizes $n$ and two backbones (ResNet-18 on all
datasets,
Vision Transformer (ViT-S/16) on RSNA). On the medical
datasets, every point estimate favored cross-validation over both holdout protocols, with reductions in AEE largest at small sample sizes and diminishing as $n$ increased. This pattern remained robust under conservative family-wise adjustment. On Tiny ImageNet, AEE was negligible under all three protocols. Test AUROC was generally similar among protocols. Fixed holdout had lower mean AEE than reshuffled holdout in 11 of 12 medical conditions, although this secondary finding was less uniformly supported. For small-sample medical image classification, we recommend cross-validation-based HPO when computational resources permit because it trades additional computation for a more reliable development-time estimate of subsequent test performance.
Read the full abstract ▾
Computer Vision
Deep Analog: Open-Set Film Emulation with Reference-Conditioned 3D LUTs
Yitong Mu
Film emulation reproduces the look of an analog film stock on a new digital photograph. We target its open-set form – matching any reference film frame from a single example – with a 3D lookup table (LUT) predicted from that reference. Real-time image enhancement predicts per-image weights over a fixed bank of 3D LUTs and blends them. We show this is a gated mixture of experts and inherits its failure: trained end-to-end against reconstruction, the gate collapses onto a single expert, so a bank of K LUTs delivers the capacity of one. An
entropy term, the enhancement-setting analogue of mixture-of-experts load balancing, restores utilization and recovers about 1 dB PSNR. The deeper constraint survives: a fixed LUT basis is closed-set, freezing the achievable looks at training time. We therefore discard the basis and predict a single 3D LUT as a residual from a reference image (StyleLUTNet), trained by self-supervision on procedurally generated color transforms. The conditional design removes the gate and generalizes open-set to unseen film stocks without paired data or retraining. Around this color backbone we build Deep Analog, a film-emulation pipeline that adds histogram-based tone matching and a physics-informed optical renderer – multi-scale grain and per-channel halation driven by parameters an inverse network regresses from the reference. On 350 self-supervised pairs the color stage reaches 22.05 dB PSNR / 0.925 SSIM and the full pipeline 21.72 dB / 0.923; the color path runs in 5.2 ms at 1080p (192 FPS) and exports a portable .cube LUT for standard editing tools. A second degeneracy in conditional LUT training – residual-scale collapse – shares the root cause and yields a general principle: auxiliary regularization must stay subordinate to reconstruction.
Read the full abstract ▾
Computer Vision
Periocular Soft Biometrics: A Survey and Applications to Multimedia Forensics and Disinformation Detection
Fernando Alonso-Fernandez, Kevin Hernandez-Diaz and Josef Bigun
Soft-biometric attributes such as gender, age, and ethnicity provide valuable ancillary evidence when full identity recognition is not feasible, supporting applications in forensic investigation, identity verification, surveillance, or detection of synthetic and manipulated media. Among biometric modalities, the periocular region is a robust source of soft-biometric cues, as it often remains visible when other parts of the face are occluded, a frequent condition in forensic evidence and surveillance footage, and can be captured across a wide range of acquisition conditions. In this paper, we provide a survey of demographic attribute estimation from periocular images, covering publicly available
datasets, methodological trends from handcrafted descriptors to deep learning architectures, and the state of the art in gender, age, and ethnicity prediction. We discuss use cases relevant to multimedia forensics and disinformation-detection applications, including demographic filtering in surveillance footage, age verification, and the detection of demographic inconsistencies in synthetic data. We also highlight open challenges, including
dataset bias, cross-domain generalisation,
fairness, ethical aspects, and the lack of forensic-oriented
benchmarks.
Read the full abstract ▾
Computer Vision
Xemo-Talker: Unlock Emotions Explicitly for Audio-Driven Talking Portrait Synthesis
Chaolong Yang, Yinuo Guo, Kai Yao et al. (+6 more)
Precise
emotion control in audio-driven talking heads remains a challenge due to the reliance on implicit
emotion regulation in existing systems, which often leads to indirect and insufficient control. Additionally, training with explicit
emotion-related losses across the entire motion space poses significant difficulties due to the inherent trade-off between accurate lip synchronization and fine-grained
emotion control. In this paper, we reveal a key finding: although emotional cues are distributed throughout the motion space, concentrating discriminative supervision on less-principal components achieves a better
emotion-lip synchronization balance, as principal components mainly encode high-energy articulation and pose variations. Building on this insight, we propose Xemo-Talker, which first learns a neutral
speech-to-motion mapping for stable articulation and lip synchronization, and then introduces a lightweight
emotion branch guided by less-principal subspace supervision. To enhance
emotion control, we design a Tri-Loss consisting of inter-class separation, intra-class compactness, and less-principal contrastive learning. Given an audio input, a reference image, and an
emotion label, Xemo-Talker achieves state-of-the-art
emotion classification accuracy while maintaining competitive lip synchronization and high inference efficiency, with performance approaching that measured on real
videos.The source code is publicly available at https://github.com/chaolongy/Xemo-Talker.
Read the full abstract ▾
All authors Chaolong Yang, Yinuo Guo, Kai Yao, Yuyao Yan, Jie Sun, Guangliang Cheng, Shibin Wu, Bin Dong, Kaizhu Huang
Computer Vision · Cross-listed from eess.IV
Decodable but Not Accessible: Auditing Distance-Based Reliability Estimation on Disentangled Skin-Lesion Representations
Duc-Vinh Tran
Distance-based reliability estimation assumes that a representation's geometry reflects its trustworthiness, yet this assumption is rarely tested under training interventions that reshape geometry directly. We audit this assumption under domain-
adversarial representation learning using a disentanglement dose-response ladder. Three checkpoint families share the same architecture and a 16-dimensional representation, differing only in orthogonality strength (lambda = 0, 1, 5). Representation geometry changed substantially with disentanglement strength: the condition number shifted by two orders of magnitude (Kendall tau = 0.84, exact p = 2.8e-5). This change was not accompanied by improved reliability estimation: Mahalanobis-distance AUROC (ISIC-test vs. PAD-UFES) remained flat and below chance (about 0.40) at every level, with no significant association with any of five geometry metrics tested. The same failure was observed for cosine-to-centroid and pooled k-nearest-neighbor scorers, plus three non-distance-based scorers: an energy-based confidence score, Virtual-Logit Matching, and a kernel density
estimator. Seven of eight scorers converged on the same result; the energy-based score showed an isolated upward trend that we report but do not treat as evidence against the overall pattern. A supervised probe with no access to the training objective recovered domain membership from the identical
embeddings at 0.72-0.81 AUROC across every level, showing that the relevant information was not absent from the representation. These findings indicate that classification performance alone can overlook whether information in a learned representation is organized in a form that downstream reliability
estimators can use. Information can remain decodable while becoming largely inaccessible to non-probing reliability
estimators.
Read the full abstract ▾
Computer Vision
GeoUniPR: A Geometry-Consistent Unified Framework for Cross-Modal Place Recognition
Wonbong Kim, Jiatong Xiao, Rui Li et al. (+5 more)
Cross-modal place recognition (CMPR) aims to identify the same location across heterogeneous sensing modalities, such as
vision and
LiDAR. Existing methods commonly bridge the modality gap using complex
alignment modules, multi-stage training, or full fine-tuning of pretrained backbones. In this work, we revisit CMPR from the perspective of
geometric consistency and propose GeoUniPR, a unified and concise geometry-consistent framework. GeoUniPR reduces cross-modal discrepancy at the representation level by projecting
LiDAR point clouds into the camera perspective to construct Geometry-Consistent depth image views (DIV), which establish direct RGB-
LiDAR correspondence. We further augment DIV with native
LiDAR cues, including intensity and surface-normal information, yielding a multi-channel
geometric representation that improves structural consistency. Based on this representation, GeoUniPR learns a unified
embedding space using two modality-specific ViT-based encoders with identical architectures, trained through parameter-efficient adaptation without auxiliary
alignment modules, multi-stage training, or full backbone fine-tuning. In addition, we introduce Spatially-Consistent InfoNCE (SC-InfoNCE), a CMPR-specific contrastive objective that suppresses distance-induced false negatives under spatial continuity. Extensive experiments on KITTI and KITTI-360 demonstrate that GeoUniPR achieves state-of-the-art (SOTA) performance in both same-modal and cross-modal place recognition, with strong cross-
dataset generalization.
Read the full abstract ▾
All authors Wonbong Kim, Jiatong Xiao, Rui Li, Xufei Wang, Qiwen Gu, Junqiao Zhao, Chen Ye, Guang Chen
Computer Vision
From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoning
Han Zhang, Yilin Zhao, Zaid Pervaiz Bhat et al. (+5 more)
Detecting a traffic anomaly does not establish whether a
video-
language model can explain what happened, localize it in time, or identify its causes. We introduce TAR (Traffic Anomaly
Reasoning) and TAR-Bench, paired resources for training and evaluating these complementary capabilities across 10 tasks spanning question answering, temporal
reasoning, and scene understanding. TAR contains 44,040 automatically generated annotations with chain-of-thought traces for 3,670 CCTV
videos from eight public
datasets. TAR-Bench provides 960 human-curated annotations for 80 held-out clips from 17 public YouTube
videos. Evaluation of eleven
vision-
language models reveals a gap between question-answering performance and temporal or scene
reasoning. Progressively adding task groups during supervised fine-tuning improves aggregate performance on both Cosmos-Reason2-8B and Qwen3-VL-8B-Instruct. Training on all 10 tasks raises their mean
benchmark scores from 34.3 to 55.7 and from 30.9 to 53.9, respectively. These results support joint training across complementary tasks as a promising approach to traffic anomaly understanding, while highlighting persistent limitations in temporal precision and causal attribution. TAR and TAR-Bench serve as the official training and in-domain evaluation resources for AI City Challenge 2026 Track 3.The
dataset is available at https://huggingface.co/
datasets/nvidia/PhysicalAI-Traffic-Anomaly-
Reasoning
Read the full abstract ▾
All authors Han Zhang, Yilin Zhao, Zaid Pervaiz Bhat, Zheng Tang, Varun Praveen, Vidya N. Murali, David C. Anastasiu, Tomasz Kornuta
Computer Vision
UniMod: Enhancing Multi-Modal Medical Diagnosis through Cross-Modality and Within-Modality Alignment
Zijian Gu, Weikai Lin, Shuang Zhou et al. (+2 more)
Multi-modal learning combining medical images and clinical text is promising for disease diagnosis. However, standard multi-modal training leads to shortcut learning: models exploit the easier modality (e.g., diagnostic cues in text) while neglecting harder-to-learn features (e.g., subtle visual patterns). We propose UniMod, a framework that mitigates shortcut learning by requiring each modality to predict the diagnosis on its own. It supervises image-only, text-only, and multi-modal classification simultaneously, so each modality must extract diagnostic features. We add cross-modality
alignment for knowledge transfer and within-modality supervised contrastive
alignment over same-diagnosis patients. On Harvard-Glaucoma, UniMod reaches 0.850 AUC, outperforming OGM-GE and Gradient Blending by 1.6-1.8%; on CheXpert Plus, it reaches 0.966 AUC, surpassing them by over 5%. UniMod also extends to 5-class multi-label diagnosis without architectural change, improving mean AUC by 0.097 over CGGM.
Read the full abstract ▾
All authors Zijian Gu, Weikai Lin, Shuang Zhou, Zihan Chen, Song Wang
Computer Vision
Site Is Decodable Before Pretraining: Negative Controls for Probing Frozen Brain-MRI Foundation Models
Saman Rahbar
Brain foundation models are often tested with a probe. The model is frozen, a simple classifier is trained on its output, and the classifier's accuracy is taken as a sign of what pretraining learned. We show that this reading can be wrong unless two controls are reported with it. We probed three frozen 3-D
brain-MRI models at five depths, on two independent multi-site cohorts (ABIDE-I and ABIDE-II). The site where a scan was acquired could be predicted at about 0.9, on a scale where 0 is chance and 1 is perfect. No clinical or demographic target came close. An untrained copy of the same model predicted site just as well on both cohorts. Untrained models of three architectures, each built with three random seeds, did the same on ABIDE-I. The raw image, shrunk to 12x12x12 voxels and given to no model at all, predicted site at 0.95. Pretraining did not create the site signal. It was already in the scans. We also looked for any clinical target on which pretraining beats an untrained model, and found no reliable case. Adjusting for age, sex and diagnosis left site prediction almost unchanged, and so did a 4.6-fold increase in the number of voxels. Resting-state fMRI from the same people showed the same pattern. Site was the most predictable attribute of
brain connectivity (0.81), and it survived five untrained layers while age did not. We recommend reporting two controls with every probe of a
brain foundation model: running the probe again on the untrained model, and on the raw input. We release the code for both.
Read the full abstract ▾
Computer Vision
MRIComp4Flow: Compression of 3D Brain MRI for Training Multi-Modal Generative Models
Lisa K. Fischer, Mykhailo Riabets, Daniel Rueckert et al. (+3 more)
Large-scale multi-modal MRI
datasets impose substantial storage and I/O costs, limiting the training of 3D generative models on commodity infrastructure. While lossy compression is known to preserve accuracy for discriminative
segmentation networks, its effect on generative models, which must learn the full data distribution rather than a decision boundary, is unexplored. We study whether standard image codecs can effectively compress semantically rich
brain tumor MRI while preserving the fidelity required to train and deploy a 3D MRI generative model. Each 3D volume is compressed with JPEG2000 or a near-lossless JPEG-LS pipeline. Next, a Wavelet Flow Matching model, conditioned on BraTS image sequences (T1n, T1c, T2, T2f), is trained on compressed data, and the resulting models are evaluated on the validation set. At a 20:1 compression ratio, synthesis quality is statistically equivalent to a model trained on uncompressed data within a pre-specified margin ($\Delta$PSNR $<1$,dB, $\Delta$SSIM $<0.02$; paired TOST $p=[[p]]$): mean PSNR is 27.3,dB vs. 27.0,dB and mean SSIM is 0.95 vs. 0.96 across modalities. Our results indicate that JPEG2000 compression is a practical step toward scalable 3D MRI generative modeling without degrading synthesis quality. The codebase is available at https://github.com/lisafis/MRIComp4Flow .
Read the full abstract ▾
All authors Lisa K. Fischer, Mykhailo Riabets, Daniel Rueckert, Benedikt Wiestler, Anke Meyer-Baese, Sandeep Nagar
Computer Vision
SeFaR: Semantic Feature-aware Robustness Testing of Deep Neural Networks
Nusrat Jahan Mozumder, Divya Gopinath, Corina Pasareanu et al. (+1 more)
Deep neural networks are increasingly deployed in safety-critical domains as perception modules, where failures are often caused due to rare and under-represented scenarios. This necessitates the need to evaluate the semantic robustness of perception models; conformance of behavior to high-level requirements over real-world perceptual variability. To address this, we propose SeFaR, a framework for systematic semantic-feature-centric testing of
vision models. Given a natural-
language requirement and a set of satisfying inputs, SeFaR evaluates robustness with respect to diverse realistic semantic variations that preserve requirement satisfaction. The approach employs a novel hierarchical concept model enabling structured exploration of the feature space and incorporation of domain knowledge via user-defined concepts. State-of-the-art
diffusion and
vision-
language models are leveraged to generate photorealistic semantics-preserving perturbations and identification of previously unknown features impacting behavior. A feedback-driven adaptive process is adopted to generate interpretable failure-inducing semantic concepts along with corresponding test inputs. Evaluation on case studies demonstrates that the proposed framework effectively satisfies requirement preconditions while identifying requirement-independent features that influence model decisions, enabling it to both uncover faults and relate them to such features.
Read the full abstract ▾
All authors Nusrat Jahan Mozumder, Divya Gopinath, Corina Pasareanu, Matthew Dwyer
Computer Vision
TRACE-GS: On-Policy Trajectory Distillation with Privileged Geometric Conditioning for Sparse-View 3DGS Restoration
Linlian Jiang, Yuchen Xi, Sadman Rakib Pinon et al. (+3 more)
We present TRACE-GS, an on-policy trajectory
distillation framework that leverages privileged
geometric conditioning at training time, thereby adapting a
diffusion prior to sparse-view 3D Gaussian Splatting (3DGS) restoration. Rather than pursuing increasingly sophisticated restoration architectures, we identify a more fundamental limitation shared by existing
diffusion-based approaches: supervision at independently noised states does not cover those reached during inference. In sparse-view 3DGS, under-constrained geometry biases denoising from the outset, and the resulting deviations compound along the rollout. TRACE-GS instead performs on-policy trajectory
distillation: a teacher conditioned on richer geometry from additional training views supplies targets along the sparse-view student's own rollout, aligning denoising directions and cross-view responses at each visited state. This training-only geometry places TRACE-GS in the learning using privileged information (LUPI) setting. At deployment, only the sparse-view student is retained, and its restored renderings serve as pseudo-observations for 3DGS refinement. To the best of our knowledge, TRACE-GS is the first to derive on-policy supervision from privileged geometry for sparse-view 3DGS restoration, achieving consistent gains and strong generalization across
datasets and sparse-view settings.
Read the full abstract ▾
All authors Linlian Jiang, Yuchen Xi, Sadman Rakib Pinon, Ruigang Yang, Yang Wang, Xinxin Zuo
Computer Vision
Chain of Spatial Thoughts: Modality-Agnostic Spatial Grounding for Vision Language Models
Hunter Schofield, Mohammed Elmahgiubi, Mohammad Mahdavian et al. (+4 more)
Spatial understanding is fundamental to
embodied intelligence, underpinning applications such as robotic manipulation,
embodied navigation, and autonomous driving. Although recent
vision-
language models (VLMs) have achieved impressive performance on spatial
reasoning benchmarks, state-of-the-art approaches typically rely on additional spatial encoders or architectural modifications during inference, increasing computational cost. We introduce Space
Tokens, a lightweight, architecture-agnostic framework that equips VLMs with explicit continuous spatial representations without requiring additional inference-time modules. By distilling scene-level 3D geometry and object-centric spatial attributes into continuous latent
tokens, our method enables these modalities to be directly incorporated into a chain-of-thought
reasoning process, thereby improving the VLM's spatial
reasoning capabilities. At the same time, the learned representations can be explicitly decoded to verify that they encode meaningful
geometric information, while the unified
token interface remains extensible to additional modalities. Experiments on VSI-Bench improve Qwen3-VL-8B by 4.3% and SenseNova-SI-1.3 by 1.3%, while achieving state-of-the-art performance on object size (79.2%) and room size estimation (75.7%). These results demonstrate that continuous spatial
tokens provide an effective, interpretable, and computationally efficient mechanism for integrating
geometric reasoning into large
vision-
language models.
Read the full abstract ▾
All authors Hunter Schofield, Mohammed Elmahgiubi, Mohammad Mahdavian, Richard Shi, Jinjun Shan, Amir Rasouli, Dongfeng Bai
Computer Vision · Cross-listed from cs.AI
Beyond Decision Boundaries: Relational Geometry Attacks on Contrastive Embedding Manifolds
Fei Zhao, Peiyuan Zhang, Xi Li et al. (+2 more)
Contrastive learning and Siamese
embedding models have become the foundation of modern verification systems, where decisions are governed not by discrete classification boundaries, but by relational geometry in
embedding space. However, existing
adversarial attacks remain fundamentally classification-centric, overlooking the vulnerability of relational geometry. In this paper, we introduce a geometry-aware
adversarial attack framework that reformulates attacks on contrastive systems as
manifold-level relational corruption. Instead of targeting individual predictions, the proposed framework systematically distorts similarity organization within the
embedding manifold by pushing positive pairs apart while simultaneously pulling negative pairs closer, ultimately collapsing and inverting pairwise similarity structure. To enable scalable deployment, we shift iterative online optimization into an offline
adversarial geometry deformation prior learning stage and train a lightweight feed-forward generator that learns generalized geometry deformation patterns from the victim model. Once trained, the generator produces
adversarial perturbations through a single forward pass without requiring online gradient computation, enabling real-time online attacks against similarity-based verification systems. Experimental results across multiple verification architectures demonstrate substantial degradation of verification performance together with severe
manifold-level relational corruption. On the Markmatch verification system, the proposed attack reduces accuracy from 95.4% to 38.6% while completely reversing the positive-negative similarity structure.
Read the full abstract ▾
All authors Fei Zhao, Peiyuan Zhang, Xi Li, Chengcui Zhang, Nitesh Saxena
Computer Vision
A Convolutional Layer Activation Dimensionality Reduction for Out-of-Distribution and Adversarial Attack Detection Methods
Leandro de Souza Rosa, Lorenzo Capelli, Clara Nunes Barrancos et al. (+2 more)
Despite the success of convolutional neural networks in image classification tasks and their general application in multi-modal models, their susceptibility to out-of-distribution and
adversarial attack samples raises concerns regarding trustworthiness and safety. Among the approaches to tackle such issues, detection methods that analyze the model's intermediate activations to estimate a confidence score are a promising family that evaluates the decision process, relying on a dimensionality reduction step to enable efficient downstream processing of the high-dimensional activations. However, when considering convolutional layers, the dimensionality reduction methods in the literature either lack a mechanism to control the compression/information-loss trade-off or yield large representations. In this paper, we carefully analyze two state-of-the-art detection methods and their dimensionality reductions for convolutional layers and develop a novel reduction method with a controllable high-compression level. We extend these two state-of-the-art detection methods, enabling the usage of any dimensionality reduction, and evaluate their performance on out-of-distribution and
adversarial attack detection. Results show that the detection methods with the proposed dimensionality reduction consistently perform better than, or comparable to, the strongest alternative. Furthermore, the proposed method is shown to reduce computation and
memory footprints, given that it has the highest compression among the compared methods.
Read the full abstract ▾
All authors Leandro de Souza Rosa, Lorenzo Capelli, Clara Nunes Barrancos, Mauro Mangia, Riccardo Rovatti
Computer Vision
More Accurate, Less Human: Gestalt Grouping in Vision Models
Sudhanva Manjunath Athreya and Sai Phani Kumar Malladi
Human
vision organizes what it sees into wholes: same-colored points group into series, similar marks cohere into categories, and shapes complete into recognizable objects. These are the Gestalt operations that visualization design builds on. Whether
vision models organize visual content this way has not been systematically tested. We introduce a behavioral battery that scores models against human data from prior perception studies on four grouping tasks: mark-color odd-one-out, color-series counting, silhouette recognition, and object odd-one-out. We apply it to 45 models across five training families: supervised, self-supervised, and contrastive
vision-
language encoders, open-weight VLMs, and closed foundation models. The battery reveals that agreement with human responses captures aspects of perceptual organization that conventional performance metrics fail to distinguish, with several closed models exhibiting substantially lower
alignment than their
benchmark accuracy would suggest. Scoring against published perception data therefore gives visualization research a reusable yardstick, requiring no new user study, for
auditing whether the models now entering visualization pipelines organize what they see the way their human audience does.
Read the full abstract ▾
Computer Vision
Human versus Computer Vision
Elena Sirotkina
Computer
vision saliency models predict where people will look, one map per image, and a predicted-
attention industry sells those maps in place of measuring real viewers. I evaluate the leading models against every publicly available eye-tracking corpus that records viewer demographics, comprising 27,384 participants across eight
datasets, including 11.4 million webcam gaze points from 2,993 US adults recruited to national quotas while viewing circulating news images. I show that a single map predicts where every group looks about equally well, but systematically ignores how long each place holds a group's
attention. That time follows a clear sociodemographic and political fingerprint, so the map carries a systematic bias against the youngest and oldest viewers and against Black, lower-income, and Republican-leaning ones. I use each group's own gaze to test whether it carries a learnable pattern, across every demographic axis the sample supports. Showing which audiences these systems can learn and what determines this sets the standard for judging claims that computer
vision captures diverse human
vision.
Read the full abstract ▾
Computer Vision
DoseBridge: Denoising Diffusion Bridge Model for Dose Prediction in Lung Intensity-Modulated Proton Therapy
Zerun Zhang, Xiaoda Cong, Xiangkun Xu et al. (+2 more)
Most radiotherapy dose-prediction models use only CT images and anatomical structures, although intensity-modulated
proton therapy (IMPT) dose also depends strongly on beam geometry and available clinical
datasets are often small. We present DoseBridge, a denoising
diffusion bridge model that uses the patient CT as a structured bridge endpoint and encodes plan-specific beam geometry in a spatially aligned beam mask. Multiscale fusion combines CT, target, organ-at-risk, and beam-mask representations with 1.95% additional parameters. DoseBridge was retrospectively evaluated on single-institution CT images and treatment plans from 52 patients with advanced-stage lung cancer treated with 60 Gy in 30 fractions; 42 cases were used for training and 10 for testing. Performance was assessed using image-similarity, dose-volume, and Lyman-Kutcher-Burman normal-tissue complication probability (NTCP) metrics and compared with two deep-learning models. On the test cohort, DoseBridge achieved a mean absolute error of 4.170 Gy, peak signal-to-noise ratio of 23.06 dB, and structural similarity index of 0.798, outperforming both comparison models on these metrics. Clinical target volume D95 differed from the reference dose by 0.62 +/- 1.6 Gy; signed organ-at-risk mean-dose differences ranged from -0.32 to 0.24 Gy, and NTCP differences were -0.40 +/- 2.2 and 0.52 +/- 3.4 percentage points for acute esophagitis and radiation pneumonitis, respectively. Changing only the beam mask redirected predicted low-dose entrance regions while preserving the high-dose target region. To our knowledge, DoseBridge is the first denoising
diffusion bridge model for radiotherapy dose prediction. These results support its feasibility as a beam-aware planning prior for lung IMPT, pending evaluation in larger external cohorts.
Read the full abstract ▾
All authors Zerun Zhang, Xiaoda Cong, Xiangkun Xu, Peter Y. Chen, Xuanfeng Ding
Computer Vision
Motion Artifact-Aware Self-Supervised Representation Learning for 3D Brain MRI Motion Artifact Reduction
Mojtaba Safari, Shansong Wang, Zach Eidex et al. (+4 more)
Patient motion remains a source of image degradation in
brain MRI, leading to signal loss, blurring, and
geometric distortion that compromise quantitative analysis. Existing deep learning methods for motion correction typically rely on paired clean-corrupted data or k-space acquisitions, which are rarely available in clinical settings. We propose SSRL-MAR, a motion artifact-aware unpaired representation learning framework for motion artifact reduction that requires neither paired training data nor explicit motion labels. SSRL-MAR employed a three-stage training strategy: (1) contrastive learning on 3D patches to extract motion representations by contrasting clean and synthetically corrupted images, (2) a motion artifact-aware synthesis network to generate motion artifacts from clean scans, and (3) a motion artifact-aware generator to restore clean volumes using the learned degrader for self-supervised supervision. On in-silico
dataset, SSRL-MAR achieved PSNR 23.81dB, SSIM 91.55%, and NMSE 0.79%. On in-vivo MR-ART
dataset, the pretrained model reduced motion distortion, and unsupervised domain adaptation further improved anatomical fidelity. Against a source-only supervised model trained on the same simulated pairs, SSRL-MAR improved PSNR by up to 2.0 dB on MR-ART after unsupervised domain adaptation, and remained within 0.25-0.47 dB of an oracle supervised model that requires real paired data unavailable in practice. At the milder motion level, volumetric error in structures such as the corpus callosum and ventricular system decreased by more than 50%, confirming improved neuroanatomical consistency. These results indicate that SSRL-MAR provides a robust and scalable image-domain solution for 3D
brain MRI motion correction, enabling reliable structural quantification in large-scale neuroimaging studies without requiring prospectively acquired pairs or acquisition-specific calibration.
Read the full abstract ▾
All authors Mojtaba Safari, Shansong Wang, Zach Eidex, Matthew Goette, Tonghe Wang, Zhen Tian, Xiaofeng Yang
Computer Vision
MAD-HOI: Masked Autoregressive Diffusion for Generating Articulated Hand Object Interactions from Text
Ananya Bal, Kartik Sharma, Ethan Lai et al. (+4 more)
Methods for text-based generation of hand-object interaction (HOI) sequences primarily focus on producing smooth, physically plausible trajectories. A truly utilitarian method should additionally support variable-length generation, composite motion sequences, motion completion and infilling, and reliable termination without compromising physical plausibility. Standard
diffusion models for HOI generation are typically trained only for text-to-motion generation on atomic motions and require the motion length to be specified a priori. Autoregressive (AR) methods provide greater sequence-level flexibility, but commonly depend on discrete motion codes, which can lose contact-sensitive motion detail. To address these key limitations, we present a model performing Masked Autoregression with
Diffusion for HOI generation (MAD-HOI). Our method starts by encoding hand and object motions in a continuous latent space while keeping them disentangled to maintain stream-wise control. This is followed by a masked autoregressive
transformer to predict context features that condition a flow-matching head. MAD-HOI is capable of motion generation for atomic and composite articulated sequences, conditioned motion completion and infilling, as well as EOM (End of Motion) prediction from a single training objective. We provide comprehensive evaluations for these capabilities and
benchmark our method on the ARCTIC and GRAB
datasets. Our experiments show that MAD-HOI achieves the strongest text-motion
alignment and distributional generation quality against baselines on both ARCTIC and GRAB, while maintaining competitive
geometric accuracy and physical plausibility.
Read the full abstract ▾
All authors Ananya Bal, Kartik Sharma, Ethan Lai, Samyak Tiwari, Liza Dahiya, Chaitanya Chawla, Laszlo A. Jeni
Computer Vision
P3CA: Encoder-Agnostic Interpretation of Vision Foundation Model Embeddings via Spatial Probing
Amoon Jamzad, Dilakshan Srikanthan, Faranak Akbarifar et al. (+2 more)
Vision foundation models are increasingly used as reusable encoders in medical image computing, yet their high-dimensional spatial
embeddings are difficult to inspect beyond downstream task performance or global dimensionality reduction. We propose position-prompted PCA (P3CA), an encoder-agnostic method for local probing of channel-rich spatial tensors. Given a user-selected spatial
prompt, P3CA estimates the feature normalization and dominant covariance directions within that region, then applies the resulting projection to the full tensor to visualize where locally informative directions are expressed. This produces a region-conditioned representation lens without modifying the encoder, retraining, or requiring task-specific labels. We implement P3CA in EmbedVision, an interactive 3D Slicer-based workflow, and evaluate it across natural images, colorectal pathology foundation-model
embeddings, and spatial transcriptomic tensors. Across these settings, prompted projections reveal local structure suppressed by global PCA, improve
prompt-matched pathology discrimination from frozen three-dimensional projections, and support comparison between learned and measured spatial representations.
Read the full abstract ▾
All authors Amoon Jamzad, Dilakshan Srikanthan, Faranak Akbarifar, Nooshin Maghsoodi, Parvin Mousavi
Computer Vision
4D-WAM: 4D Consistent World Modeling for Autonomous Driving
Jiacheng Fu, Yibo Yuan, Meng Tian et al. (+8 more)
Emerging World-Action Models (WAMs) have demonstrated promising performance in autonomous driving by jointly modeling future driving scene
evolution and trajectory planning. However, existing WAMs are typically trained with
video data, which is only 2D projections of the underlying 4D driving scene. Consequently, WAMs fail to understand and capture the structure of 4D scenes and thus generate visually plausible yet 4D inconsistent future predictions that mislead downstream planning. To alleviate this issue, we present 4D-WAM, a model that leverages
geometric foundation models for training-time supervision to enable 4D consistent world modeling. Specifically, we feed WAM-predicted future frames into a
geometric foundation model, and use 4D-aware responses to define a 4D consistency loss. This loss encourages the model to understand, represent, and predict physically consistent 4D scenes during training, without additional inference cost. Moreover, we identify an early-decision phenomenon in WAMs and propose a decision-oriented timestep sampling strategy that emphasizes supervision at early, high-noise stages, where driving decisions are primarily formed. By propagating 4D supervision to this critical decision-formation phase, the proposed strategy further improves trajectory planning. Extensive experiments demonstrate that 4D-WAM effectively models 4D consistent scene
evolution and achieves state-of-the-art performance on challenging NAVSIM-v1 and NAVSIM-v2
benchmarks.
Read the full abstract ▾
All authors Jiacheng Fu, Yibo Yuan, Meng Tian, Yue Li, Jiangtong Zhu, Jianhua Han, Yueyi Zhang, Jianwu Fang, Jianru Xue, Hang Xu, Zhiwei Xiong
Computer Vision
Signpost Watermarking: Joint Optimization for Visual Watermark Coexistence
Shruti Agarwal, Vishal Asnani and John Collomosse
We present a method for training imperceptible visual watermarks to coexist with other such watermarks. Recent work has shown that independently trained image watermarking models can coexist with surprisingly limited interference, enabling watermark ensembling. However, this coexistence is a serendipitous property rather than an explicit optimization objective, leaving interference uncontrolled and potentially reducing decoding robustness or visual quality. We first show empirically that the same coexistence property extends to
video watermarking. We then show that both image and
video watermarks can be trained with a decoder-aware objective to improve coexistence. Our results suggest a practical path to signpost watermarks that indicate the presence of independently deployed provenance watermarking systems, supporting layered provenance signaling for content authenticity and rights.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
When Repository Labels Are Not Image-Level Truth: A Supervision Auditing Framework for Chest Radiograph AI
Yesika Alexandra Agudelo-Londoño, Jhon Wilmer Pino-Román, Brahian Carrera Rodríguez et al. (+9 more)
Public chest X-ray repositories are widely used to train medical AI systems, yet their labels are typically extracted from radiology reports rather than verified directly on images. As a result, repository labels are often treated as image-level ground truth without validating whether they reflect what is actually visible in the radiograph. We introduce Repository Supervision
Auditing (RSA), a framework that evaluates repository-derived labels against expert image-level annotations before model development. Using cardiomegaly in MIMIC-CXR as a case study, RSA compares repository labels with radiologist-reviewed image annotations, characterizes disagreement sources, and builds a curated cohort for deployment-oriented evaluation. Repository-derived cardiomegaly labels showed near-zero agreement with expert image-level assessment, identifying only 1% of expert-confirmed cases. Most discrepancies resulted from non-mention rather than explicit report negation, with expert-confirmed cardiomegaly identified in nearly half of studies assigned a repository-derived No Finding label. Using the resulting expert-curated cohort, a DenseNet121 model achieved a test ROC-AUC of 0.853. These findings show that repository labels may not reliably represent image-level truth and highlight supervision
auditing as a critical step for developing trustworthy medical imaging AI.
Read the full abstract ▾
All authors Yesika Alexandra Agudelo-Londoño, Jhon Wilmer Pino-Román, Brahian Carrera Rodríguez, José Miguel Castañeda-Bedoya, Juan Pablo Gómez-López, Aura C. Puche-Sarmiento, Niharika S. D'Souza, Juan Sebastian Osorio-Valencia, Jon E. Duque-Grajales, Jazmín Ximena Suárez-Revelo, Jorge Mario Vélez-Arango, Gabriel Castrillón
Computer Vision
LEGO: Leveled Language Gaussian Splatting
Yuning Peng, Haiping Wang, Yuan Liu et al. (+3 more)
We introduce LEGO for advanced open-vocabulary scene understanding. Beyond basic concept recognition, its core innovation lies in capturing the intrinsic semantic hierarchies within the scene, such as the "flowerpot -> bouquet -> bud -> petal" lineage. While foundation models like SAM can identify multi-granular structures in 2D, their partitions are strictly perspective-bound and lack cross-view consensus. LEGO self-adaptively re-grades volatile multi-view SAM granularities into a unified, 3D-consistent hierarchy. This provides precise supervision for the structurally coherent, multi-level
segmentation of 3D scenes. By grounding these segments with CLIP
embeddings, LEGO recovers open-vocabulary semantic logic across hierarchical levels. Furthermore, by incorporating spatial relationships, we elevate these segments into level-wise
language scene graphs, effectively empowering Large
Language Models to perform complex, context-aware spatial
reasoning and precise visual grounding. Experimental results demonstrate that LEGO establishes new state-of-the-art performance across both promptable and open-vocabulary 3D
segmentation benchmarks, exhibiting advanced hierarchical scene decomposition and context-aware spatial
reasoning.
Read the full abstract ▾
All authors Yuning Peng, Haiping Wang, Yuan Liu, Yipeng Lu, Zhen Dong, Bisheng Yang
Computer Vision
Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots
Shravan Venkatraman, Omkar Thawakar, Ritesh Thawkar et al. (+2 more)
Self-improvement for
multimodal large
language models (MLLMs) is typically driven by reward-based methods that provide only coarse scalar feedback.
Distillation offers a richer alternative through dense
token-level supervision, but in the visual domain it usually depends on privileged context constructed using external annotations and tools, or stronger models. We introduce CVPD (Contrastive Counterfactual Visual Process
Distillation), which, to the best of our knowledge, is the first fully self-contained framework for dense, on-policy,
token-level visual self-
distillation for MLLMs. CVPD identifies visual blind spots where zooming into a region changes and sharpens the model's answer distribution, while removing the same region leaves the full-image behavior largely unchanged. Such regions reveal perceptual information that the model can encode but fails to consistently utilize under full-image conditioning. We propose a three-gate Counterfactual Criterion that identifies these regions directly from the model's own responses and converts them into dense contrastive supervision for self-
distillation. On Qwen3-VL-8B-Instruct, CVPD outperforms six self-evolving baselines across twelve
benchmarks, including methods that rely on external GPT-4o supervision, without a single regression. It achieves gains of $+3.60$ on OCRBench, $+3.38$ on MMStar Fine-Grained Perception, and $+3.08$ on MMStar Logical
Reasoning, while maintaining or improving performance on broader
multimodal benchmarks.
Read the full abstract ▾
All authors Shravan Venkatraman, Omkar Thawakar, Ritesh Thawkar, Abdelrahman Shaker, Rao Muhammad Anwer
Computer Vision
Multimodal Model Diffing for Feature Discovery and Control
Hunar Batra, Lachin Naghashyar, Ashkan Khakzar et al. (+4 more)
Multimodal Large
Language Models (MLLMs) exhibit strong visual understanding, yet the internal features that cause these behaviors remain difficult to identify, audit, or control. While applicable to post-hoc inspection, hidden states that are decomposed into interpretable feature directions using sparse autoencoders (SAEs) neither readily isolate which features are changed by
multimodal training, nor are they directly useful for targeted control. We introduce MMDiff, a
multimodal model-diffing framework that trains
multimodal SAEs and turns them into feature-level interfaces for discovering and controlling
multimodal behavior. MMDiff supports three uses: (i) feature isolation, by diffing a base-LM SAE against its
multimodal-adapted counterpart to identify features altered by
multimodal training; (ii) task-specific feature detection, via per-
token contrastive firing analysis that isolates causal features; and (iii) feature-level control, by causally removing or steering the discovered feature directions. We train
multimodal SAEs for three MLLM families, LLaVA-MORE, PaliGemma 2, and InternVL3.5, and evaluate on visual-spatial understanding,
multimodal safety, and OCR. MMDiff discovers sparse, causally specific features whose removal selectively degrades target behaviors by an average of 12% on spatial tasks and 17% on OCR, and reduces attack success rate by 24% on
multimodal safety attacks, with no impact on VQA performance. Steering these features improves spatial and OCR accuracy by +3.6% and +1.8% on average over a standard single-layer steering baseline. These results show that
multimodal SAEs can serve not only as interpretability tools, but as mechanisms for
auditing, steering, and controlling MLLMs behavior toward safer and more capable generations.
Read the full abstract ▾
All authors Hunar Batra, Lachin Naghashyar, Ashkan Khakzar, Philip Torr, Christian Schroeder de Witt, Constantin Venhoff, Ronald Clark
Computer Vision
Learning How the World Evolves: Extrapolative Video World Models via Latent Dynamics Reasoning
Haodong Li, Shaoteng Liu, Tianyu Wang et al. (+7 more)
The world evolves following its dynamics, i.e., its laws of motion. However, leading
video diffusion models largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduce Latent Dynamics
Reasoning (LDR). LDR casts the latent transition as an explicit kinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on a structured latent rather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-box physics
benchmark spanning five tasks (uniform motion, parabola, collision, bouncing, looming), focusing on out-of-distribution scenarios that reveal whether a model has truly learned the underlying dynamics. LDR extrapolates the learned dynamics far better: the gap between its in- and out-of-distribution error is over 20$\times$ smaller than the
video diffusion baseline's, under both single- and joint-task training at 256$^2$ resolution, while using 26$\times$ fewer parameters and running 143$\times$ faster. LDR can even generalize under severe shift: for example, trained only on red balls moving left-to-right, it correctly predicts the motion of a blue square moving right-to-left. To our knowledge, this is the first
video world model that extrapolates learned dynamics beyond its training distribution. Project page: https://lat-dyn-reason.github.io/
Read the full abstract ▾
All authors Haodong Li, Shaoteng Liu, Tianyu Wang, Chongjian Ge, Sihui Ji, Jiahan Zhang, Xin Lin, Haolin Lu, Zhe Lin, Manmohan Chandraker
Computer Vision · Cross-listed from cs.CR
Overcoming Data Scarcity and Confidentiality in Hardware Assurance via Synthetic Generation
Gijung Lee, Ronald Wilson, Damon L. Woodard et al. (+1 more)
Hardware assurance relies on scanning electron microscopy (SEM) to verify nanoscale structures, but assembling the large, high-quality
datasets required for automated analysis is impeded by time-intensive acquisition and strict intellectual property (IP) constraints on proprietary designs. We propose a
privacy-preserving pipeline that secures IP by heavily distorting the functional design while generating a visually realistic synthetic
dataset from a small set of initial examples. A StyleGAN first learns the distribution of hardware layout masks to generate novel, macroscopically varied structures. Subsequently, a conditional GAN (Pix2PixHD) translates these masks into realistic SEM images that preserve authentic textures and noise. The primary finding of this work is that a
segmentation model trained exclusively on this synthetic data not only demonstrates a successful "sim-to-real" transfer to real images but also outperforms a baseline model trained on the limited real
dataset. Because the underlying synthetic layouts are demonstrably novel and reproduce none of the specific proprietary routing of the original design, deploying the final
segmentation model mitigates the risk of exposing sensitive IP to attacks like gradient inversion and membership inference, providing a highly secure, high-performance solution for hardware assurance.
Read the full abstract ▾
All authors Gijung Lee, Ronald Wilson, Damon L. Woodard, Domenic Forte