Computer Vision · Cross-listed from cs.LG
TEMPEST: Temporal Embeddings for Scalable Driver Identification via Angular Margin Learning
Kyle Musgrove, Dylan B. Lewis, Sarah Powers et al. (+2 more)
Scalable driver identification requires
embedding models that maintain discriminative performance as fleet size grows, yet existing triplet-loss formulations degrade rapidly with driver pool size and overfit to session-specific patterns under rigorous temporal evaluation. We introduce TEMPEST, a Temporal Convolutional Network
embedding model trained with an additive angular margin (ArcFace) loss that enforces global class-level separation in a normalized angular space. TEMPEST maps 60-second
multimodal driving windows to compact 96-dimensional
embeddings, supporting truly dynamic enrollment without any retraining or classifier refitting. Under rigorous temporal evaluation on a 45-driver
dataset, TEMPEST achieves 91.71% Rank-1 accuracy, outperforming the best classical model by 17.9 pp and the strongest triplet-loss baseline by 58.4 pp. TEMPEST degrades by only 4.3 pp when growing the subject pool from 10 to 45 drivers, compared to 22 pp and 32.5 pp for supervised and unsupervised triplet-loss baselines, and its cross-session advantage is corroborated on the public KIA Soul
dataset, where it outperforms the best classical model by 7.3 pp within-session and 14.3 pp cross-session. With 720K parameters, a 2.80 MB footprint, and 50-epoch convergence, TEMPEST establishes a rigorous, reproducible baseline for scalable behavioral driver biometric identification.
Read the full abstract ▾
All authors Kyle Musgrove, Dylan B. Lewis, Sarah Powers, Emma J. Reid, Hector Santos-Villalobos
Computer Vision · Cross-listed from cs.CL
TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
Andong Hua, Colton Bishop, Igor Mordatch et al. (+5 more)
Multimodal large
language models (MLLMs) should exhibit cross-modal consistency, generating consistent responses given semantically equivalent inputs across modalities. However, we observe a systematic discrepancy in model predictions under such cross-modal variations. Specifically, we define the modality gap as the difference in model performance under semantically equivalent textual and
multimodal inputs. We introduce TokenSwap, a method that constructs such inputs by replacing textual concepts with semantically aligned images, resulting in sequences where visual
tokens are interleaved with text
tokens. Based on TokenSwap, we transform existing text-based
benchmarks (e.g., MMLU) into image-interleaved counterparts, resulting in TokenSwap-Bench. Across 42 MLLMs, we observe a pervasive modality gap, with performance decreasing by 4.2% to 47.4% when moving from text-only to image-interleaved inputs, averaging 19.6% +/- 3.3% across models. Notably,
reasoning models exhibit consistently smaller gaps, achieving an average gap of 10.1% compared to 25.5% for non-
reasoning models, suggesting greater robustness in
multimodal reasoning across equivalent textual and visual representations. In contrast, neither prompting strategies nor scaling training compute alone reliably reduces the modality gap. Finally, we demonstrate that incorporating TokenSwap during training effectively mitigates this gap, improving cross-modal
alignment while preserving strong text-only and
vision-
language model performance. The project page is available at https://dongxzz.github.io/TokenSwap/.
Read the full abstract ▾
All authors Andong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini, Jindong Gu, Aleksandra Faust, Rebecca Roelofs, Yao Qin
Computer Vision
Towards Large Model Feature Coding
Youwei Pang, Changsheng Gao, Dong Liu et al. (+2 more)
Large models have delivered remarkable performance across a wide range of perception and generation tasks, yet practical deployment is increasingly constrained by computational and
memory budgets, as well as
privacy requirements. Split execution alleviates these constraints by partitioning computation across devices, but it inevitably introduces intensive transmission and storage of intermediate features. Unlike conventional feature coding for CNNs that typically targets homogeneous spatial activation maps, modern large models generate heterogeneous features with varying statistical distributions and compression tolerances, e.g., multi-level/multi-modal representations and autoregressive context caches. These characteristics necessitate treating large model feature coding (LaMoFC) as a fundamental system component and call for a systematic evaluation framework. In this paper, we present a comprehensive
benchmark and evaluation framework for LaMoFC. We first build the feature
dataset LaMoFCBench, covering diverse task requirements across 4 categories and 16 scenarios while integrating widelyadopted architectures and various split-computing settings. We then specify representative split points according to practical application scenarios to extract intermediate features, establishing a unified pipeline for fair and reproducible comparisons. Finally, we
benchmark mainstream universal feature codecs, exposing the profound misalignment between existing coding paradigms and the heterogeneous nature of large model features. These findings reveal that LaMoFC demands a fundamental departure from existing paradigms, and LaMoFCBench provides the shared empirical foundation to drive this transition. The data and code will be available at https://github.com/lartpang/LaMoFCBench.
Read the full abstract ▾
All authors Youwei Pang, Changsheng Gao, Dong Liu, Huchuan Lu, Weisi Lin
Computer Vision
Zhe Cheng, Wenyu Chen, Fode Zhang et al. (+1 more)
Large
vision-
language models (LVLMs) often hallucinate content that is fluent yet unsupported by the image, limiting their reliability in real-world deployment. We show that a key failure mode arises from route competition: even when visual
tokens receive
attention, the final
token decision can be dominated by the textual pathway, causing the decoder to follow linguistic priors over visual evidence. To mitigate this, we propose a training-free, decision-aligned intervention that decomposes each
attention head into a visual route and a text route, and estimates their
token-level effects using an efficient one-forward/one-gradient approximation. These estimates reveal route conflict within heads and identify prior-dominant ones, enabling selective suppression of only the text route while keeping the visual route intact. Across five
benchmarks spanning discriminative and generative settings, our method consistently reduces
hallucination-related errors across models with limited impact on overall
multimodal performance, while incurring a modest inference-time overhead.
Read the full abstract ▾
All authors Zhe Cheng, Wenyu Chen, Fode Zhang, Dehuan Shen
Computer Vision
Soft Tuy-Completeness for Robust Projection Selection in Cone-Beam CT
Linda-Sophie Schneider and Andreas Maier
This work introduces a continuous soft near-orthogonality score and a resolution-aware saturated coverage objective for projection selection in region-of-interest focused cone-beam CT, grounded in Tuy's completeness theory. Replacing the binary hit-or-miss model of classical Tuy completeness with a graded, differentiable formulation preserves a direct link to achievable feature sizes while enabling both efficient approximate and exact optimisation. We establish that the underlying discrete decision problems are NP-complete via
polynomial-time reductions from Set Cover, motivating a submodular greedy
algorithm with proven $(1-1/\mathrm{e})$ approximation guarantees and a mixed-integer linear program (MILP) that provides certified optimality bounds. The MILP serves as a quality certificate for the greedy solution rather than a competing optimiser. The primary empirical finding confirms this relationship: across a systematic
benchmark spanning six target regions, multiple projection budgets, and four controlled occlusion conditions, the pooled median greedy-to-MILP objective ratio was 0.998, with a substantial fraction of cases certified globally optimal. A binary formulation is included as a diagnostic baseline; it strengthens hard directional completeness but is weaker on the continuous coverage scale. We additionally introduce Effective Spatial Resolution (ESR), a physically interpretable trajectory-level diagnostic that maps directional sampling gaps to achievable feature sizes. ESR correlates reliably with matched reconstruction quality across projection budgets and occlusion levels, providing a practical bridge between the selection stage and the image domain without requiring reconstruction.
Read the full abstract ▾
Computer Vision
Machine Intelligence that Understands Visual and Linguistic Information and Interacts with Humans and Environments
Van Quang Nguyen
Advancements at the intersection of computer
vision and natural
language processing are crucial for applications like assistive tech, multimedia querying, and robotics. This dissertation proposes novel architectures to improve intelligent
agents across three key
vision-
language tasks: image captioning, visual dialog, and interactive instruction following. First, we address limitations in visual representation for image captioning. Traditional models rely on region-based features from CNN detectors, which lack global context and suffer from high computational overhead. We propose GRIT (Grid and Region-based Image captioning
Transformer), a
transformer-only architecture. By integrating grid and region features using a DETR-based detector, GRIT enables end-to-end training and out-performs prior methods in both inference accuracy and speed. Second, we tackle visual dialog, which requires multi-turn conversation about an image. The challenge lies in efficiently modeling interactions between multiple inputs (image, question, history). We introduce LTMI (Light-weight
Transformer for Many Inputs). Utilizing a specialized
attention block, an LTMI layer matches the representational power of a standard
Transformer extension while utilizing less than one-tenth of its parameters, as validated on the VisDial
dataset. Finally, we study interactive instruction-following for
embodied AI using the ALFRED
dataset. We propose a framework featuring a two-stage instruction interpretation: it first decodes
language directives independently of visual context to predict a tentative action-object sequence, which is then fused with visual features for final execution. Using multiple
egocentric views and hierarchical
attention, our method accurately localizes objects and achieves a state-of-the-art unseen success rate of 8.37%.
Read the full abstract ▾
Computer Vision
MGVQ: Synergizing Multi-dimensional Sensitivity-Aware and Gradient-Hessian Fusion for Vector Quantization
Zhong Wang, Zukang Xu, Xing Hu et al. (+1 more)
Vision-
Language Models (VLMs) achieve outstanding performance, yet their huge model size severely hinders deployment on edge devices with limited resources. As an efficient model compression technique, vector quantization (VQ) excels in ultra-low-bit representation, which maps model weights to discrete codewords in a compact codebook to cut
memory consumption and transmission overhead while preserving model capability. Direct VQ application to VLMs still has two core limitations. First, cross-modality weight distribution differences brought by visual and textual inputs cannot be well fitted by a single unified codebook. Second, current second-order error compensation ignores first-order gradient information, causing weight deviation from pre-trained optimal states, gradient drift and biased compensation results. This work proposes MGVQ, a novel vector quantization framework integrating multi-dimensional sensitivity perception and gradient-Hessian fusion. It consists of two core modules: sensitivity-guided structured mixed-precision quantization dynamically assigns different bit-widths according to channel sensitivity via combined global and local sensitivity analysis for refined resource allocation; gradient-aware second-order error compensation embeds first-order gradients into error correction, and adopts Kronecker and Block-LDL decomposition to ensure low computational cost. Extensive experiments on mainstream VLMs including LLaVA-onevision, InternVL2 and Qwen2-VL verify the effectiveness of MGVQ. In 2-bit quantization settings, MGVQ surpasses existing advanced post-training quantization methods significantly, achieving a maximum accuracy improvement of 4.9 points (71.4% vs 67.0% on InternVL2-26B). The proposed method realizes stable and efficient ultra-low-bit VLM quantization, greatly promoting the practical deployment of
multimodal large models in resource-limited environments.
Read the full abstract ▾
All authors Zhong Wang, Zukang Xu, Xing Hu, Dawei Yang
Computer Vision
SkySeg: Collaborative Onboard Semantic Segmentation with Heterogeneous UAVs in the Wild
Anqi Lu, Yun Cheng, Youbing Hu et al. (+3 more)
The demand for unmanned aerial vehicle (
UAV)-based image acquisition and analysis has surged, with
UAVs increasingly utilized for semantic
segmentation tasks. To meet the real-time analysis requirements of
UAV remote sensing missions, performing onboard computation and making decisions based on the results is a natural approach. However, deploying semantic
segmentation on resource-constrained
UAV platforms presents two significant challenges: 1) hardware constraints limit the ability of
UAVs to perform real-time semantic
segmentation, and 2) environmental variations during flight cause data distribution shifts, deviating from the original training data. To address these issues, this paper introduces SkySeg, a heterogeneous multi-
UAV air-air cooperation framework that integrates computer
vision and flight pattern to enable onboard semantic
segmentation using low-cost sensors. SkySeg employs an efficient information fusion inference method, combining low-definition, wide-area images with high-definition, focused-area images. Additionally, it incorporates a cross-device test-time adaptation (TTA) strategy to enhance
segmentation performance in dynamic environments by collaboratively addressing distribution shifts of test data streams across
UAVs. Experimental results demonstrate that our SkySeg framework accelerates inference latency by approximately 3.6x, improves onboard
segmentation accuracy by 5.91%, and achieves a 10.91% average accuracy gain in the wild.
Read the full abstract ▾
All authors Anqi Lu, Yun Cheng, Youbing Hu, Zhiqiang Cao, Jie Liu, Zhijun Li
Computer Vision
Deep Learning-Based Automated Quantification of TIMI Myocardial Perfusion Frame Count (DL-TMPFC) from Coronary Angiography: A Novel Framework for Rapid Assessment of Microvascular Dysfunction
Si Li, Yuanqing He, Chenkai Hu et al. (+7 more)
Aims: Coronary microvascular dysfunction (CMVD) affects approximately 40%-60% of patients with ischemia and non-obstructive coronary arteries, yet diagnosis remains challenging due to reliance on invasive functional testing or subjective Thrombolysis In Myocardial Infarction (TIMI) flow grade. The TIMI Myocardial Perfusion Frame Count (TMPFC) offers an objective, angiography-based quantitative measure of CMVD, but its clinical translation is hindered by cumbersome manual calculation and insufficient validation. This study aims to develop and validate a deep learning-powered TMPFC calculation (DL-TMPFC), enabling integration into clinical workflows. Methods and results: DL-TMPFC framework comprised two components. A stenosis detection network first excluded obstructive coronary artery disease (CAD). A territory-aware
segmentation network then identified perfusion territories and TMPFC calculation module automatically determined the first and last frames from angiographic sequences. The framework was validated in a cohort of 655 patients (445 of obstructive CAD, 100 of confirmed CMVD, 110 of control group) from three independent institutions. DL-TMPFC showed excellent agreement with expert manual measurements (bias: -0.93 frames; 95% LoA: -5.33 to +3.47; r =0.98). DL-TMPFC markedly enhanced clinical feasibility by fully automating TMPFC and removing observer dependence. Clinically, DL-TMPFC accurately identified CMVD across a full spectrum of coronary pathologies and captured the continuous severity of CMVD beyond binary classification, enabling quantitative risk stratification. Conclusion: DL-TMPFC enabled automatic, standardized, and accurate quantification of CMVD directly from routine angiography. By providing an automatic and objective measure, this tool provided immediate diagnostic information for timely recognition and management of CMVD in clinical practice.
Read the full abstract ▾
All authors Si Li, Yuanqing He, Chenkai Hu, Xiaogang Guo, Huay-Cheem Tan, Chieh Yang Koo, Xuan Zhang, Lei He, Jingyuan Zeng, Shan Xiao
Computer Vision
Weierstrass Positional Encoding for Vision Transformers
Zhihang Xin, Rui Wang, Xitong Hu et al. (+1 more)
Vision Transformers have achieved remarkable success in computer
vision, but their common use of learnable one-dimensional positional encodings weakens the inherent two-dimensional spatial structure of images after patch flattening. Existing positional encodings often lack
geometric constraints and do not preserve a monotonic relationship between Euclidean spatial distances and sequential index distances, limiting ViTs' ability to exploit spatial proximity priors. Motivated by the usefulness of periodicity in positional encoding, we propose Weierstrass
elliptic Positional Encoding (WePE), a mathematically grounded method for encoding two-dimensional coordinates in the complex domain. WePE maps normalized 2D patch coordinates onto the complex plane and constructs compact four-dimensional positional features using the Weierstrass
elliptic function and its derivative. The double periodicity provides a principled representation of 2D positions, and its intrinsic
lattice structure naturally matches the regular geometry of image patch grids. Its nonlinear
geometric properties help model spatial distance relationships more faithfully, while the algebraic addition formula enables relative positional information between arbitrary patch pairs to be derived directly from their absolute encodings. WePE is plug-and-play and resolution-agnostic, allowing seamless integration into existing ViTs. Extensive experiments show that WePE brings consistent performance gains in most settings. With precomputed lookup tables, these improvements introduce no noticeable computational or
memory overhead. Additional analyses and ablation studies further validate the effectiveness of the proposed method.
Read the full abstract ▾
All authors Zhihang Xin, Rui Wang, Xitong Hu, Xiaojun Wu
Computer Vision
GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation
Kaichen Zhou, Yuzhen Chen, Fangneng Zhan et al. (+8 more)
Video world models can generate realistic futures from a single instruction, but they often fail to track the same physical points consistently across time. As a result, the generated
videos appear plausible, yet lack the physical grounding required for reliable action execution, such as
robot manipulation. We present GEM-4D, a geometry-grounded
video world model that resolves this limitation by injecting dense 4D correspondence supervision distilled from a pretrained geometry foundation model into the
video generative backbone during training. This supervision enables the model to jointly capture appearance and
geometric structure while retaining a single-stream architecture with no additional inference cost. We further introduce an inverse dynamics module that converts correspondence-consistent
video rollouts into executable
robot trajectories, enabling direct deployment in both real-world and simulated manipulation. GEM-4D achieves state-of-the-art performance on both
video prediction and
geometric consistency across both simulation and realistic scenarios and improves real-world manipulation success from 61% to 81%. Additional results are available at https://gem-4d.github.io/.
Read the full abstract ▾
All authors Kaichen Zhou, Yuzhen Chen, Fangneng Zhan, Hang Hua, Grace Chen, Xinhai Chang, Ao Qu, Yilun Du, Zhuang Liu, Paul Pu Liang, Mengyu Wang
Computer Vision · Cross-listed from cs.LG
MedExpMem: Adapting Experience Memory for Differential Diagnosis
Qianhan Feng, Zhongzhen Huang, Yakun Zhu et al. (+4 more)
Experienced physicians develop diagnostic expertise through clinical practice, acquiring not only disease knowledge but also the ability to differentiate confusable conditions. Current medical
vision-
language models (VLMs) lack this capability – their parameters encode static knowledge that does not evolve across diagnostic encounters. We propose MedExpMem, an experience
memory framework enabling VLM-based diagnostic
agents to accumulate differential diagnosis expertise. Unlike
retrieval-augmented generation, which retrieves encyclopedic disease descriptions, MedExpMem memorizes discriminative experience derived from the
agent's own diagnostic failures and organizes them as pairwise differential notes encoding key discriminators, actionable decision rules and
reasoning error patterns. The framework adopts a two-phase construction process mirroring physician learning: initial practice exposes knowledge gaps, and reflective re-diagnosis refines understanding. When encountering new cases, the
agent retrieves experience
memory to guide differential
reasoning. We evaluate MedExpMem on a radiology
benchmark spanning 11 subspecialties. Results demonstrate consistent accuracy improvements, maximum 7.0%, across diverse models and scales. Analytical experiments validate experience quality and robustness, demonstrating MedExpMem as a competitive method addresses medical adaptation needs beyond the reach of parameteric learning.
Read the full abstract ▾
All authors Qianhan Feng, Zhongzhen Huang, Yakun Zhu, Yannian Gu, Winnie Chiu Wing Chu, Xiaofan Zhang, Qi Dou
Computer Vision · Cross-listed from eess.IV
An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation
Xiaofeng Liu, Qianru Zhang, Thibault Marin et al. (+4 more)
The synergistic interpretation of anatomical information from computed
tomography (CT) and metabolic information from positron emission
tomography (PET) is important to oncologic imaging. However, existing deep learning methods for PET/CT remain largely task-specific, are often trained on single-center cohorts, or adopt dual-branch fusion schemes that delay cross-modal interaction and underutilize early spatial correspondence between PET and CT. To address these limitations, we present an open-source, multi-center, whole-body FDG PET/CT foundation model utilizing 4,997 harmonized scans from four public
datasets. Our framework employs hierarchical UNet-shaped backbones with early channel-wise concatenation, enabling anatomical and metabolic features to interact from the first
embedding layer onward. We further introduce a masked autoencoding objective based on zero-mean imputation, combined with a weighted global reconstruction loss. This design avoids non-physical intensity discontinuities at masked-region boundaries that arise from learnable mask
tokens. On downstream AutoPET lesion
segmentation, the proposed models demonstrate strong label efficiency: with only 10% of the labeled training data, they achieve performance comparable to models trained from scratch on the full
dataset. Under extreme 5-shot linear probing, joint PET/CT pretraining also achieves higher Dice scores than separated-modality pretraining. This multi-center foundation model demonstrates label efficiency and cross-modality representation learning for PET/CT tumor
segmentation. It provides a robust, open-source basis for advancing automated oncologic imaging, significantly reducing the need for large-scale manual annotations in clinical practice.
Read the full abstract ▾
All authors Xiaofeng Liu, Qianru Zhang, Thibault Marin, Menghua Xia, Chi Liu, Georges El Fakhri, Jinsong Ouyang
Computer Vision · Cross-listed from eess.IV
Mapping Tomato Cropping Systems in California Using AlphaEarth Geospatial Embeddings and Deep Learning Analysis
Mohammadreza Narimani, Alireza Pourreza and Parastoo Farajpoor
Field-scale crop maps support supply-chain
forecasting and policy, yet statewide crop identification still often depends on retrospective surveys or remote-sensing workflows built around hand-engineered spectral features. Those pipelines can be accurate, but they require repeated preprocessing and often lose robustness across years. This study evaluated whether Google DeepMind's AlphaEarth geospatial
embeddings can serve as an analysis-ready alternative for mapping processing tomato systems in California. LandIQ 2018 crop polygons were used to assemble a balanced reference
dataset of 4,742 tomato and 4,742 non-tomato fields. For each polygon, 64-band AlphaEarth
embedding chips were extracted and aligned with binary masks, then divided into spatially independent training (n = 6,638), validation (n = 1,422), and test (n = 1,424) sets. A U-Net
segmentation model was trained on AWS SageMaker using a composite masked binary cross-
entropy and soft Dice loss. To complement hard predictions, Monte Carlo dropout was retained at inference and repeated 100 times per chip to estimate predictive mean and variance. On the independent test set, the model achieved 99.19% pixel accuracy, 98.69% precision, 99.40% recall, 99.04% F1 score, 98.11% intersection over union, and 99.02% chip accuracy. Uncertainty maps were consistently highest near field edges and low within field interiors. The results show that AlphaEarth
embeddings retain crop-relevant spatial and temporal structure and can support accurate, field-scale tomato mapping without manual feature engineering.
Read the full abstract ▾
Computer Vision
MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue
Anna Deichler, Jim O'Regan, Fethiye Irmak Dogan et al. (+4 more)
Grounding
language in the physical world requires AI systems to interpret references that emerge dynamically during conversation. While current
vision-
language models (VLMs) excel at static image tasks, they struggle to resolve ambiguous expressions in spontaneous, multi-turn dialogue. We address this gap by introducing (1) a
benchmark for referential communication in dynamic 3D environments, built from 6.7 hours of
egocentric VR interaction with synchronized
speech, motion, gaze, and 3D scene geometry, and (2) a two-stage grounding pipeline that explicitly resolves conversational ambiguity before visual localization. The
benchmark includes over 4,200 manually verified referring expressions spanning full, partitive, and pronominal types. Our contextual rewriting approach improves grounding performance by 11-22 percentage points on average, with a pure detector (GroundingDINO) reaching 56.7% on pronominals after rewriting, nearly double the best end-to-end baseline. Results demonstrate that decoupling linguistic
reasoning from visual perception is more effective than end-to-end approaches for conversational grounding.
Read the full abstract ▾
All authors Anna Deichler, Jim O'Regan, Fethiye Irmak Dogan, Lubos Marcinek, Anna Klezovich, Iolanda Leite, Jonas Beskow
Computer Vision
SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching
Xuefei Sun, Xujia Zhang, Brendan Crowe et al. (+2 more)
Zero-shot 3D visual grounding requires localizing objects in unstructured environments from free-form natural
language. Recent
vision-
language model (VLM) approaches achieve promising results but rely on view-dependent
reasoning or implicit representations, limiting spatial consistency and interpretability for compositional queries. We propose SceneGraphGrounder, a framework that reformulates 3D grounding as structured graph matching over a reconstructed 3D scene graph. To enable this formulation, we introduce a visual marker prompting strategy that enables a VLM to infer object-object relationships from 2D views, which are subsequently lifted into a persistent 3D scene graph encoding both spatial and semantic relations. Given a query, we construct a query graph and perform constrained
alignment with the scene graph, ensuring multi-view consistency and interpretable
reasoning. Experiments on the ScanRefer
benchmark demonstrate that our method achieves competitive performance among zero-shot approaches, using only RGB-D inputs. We further validate our framework through real-world deployment on a mobile
robot, demonstrating robust spatial
reasoning in long-horizon physical environments. We will make our code publicly available upon acceptance.
Read the full abstract ▾
All authors Xuefei Sun, Xujia Zhang, Brendan Crowe, Doncey Albin, Christoffer Heckman
Computer Vision
BodyReLux: Temporally Consistent Full-Body Video Relighting
Li Ma, Mingming He, Xueming Yu et al. (+4 more)
Being able to relight human performance is a fundamental task for post production and content creation. We present BodyReLux, a subject-specific
video diffusion-based framework for relighting full-body human performances in a temporally consistent way. Our model is trained on a hybrid
dataset of pixel-aligned
video relighting pairs, covering a diverse combination of lighting conditions, performances and viewpoints. To acquire such
dataset, we combine traditional static One-Light-at-a-Time (OLAT) capture and a novel dynamic performance capture in which two smoothly varying lighting sequences are rapidly interleaved. Because the lighting operates above the human flicker-fusion threshold, the interleaving does not appear to strobe. We train our
video relighting model from a pretrained text-to-
video model to fully leverage the generative priors for producing high quality
videos. To achieve accurate lighting control, we introduce a new lighting conditioning method that represents each light source as a
token. We further condition on sequences of lighting using masked
attention to support dynamic lighting control. Together with a carefully designed data augmentation pipeline, we achieve photorealistic, robust, and temporally consistent
video relighting of subject-specific human performances.
Read the full abstract ▾
All authors Li Ma, Mingming He, Xueming Yu, David M. George, Ahmet Levent Taşel, Paul Debevec, Julien Philip
Computer Vision
Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models
Steven Chen, Shivesh Khaitan and Nemanja Djuric
We present an approach to improve 3D vehicle labeling in self-driving applications through zero-shot inference of vehicle information, leveraging Vehicle Make and Model Recognition (VMMR) methods. The proposed approach utilizes a
Vision Language Model (VLM) to both infer a vehicle's make, model, and generation from image crops, and output accurate 3D bounding box dimensions to seed manual labeling. We evaluate the impact of iterative
prompt engineering and the choice of different VLMs on both vehicle bounding box inference and make/model/generation recognition. When compared to strong baselines, the proposed approach not only shows high accuracy, but also excels in mitigating specific failure modes where VLMs provide better dimensions than initial
lidar-aided human annotated labels (e.g., in cases of significant vehicle occlusion). Experiments on both public and proprietary data strongly suggest that our conclusions are generalizable across different labelers and
datasets. The results demonstrate that integrating VLMs into the labeling process can reduce manual labeling time while increasing label quality.
Read the full abstract ▾
Computer Vision
BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model
Gonçalo Gomes, Bruno Martins and Chrysoula Zerva
Image captioning evaluation remains a significant challenge, as
vision-
language models evolve toward more challenging capabilities such as generating long-form and context-rich descriptions. State-of-the-art evaluation metrics involve extensive computational costs associated with the use of Large
Language Models (LLMs) as judges, or instead suffer from the limitations of standard CLIP-based encoders, such as strict
token limits, lack of fine-grained sensitivity, or lack of compositional generalization by treating captions as “bags-of-words.” We propose a new learned metric that tackles the aforementioned challenges, based on a lightweight cross-encoder that is initialized from a visual question-answering model checkpoint, balancing a strong weight initialization with computational efficiency. Our training scheme uses a carefully assembled data mixture for supervised learning, featuring
adversarial LLM-based data augmentations to enhance model sensitivity to fine-grained visual-linguistic errors. We also introduce a new
benchmark designed to assess detailed captioning evaluation across diverse scenarios. Experimental results demonstrate that the proposed metric achieves state-of-the-art performance while maintaining the efficiency required for large-scale benchmarking, quality-aware decoding, or reward guidance.
Read the full abstract ▾
Computer Vision
AVI-HT: Adaptive Vision-IMU Fusion for 3D Hand Tracking
Ziyi Kou, Ankit Kumar, Mia Huang et al. (+4 more)
We present AVI-HT, an adaptive visual-IMU fusion approach for tracking 3D hand poses by jointly modeling the
egocentric image with on-glove 6-DoF IMU signals. AVI-HT achieves significantly improved accuracy and availability, particularly in hand-object interaction (HOI) scenarios involving heavy visual occlusion. Two complementary ingredients underpin its success: (1) synchronized multi-modal training data pairing on-body
vision-IMU sensor streams with ground-truth 3D hand poses from a motion-capture system, and (2) a cross-sensor deep
attention mechanism that adaptively modulates the trust assigned to the
vision and individual IMU sensors. To evaluate AVI-HT in real-world settings, we conduct extensive experiments on our DexGloveHOI
dataset that consists of 100K+ pairwise
vision-IMU samples with synchronized 3D annotated poses, in which users manipulate a variety of objects during daily tasks. We compare against multiple single- and multi-modal tracking approaches under two hand models (UmeTrack, MANO). The results show that AVI-HT reduces mean keypoint error by 16.1% and its wrist-aligned variant by 24.2% over the baselines. Ablation studies further reveal the per-finger contribution of IMU sensors across activity types, and the model's sensitivity to IMU noise and temporal misalignment in
vision-IMU fusion.
Read the full abstract ▾
All authors Ziyi Kou, Ankit Kumar, Mia Huang, Taylor Niehues, Vatsal Mehta, Ergys Ristani, Li Guan
Computer Vision · Cross-listed from eess.IV
HyperBench: Standardizing and Scaling Synthetic Evaluation for Hyperspectral Super-Resolution
Ritik Shah and Marco F. Duarte
Hyperspectral super-resolution (HSR) reconstructs a high-spatial-resolution hyperspectral image by fusing a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI). In the absence of real-world paired data, HSR methods are evaluated almost exclusively on synthetic experiments derived from hyperspectral
datasets through Wald's protocol. Despite the protocol's widespread adoption, its practical implementation varies markedly across research works, typically relying on a single (usually Gaussian) or very few point spread functions (PSFs), one or two spectral response functions (SRFs), and a couple of spatial downsampling factors. As a result, reported performance figures are difficult to compare across the literature, in addition to being often difficult to reproduce; furthermore, they may not generalize across realistic sensing conditions. We introduce HyperBench, a unified and extensible framework that standardizes synthetic experimentation for HSR. HyperBench supports diverse degradation configurations spanning ten PSFs, four SRFs derived from operational multispectral sensors, configurable spatial downsampling factors, and matched additive white Gaussian noise; its goal is to automate large-scale evaluation and structured logging. By decoupling model development from experimental design, the framework enables reproducible, apples-to-apples cross-method comparison with minimal friction. We use HyperBench to evaluate six recently proposed HSR methods across a 70-configuration sweep on four widely used hyperspectral scenes and observe that the inter-method PSNR spread widens from approximately 5 dB on the easiest PSF to over 13 dB on the hardest - a fragility that is structurally invisible to the prevailing single-configuration evaluation protocol. HyperBench code is available at https://github.com/ritikgshah/HyperBench .
Read the full abstract ▾
Computer Vision
MRecover: A Conditional Generative Model for Recovering Motion-Corrupted MR images Using AI Generated Contrast
Jinghang Li, Tales Santini, Courtney Clark et al. (+13 more)
Hippocampal subfield
segmentation requires high-resolution T2w turbo
spin echo (TSE) MRI, yet this sequence is susceptible to motion artifacts, leading to substantial data loss. We developed a conditional generative model (MRecover) that synthesizes routinely acquired T1w images to create TSE images with autoregressive slice conditioning for volumetric consistency. Trained on 7T MRI data (n=577), the model achieved high in-domain fidelity (n=148, SSIM=0.84, FSIM=0.94) and generalized well to out-of-domain 3T data: subfield volumes from synthesized and the as-acquired images closely matched: (n=416, r=0.87-0.97) and yielded 31.8% more analyzable subjects in the motion-affected ADNI3
dataset after quality control (593 vs 450). The synthesized images also achieved larger effect sizes due to increasing the sample size for diagnostic group differences in hippocampal subfield atrophy (whole hippocampus $\epsilon^2$= 0.121-0.100 vs. 0.086-0.062, left-right hemispheres). Project page: https://jinghangli98.github.io/MRecover/
Read the full abstract ▾
All authors Jinghang Li, Tales Santini, Courtney Clark, Bruno de Almeida, Cong Chu, Salem Alkhateeb, Andrea Sajewski, Jacob Berardinelli, Hecheng Jin, Tobias Campos, Jeremy J. Berardo, Joseph Mettenburg, Ariel Gildengers, Howard J. Aizenstein, Minjie Wu, Tamer S. Ibrahim
Computer Vision · Cross-listed from cs.LG
Hierarchical Variational Policies for Reward-Guided Diffusion
Kushagra Pandey, Farrin Marouf Sofian, Jan Niklas Groeneveld et al. (+2 more)
Adapting pretrained
diffusion models to downstream objectives such as inverse problems often requires expensive test-time guidance or optimization. We propose a principled framework for generating high-quality reward-aligned samples at substantially reduced inference cost. Our approach formulates test-time adaptation as a hierarchical variational model, where control is amortized into a lightweight yet expressive
stochastic policy. This formulation naturally supports few-step
diffusion sampling: large step sizes enable fast inference, while the learned policy maintains sample quality by providing structured per-step control. The resulting fully amortized sampler achieves a strong quality–speed tradeoff, matching or exceeding recent test-time scaling baselines while requiring significantly less compute. For example, on 4x super-resolution, our method achieves better perceptual quality with more than 5x faster inference compared to the best-performing baseline. We further extend our approach to a semi-amortized regime that combines cheap amortized proposals with limited test-time optimization, achieving state-of-the-art perceptual quality across several challenging inverse problems.
Read the full abstract ▾
All authors Kushagra Pandey, Farrin Marouf Sofian, Jan Niklas Groeneveld, Felix Draxler, Stephan Mandt
Computer Vision
Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming
Yue Zhou, Erxuan Wu, Yikang Sun et al. (+5 more)
Vision-
Language Models (VLMs) have significantly advanced medical visual question answering, yet their performance in ultrasound remains suboptimal. In clinical practice, sonographers explicitly focus on lesion regions to formulate reports, though diagnostic interpretations sometimes vary due to inherent subjectivity. However, existing VLMs are not explicitly structured to interactively zoom into lesions prior to diagnosis; moreover, they typically treat annotations as unbiased ground truths, failing to account for their inherent subjectivity and ambiguity. In this paper, we propose a framework specifically designed to consider the sonographer's cognitive workflow. We first introduce a structured Zoom-then-Diagnose paradigm, which replicates the interactive search process to enable lesion-focused
reasoning. Furthermore, within the Group Relative Policy Optimization (GRPO) framework, we introduce an uncertainty-aware reward derived from
stochastic group-wise rollouts to estimate prediction consistency as a proxy for model confidence. Together, these two components encourage the model to reinforce accurate predictions on clear cases while remaining cautious under ambiguity. Experiments across liver, breast, and thyroid
datasets show that our framework improves lesion localization by 39.3%, demonstrating that our model has learned the ability to actively look closer and diagnose.
Read the full abstract ▾
All authors Yue Zhou, Erxuan Wu, Yikang Sun, Hongjoo Lee, Yuan Bi, Huixiong Xu, Nassir Navab, Zhongliang Jiang
Computer Vision
Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?
Tianyi Zhang, Mahtab Bigverdi and Ranjay Krishna
Vision-
language models (VLMs) are increasingly augmented with continuous or latent non-textual
tokens intended to support "visual thinking." Despite improved task accuracy, this alone does not show that models actually use these
tokens for
reasoning – gains may arise from confounds such as added context length, special-
token anchoring, or training-time regularization. We formalize a diagnostic principle, Ablate-to-Validate, for testing whether latent-
token content is genuinely utilized, and instantiate it as the
Token Replacement Test (TRT), a standardized suite of content-replacement ablations. TRT holds the
prompt, image,
token budget, and decoding fixed while replacing intermediate
tokens with zero, random, first-repeat, or oracle alternatives, isolating whether performance depends on
token content or merely on
token presence. As a controlled testbed, we study relative depth
reasoning with LLaVA-13B and Qwen2.5-VL-3B, training models to predict and consume continuous or discrete depth spans across multiple frozen encoders (SigLIP2, CLIP, DINOv2) and
token budgets. We additionally apply TRT to three off-the-shelf visual-thinking systems (Mirage, Mull-
Tokens, CoVT) on BLINK, VSP, and CV-Bench. Across all settings, accuracy gains are a misleading proxy for latent-
token reasoning: VLMs retain most improvement even when
token content is corrupted or replaced, revealing a persistent gap between having a latent channel and using it as an information bottleneck. We recommend TRT as a standard diagnostic alongside accuracy for any method introducing continuous thought
tokens.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
VRXU-net: A Deep Learning Approach for Brain Ischemic Stroke Lesion Detection and Segmentation in T1W MRI
Sayed Amir Mousavi Mobarakeh
When the blood supply to the
brain is obstructed by a clot, oxygen delivery to
brain tissues becomes insufficient, leading to cellular necrosis. In healthcare settings, accurately identifying and delineating ischemic lesion boundaries is essential for treatment and
surgical planning. However, ischemic stroke lesions vary widely in shape, size, and location, and in grayscale MRI modalities such as T1W they may resemble surrounding
brain structures. This makes lesion detection and
segmentation a challenging task for clinicians. This study introduces a novel VRU-Net architecture, derived from visual features, residual connections, and a U-shaped network, for detecting and segmenting ischemic stroke lesions in 3D
magnetic resonance imaging scans. The proposed method first uses a modified VGG model to identify ischemic stroke in separate 2D slices. Then, a U-shaped
segmentation model with residual blocks segments the lesion in each slice. This procedure is applied independently to the axial, sagittal, and coronal planes, and the final output is generated by aggregating the three
segmentation results. To improve both performance and processing speed, a high-performance classifier is applied before the
segmentation model in a sequential framework. This strategy reduces unnecessary
segmentation of non-lesion slices and improves overall accuracy. In addition, decomposing 3D images into 2D slices reduces model complexity while allowing information from three anatomical planes to support more accurate lesion localization. The proposed model is trained on the Anatomical Tracings of Lesions After Stroke
dataset and outperforms state-of-the-art models in terms of accuracy and Dice coefficient. Moreover, the
segmentation output provides feedback that helps the classification model reduce false-positive predictions.
Read the full abstract ▾
Computer Vision
Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly
Aditya Chetan, Eric Cai, Peeyush Kushwaha et al. (+5 more)
The emergence of Large
Vision-
Language Models (LVLMs) has significantly advanced
video understanding capabilities. However, existing
benchmarks focus predominantly on coarse-grained tasks such as action
segmentation, classification, captioning, and
retrieval. Furthermore, these
benchmarks often rely on entities that can be easily identified verbally, like household objects, animals, human subjects, etc., limiting their applicability to complex, in-the-wild
video scenarios. But, many applications such as furniture assembly, cooking, etc., require step-by-step fine-grained spatio-temporal understanding of the
video, which is not sufficiently evaluated in current
benchmarks. To address this gap, we introduce Flat-Pack Bench, a novel
benchmark centered on furniture assembly tasks. Our
benchmark evaluates LVLMs on nuanced tasks, including temporal ordering of assembly actions, temporal localization of assembly state, understanding part mating, and tracking, using multiple-choice questions paired with visual
prompts highlighting relevant parts as references for fine-grained questions. Our experiments reveal that state-of-the-art LVLMs struggle significantly with fine-grained spatio-temporal
reasoning, highlighting their limitations in effectively leveraging temporal information from
videos, limited tracking ability, and understanding of spatial interactions like physical contact.
Read the full abstract ▾
All authors Aditya Chetan, Eric Cai, Peeyush Kushwaha, Bharath Raj Nagoor Kani, Utkarsh Mall, Qianqian Wang, Noah Snavely, Bharath Hariharan
Computer Vision
UniVL: Unified Vision-Language Embedding for Spatially Grounded Contextual Image Generation
Jiayun Wang, Yu Wang, Weijie Gan et al. (+2 more)
We introduce spatially grounded contextual image generation, a controllable image generation task that reframes the conditioning paradigm. Instead of supplying a reference image and a global text
prompt through two separate encoders, one for
vision and one for
language, UniVL is trained to bind semantics to spatial locations directly from a single unified visual input, where the textual instruction is rendered onto the spatial mask. This removes the need for a standalone text encoder at inference time. The resulting model supports contextual image generation by following user-specified instructions about what should appear where, while substantially reducing computation. To address this task, we propose a framework in which the UniVL encoder, adapted from an optical-character-recognition-pretrained backbone, reads the unified condition optically and produces a UniVL
embedding, fVIL, that fuses visual and semantic intent with spatial locations in a single
token sequence. A two-stage pipeline first aligns UniVL with the VAE
embedding space and then conditions a pretrained
diffusion backbone entirely on UniVL
embeddings, eliminating the standalone text encoder, such as T5. Although this reframing uses a deliberately minimal text interface, it yields strong empirical gains. On UniVL-ImgGen, a
benchmark of 477K mask-annotated images that we construct for training and evaluation, UniVL improves image quality over text-prompted baselines, reducing FID from 14 to 11 and increasing PSNR from 16 to 20. It also eliminates the text encoder entirely, reducing inference TFLOPs by up to 52% and runtime by up to 44%. Additional ablation studies validate the contributions of the proposed components, paving the way for efficient, spatially grounded image generation with a unified conditioning paradigm.
Read the full abstract ▾
All authors Jiayun Wang, Yu Wang, Weijie Gan, Zhenting Wang, Wei Wei
Computer Vision
GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillation
Sixiang Chen, Zhaohu Xing, Tian Ye et al. (+7 more)
Open-ended image generation is no longer a simple
prompt-to-image problem. High-quality generation often requires an
agent to combine a model's internal generative ability with external resources. As requests become more diverse and demanding, we aim to develop a general image-generation
agent that can self-evolve through trajectories and use tools more effectively across varied generation challenges. To this end, we propose GenEvolve, a self-evolving framework based on Tool-Orchestrated Visual Experience
Distillation. In GenEvolve, each generation attempt is modeled as a tool-orchestrated trajectory, where the
agent gathers evidence, selects references, invokes generation skills, and composes them into a
prompt-reference program. Unlike existing agentic generation methods that mainly rely on image-level scalar rewards, GenEvolve compares multiple trajectories for the same request and abstracts best-worst differences into structured visual experience, provided only to a privileged teacher branch. Inspired by on-policy self-
distillation, Visual Experience
Distillation provides dense
token-level supervision, helping the student internalize better search, knowledge activation, reference selection, and
prompt construction. We further construct GenEvolve-Data and GenEvolve-Bench. Experiments on public
benchmarks and GenEvolve-Bench show substantial gains over strong baselines, achieving state-of-the-art performance among current image-generation frameworks. Our website is as follows: https://ephemeral182.github.io/GenEvolve/
Read the full abstract ▾
All authors Sixiang Chen, Zhaohu Xing, Tian Ye, Xinyu Geng, Yunlong Lin, Jianyu Lai, Xuanhua He, Fuxiang Zhai, Jialin Gao, Lei Zhu
Computer Vision
Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models
Dong Chen, Fangyun Wei, Ziyu Wan et al. (+18 more)
We introduce Lens, a 3.8B-parameter T2I model that achieves performance competitive with, and in several cases surpassing, state-of-the-art models with more than 6B parameters across various
benchmarks, while requiring significantly less training compute. For example, Lens requires only about 19.3% of the training compute used by Z-Image. The training efficiency of Lens stems from two key strategies beyond its compact model size. First, we maximize data information density per training batch by (i) training on Lens-800M, a
dataset of 800M densely captioned image-text pairs whose captions are generated by GPT-4.1 and contain approximately 109 words on average, providing richer semantic supervision than conventional short captions, and (ii) constructing each batch from images with multiple resolutions and diverse aspect ratios, thereby enlarging the effective visual coverage of each optimization step. Second, we improve convergence speed through careful architectural choices, including adopting a semantic VAE that provides better latent representations and employing a strong
language encoder that accelerates optimization while enabling
multilingual generalization from English-only training data. After pre-training, we apply RL with taxonomy-driven
prompts (Lens-RL-8K) and structured reward rubrics to suppress artifacts and improve visual quality, a reasoner module with training-free system
prompt search to better align user requests with the model, and
distillation-based acceleration for 4-step inference. Through efficient training and systematic optimization, Lens generalizes to arbitrary aspect ratios from 1:2 to 2:1 and resolutions up to 1440^2, and supports
prompts in several commonly used
languages. Thanks to its compact size, Lens generates a 1024^2 image in 3.15 seconds on a single NVIDIA H100 GPU, while its distilled turbo version performs 4-step generation in 0.84 seconds.
Read the full abstract ▾
All authors Dong Chen, Fangyun Wei, Ziyu Wan, Dongdong Chen, Jiawei Zhang, Jinjing Zhao, Sirui Zhang, Yang Yue, Zhiyang Liang, Baining Guo, Chong Luo, Jianmin Bao, Ji Li, Lei Shi, Qinhong Yang, Xiuyu Wu, Xuelu Feng, Yan Lu, Yanchen Dong, Yitong Wang, Yunuo Chen