Computer Vision
Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain
Alessandro Gambetti, Qiwei Han, Cláudia Soares et al. (+1 more)
Vision-
Language Models (VLMs) struggle when applied to medical image-text data, yet the tools available to diagnose this failure remain limited. Existing representation
alignment metrics are symmetric, collapsing both modalities into a single score and hiding which modality drives cross-modal degradation. We introduce the Spectral
Alignment Score (SAS), an asymmetric metric that projects both modalities onto the principal eigenbasis of an anchor modality and computes
eigenvalue-weighted per-eigenmode correlations, resulting in directional scores whose difference quantifies modality information imbalance. We embed SAS within a benchmarking framework evaluating 15 VLMs across natural and medical image-text
datasets alongside 6
alignment metrics and bidirectional
retrieval. Our experiments show that medical images retain richer structural information than their paired clinical reports, a directional asymmetry invisible to all competing metrics, and that SAS achieves the strongest zero-label correlation with
retrieval performance in the medical domain, positioning it as a practical diagnostic tool for clinical deployment. Code is available at this URL: https://github.com/iamalegambetti/medical-vlms-assessment.
Read the full abstract ▾
All authors Alessandro Gambetti, Qiwei Han, Cláudia Soares, Hong Shen
Computer Vision
COMBINER: Composed Image Retrieval Guided by Attribute-based Neighbor Relations
Zixu Li, Yupeng Hu, Zhiwei Chen et al. (+3 more)
Composed Image
Retrieval (CIR) represents a challenging
retrieval task that targets locating specific images through
multimodal inputs. Despite recent progress in CIR techniques, prior approaches often overlook cases where images appear visually alike yet differ in attributes, potentially undermining both
multimodal feature fusion and similarity modeling. To mitigate this limitation, we design a unified representation of cross-modal features based on attribute prototypes. Nevertheless, the task is far from straightforward, owing to three core issues: (1)
entanglement in attribute-level semantics, (2) inconsistency across modalities, and (3) supervised signal missing. To tackle the above obstacles, we introduce a COMposed image
retrieval network guided By attrIbute-based NEighbor Relations (COMBINER). Specifically, we first design an Adaptive Semantic Disentanglement module, which is capable of disentangling attribute features based on
multimodal primitive features. Secondly, we propose a Unified Prototype-based Composition module, which can construct cross-modal unified prototypes (CUP) and facilitate
multimodal feature composition. Finally, we introduce a Dual Relations Modeling module, which can mine pairwise and neighbor relations based on attribute similarity. Compared to traditional neighbor relations modeling CIR methods, COMBINER represents the first study addressing the phenomenon of visually similar but attribute-unrelated samples. It achieves a more accurate understanding of the semantic relations among samples by employing an attribute prototype-based similarity metric. Comprehensive experiments conducted on three
benchmark datasets confirm the effectiveness of our proposed COMBINER. The implementation of our method will be accessed at https://github.com/Lee-zixu/COMBINER
Read the full abstract ▾
All authors Zixu Li, Yupeng Hu, Zhiwei Chen, Haokun Wen, Xuemeng Song, Liqiang Nie
Computer Vision
4D Reconstruction from Sparse Dynamic Cameras
Kazuki Ozeki, Shun Kenney, Yuto Shibata et al. (+6 more)
Although dynamic 3D (i.e., 4D) reconstruction from a
monocular dynamic camera has recently advanced, it remains fundamentally limited by depth ambiguity. In this paper, we focus on an alternative practical way, i.e., sparse dynamic camera setup, where a handful of independently moving cameras capture the same subjects. While keeping capture costs low, this setup introduces multi-view constraints and remains practical for real-world
video production such as sports, concerts, and TV shows. Despite its potential, our experiments show that naive extensions of existing
monocular or dense-fixed camera-based methods are insufficient since they fail to resolve the complex spatiotemporal inconsistencies across views and time. To fill this gap, we propose a simple yet effective 3D track initialization method designed to ensure spatiotemporal consistency by integrating inter-camera feature matching with intra-camera point tracking. Additionally, we incorporate a noise-robust depth-ordering regularization loss and a spatiotemporally diverse batch sampling strategy to enhance optimization stability and cross-view generalization. Furthermore, to address the lack of standardized
benchmarks for this task, we introduce LetCamsGo, a new real-world
video dataset with 5 sequences across 4 diverse environments, recorded by three independently moving cameras and one fixed camera. Comprehensive benchmarking on LetCamsGo demonstrated that our proposed framework improves 4D reconstruction quality in dynamic regions compared with baselines, paving the way for a low-cost 4D reconstruction paradigm in the wild.
Read the full abstract ▾
All authors Kazuki Ozeki, Shun Kenney, Yuto Shibata, Eisuke Takeuchi, Takuya Narihira, Kazumi Fukuda, Ryosuke Sawata, Yuki Mitsufuji, Yoshimitsu Aoki
Computer Vision · Cross-listed from cs.CL
Fine-grained Fragment Retrieval in Multi-modal Long-form Dialogues
Hanbo Bi, Zhiqiang Yuan, Chongyang Li et al. (+7 more)
With the widespread adoption of multi-modal communication platforms, long-form dialogues interleaving text and images have become increasingly common. Users often need to retrieve coherent dialogue fragments related to specific topics, rather than isolated utterances. We propose Fine-grained Fragment
Retrieval (FFR), which locates semantically relevant multi-utterance, multi-image fragments in multi-modal long-form dialogues. We explore two settings: (1) FFR within Single-Dialogue, retrieving fragments from a given dialogue; and (2) FFR within Dialogue Corpus, retrieving from a large-scale corpus for open-domain scenarios. For (1), we introduce F2RVLM, a generation-based
retrieval model trained with reinforcement learning, using multi-objective rewards and difficulty-aware curriculum sampling to enhance fragment coherence. For (2), we develop FFRS, a two-stage system combining offline fragment-level indexing with online
retrieval. Specifically, each dialogue is decomposed into minimal semantic fragments encoded by a Fragment
Embedding Model (FEM) into a vector database; at inference, FEM rapidly recalls Top-K candidates, and F2RVLM performs fine-grained
reasoning to identify the most relevant sub-content. To support FFR, we construct MLDR, the longest multi-modal dialogue
retrieval dataset to date, and a WeChat-based real-world test set. Experiments on both
benchmarks demonstrate that F2RVLM and FFRS consistently achieve superior performance across single-dialogue and corpus-level FFR.
Read the full abstract ▾
All authors Hanbo Bi, Zhiqiang Yuan, Chongyang Li, Qiwei Yan, Zexi Jia, Jiapei Zhang, Xiaoyue Duan, Yingchao Feng, Jinchao Zhang, Jie Zhou
Computer Vision
Impostor: An Agent-Curated Benchmark for Realistic AIGC Manipulation Localization
Zhenliang Li, Yutao Hu, Qixiong Wang et al. (+5 more)
Recent advances in generative image editing have improved the realism and controllability of localized image manipulation, raising new challenges for image manipulation detection and localization (IMDL). However, existing IMDL
benchmarks still have limitations in visual realism, manipulation diversity, and generator coverage, making it difficult to reflect recent trends in image manipulation. To address these limitations, we introduce Impostor, a high-quality AI-edited image manipulation localization
dataset containing 100K manipulated images. Impostor is constructed by CraftAgent, a closed-loop
agent framework that integrates scene perception, editing planning, manipulation execution, quality validation, and iterative reflection to automatically generate diverse and visually realistic manipulated images. Moreover, Impostor contains images generated by seven recent AIGC models across three manipulation types and includes multiple manipulated regions, providing a more comprehensive
benchmark for AIGC-based IMDL. Furthermore, we propose PhaseAware-Net (PANet), a semantic-forensic framework that introduces local phase modeling and semantic-forensic consistency learning to better localize semantically plausible yet forensically disrupted manipulated regions. Extensive experiments show that Impostor poses significant challenges to existing large
vision-
language models (LVLMs) and specialized IMDL methods, while PANet achieves superior performance on Impostor and multiple public
benchmarks.
Read the full abstract ▾
All authors Zhenliang Li, Yutao Hu, Qixiong Wang, Wenpeng Du, Hongxiang Jiang, Jiasong Wu, Xiaolong Jiang, Jungong Han
Computer Vision
Optical-Guided Neural Collapse for SAR Few-Shot Class Incremental Learning
Fan Zhang, Sijin Zheng, Fei Ma et al. (+4 more)
Few-shot class-incremental learning (FSCIL) in synthetic aperture radar imagery presents unique challenges due to severe data scarcity and SAR-specific variability. In particular, strong azimuth sensitivity in SAR induces large intra-class variation and inter-class confusion, and FSCIL sequential updates further lead to catastrophic forgetting of previously learned classes. Inspired by neural collapse, we propose an optical-guided SAR FSCIL framework, which derives orthogonal feature subspaces from a data-rich optical ATR
dataset and uses them as
geometric priors to guide SAR feature learning. SAR features are projected onto these orthogonal subspaces via principal angle constraints, effectively transferring discriminative structure from the optical to the SAR domain. Specifically, our projection loss and the classifier loss optimized with a frozen simplex-ETF geometry jointly induce neural collapse by concentrating features around class means while maintaining large inter-class angles. We evaluate the approach on a
benchmark comprising an optical ATR
dataset and a SAR ATR
dataset with 24 target classes, organized into a base training session and seven incremental sessions. Compared with recent FSCIL methods including NCFSCIL and so on, our method achieves the highest final accuracy and a favorable trade-off between final performance and performance degradation. Moreover, neural collapse metrics show improved intra-class compactness and inter-class separability, indicating that the learned features more closely approximate the ideal simplex-ETF geometry.
Read the full abstract ▾
All authors Fan Zhang, Sijin Zheng, Fei Ma, Qiang Yin, Yongsheng Zhou, Fei Gao, Xian Sun
Computer Vision · Cross-listed from cs.MM
Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation
Yuxuan Bian, Zeyue Xue, Songchun Zhang et al. (+9 more)
We present Echo Infinity, an autoregressive (AR) framework towards real-time infinite
video generation that employs a learnable evolving
memory to dynamically filter, abstract, and compress any-length history at constant cost. Existing methods mainly curate
memory with predefined KV-cache schedules, fixed-ratio heuristic compression, or inference-time RoPE adaptation. These designs inevitably lose historical information and amplify compounding errors due to their limited cache window and ignorance of autoregressive generation noise. Inspired by human
memory consolidation, Echo-Infinity replaces handcrafted
memory curation with learnable
Memory Query, which are updated by
attention and a gating mechanism when past frames are evicted from the local window. The queries are optimized end-to-end with the
video diffusion transformers (DiTs), forming an evolving
memory that supports arbitrary compression ratios with constant computation independent of
video length. They also act as a generalizable generation prior, improving quality even when only the optimized initial state is used. We further introduce Unified Relative RoPE Recipe, which anchors the sink frames to start from id 0 and lets the newest frame id grow at most to the DiTs' pretrained maximum temporal RoPE id throughout training and inference, freeing the model from the finite RoPE constraint and closing the train-test RoPE extrapolation gap. In long and short
video generation, Echo-Infinity achieves state-of-the-art performance, and, to our knowledge, demonstrates promising 24-hour (>1.3 M frames) real-time rollouts for the first time, suggesting a practical path toward infinite
video generation.
Read the full abstract ▾
All authors Yuxuan Bian, Zeyue Xue, Songchun Zhang, Shiyi Zhang, Weiyang Jin, Yaowei Li, Junhao Zhuang, Haoran Li, Jie Huang, Haoyang Huang, Nan Duan, Qiang Xu
Computer Vision
SFMambaNet: Spectral-Frequency Enhanced Selective State Space Model for Correspondence Pruning
Zhihua Wang, Yanping Li and Yizhang Liu
Correspondence
pruning aims to identify inliers from an initial set of correspondences. Most existing Graph Neural Network (GNN)-based methods rely on
geometric features mapped from coarse Euclidean coordinates, which struggle to capture the subtle
geometric consistencies presented by inliers. While Mamba-based methods possess global receptive fields and long sequence modeling capabilities, they tend to accumulate substantial inconsistent features within the hidden state space, making it difficult to distinguish inliers from outliers. In this paper, we integrate frequency domain perception into this task for the first time and propose SFMambaNet, a novel Spectral-Frequency enhanced Mamba-based two-view correspondence
pruning network. Our method is collaboratively composed of two components: First, we design a Local Spectral-
Geometric Attention (LSGA) block. LSGA incorporates spectral positional encoding into local graph interactions and introduces multi-scale Mamba processing to enhance the capture of subtle
geometric consistencies and improve local feature discriminability. Building upon this, we design a Spectral-Integrated Global Mamba (SIGM) block. SIGM embeds a frequency gating mechanism within the state space, utilizing the frequency information provided by LSGA to explicitly suppress high-frequency noise accumulation within hidden states and mitigate the propagation of inconsistent features. This enhances inlier-outlier separability and achieves robust global context modeling capabilities with nearly linear complexity. Extensive experiments demonstrate that SFMambaNet outperforms current state-of-the-art methods on several challenging tasks. The code is available at https://github.com/Kirito14IT/SFMambaNet.
Read the full abstract ▾
Computer Vision
IMPose: Closed-Loop Correction Propagation for Interactive Multi-Person Pose Annotation
Haoyang Ge, Jian Ma, Ziwen Wang et al. (+5 more)
High-quality dynamic human pose annotations provide important supervision for learning fine-grained human motion, yet dense multi-person
video annotation remains labor-intensive. Sparse human corrections can reduce this effort, but their propagated locations may become unreliable and ambiguous across identities. Existing interactive refinement and temporal propagation methods do not explicitly address how imperfect propagated corrections should participate in subsequent multi-person pose inference. We propose IMPose, a correction-conditioned temporal inference framework for interactive multi-person pose annotation. IMPose separates temporal correspondence from pose estimation: propagated corrections are treated as
geometric priors rather than final predictions and are refined by a correction-aware decoder using target-frame visual evidence. A keypoint-aware identity state further combines structured pose geometry with trajectory history to preserve person- and joint-level correspondence. The refined pose and identity state are fed back to subsequent frames, forming a closed temporal correction loop. Under a controlled sparse-correction protocol on PoseTrack21, one correction improves mean average precision (mAP) from 38.87 to 67.99 and Higher Order Tracking Accuracy (HOTA) from 52.21 to 64.89. At a 20-frame correction interval, IMPose achieves 89.03% keypoint-level Object Keypoint Similarity (K-OKS), compared with 81.41% for CoTracker and 70.12% for PoseWarper-based propagation. We further extend PoseTrack21 with 188K additional person instances through 3.55M annotation updates to keypoints, bounding boxes, and person IDs using 100 annotator-hours. The annotation tool, code, and extended
dataset will be released.
Read the full abstract ▾
All authors Haoyang Ge, Jian Ma, Ziwen Wang, Qihe Wang, Jianqi Fan, Hongzhi Yu, Xingyu Chen, Kun Li
Computer Vision
Evaluating Reasoning Fidelity in Visual Text Generation
Jiajun Hong and Jiawei Zhou
Recent text-to-image (T2I) models can render highly legible and well-structured text within images, enabling applications including document generation and slide generation. However, it remains unclear whether such systems faithfully preserve
reasoning ability when complex solutions must be expressed directly through rendered text, or whether they merely imitate surface-level patterns. We investigate this question by evaluating
reasoning fidelity in visual text generation, where models must express complete
reasoning processes as images. Our evaluation includes long text rendering, factual knowledge probing, context understanding, and multi-step
reasoning. Across these settings, we find that current T2I models frequently produce semantic errors, logical inconsistencies, and incorrect intermediate steps, even when the rendered text appears visually clear. These failures contrast with the strong
reasoning performance of text-only models on the same tasks. Our findings reveal a substantial gap between visual text generation and procedural
reasoning, motivating more reliable visual text
reasoning.
Read the full abstract ▾
Computer Vision
Adaptive Calibration for Fair and Performant Facial Recognition
Ryan Brown and Chris Russell
We introduce Adaptive Calibration (AC), a novel calibration strategy for facial recognition that maps cosine similarity between normalized
embeddings to well-calibrated probabilities. By incorporating local context into calibration, Adaptive Calibration corrects for a fundamental mismatch in cosine similarity, whereby the same distance can correspond to different match probabilities in different
embedding regions. Our approach improves both overall performance and results in a fairer calibration without requiring demographic metadata. Our approach often improves worst-group ranking and probability quality across a variety of pretrained models and standard
benchmarks, with gains in low-FPR verification performance depending on the setting. AC provides a practical solution for equitable facial recognition, without requiring demographic group annotations, and while improving overall performance. Our method provides continuous, region-specific calibration, and we examine per-group performance to assess "leveling down" where
fairness comes at the cost of degraded performance for some groups.
Read the full abstract ▾
Computer Vision
ChannelTok: Efficient Flexible-Length Vision Tokenization
Sukriti Paul, Arpit Bansal and Tom Goldstein
Leading flexible
vision tokenizers achieve SOTA quality at an extreme cost, relying on parameter-heavy backbones and slow, multi-step generative decoders. We depart from this complex, spatial-
token paradigm and introduce a simple, lightweight, and fast channel-wise flexible-length tokenizer. Our method treats each latent channel as a visual
token, enabling a parameter-efficient CNN-
Transformer hybrid backbone. Furthermore, employing a
stochastic tail-dropping paradigm during training naturally forces channels to organize by semantic importance. This allows for flexible compression at inference by simply retaining the first $k$ channels, and naturally enables variable-length autoregressive image generation. We validate our approach through extensive experiments on ImageNet, demonstrating consistent quality across diverse
token budgets. The results establish a new quality-efficiency frontier: our model achieves state-of-the-art perceptual quality (rFID 2.92) while being $8.6\times$ faster in decoding and $2.1\times$ smaller (159M params) than the next-best alternative. Our work establishes channel-wise tokenization as a powerful and practical paradigm for efficient visual representation. Project page: https://channeltok.github.io
Read the full abstract ▾
Computer Vision
Imagine Before You Draw: Visual Prompt Engineering for Image Generation
Liyu Jia, Fengda Zhang, Jiachun Pan et al. (+7 more)
Incorporating visual semantic representations as an intermediate step before image generation can reduce the modeling difficulty between text and images, thereby improving generation quality. Recent works such as X-Omni and BLIP3o-Next have explored this direction, but they typically use a two-stage external pipeline: a separate autoregressive model first generates semantic
tokens, which are then fed as conditioning to an independent
diffusion decoder. When the decoder is conditioned only on the compressed semantic plan, it cannot directly access the original input, which introduces an information bottleneck that limits detail preservation in downstream tasks such as editing. Internal architectures such as Transfusion, BAGEL, and Show-o2 avoid this bottleneck by enabling cross-modal interaction within a single model, but they still face the difficult text-to-pixel modeling gap without intermediate semantic guidance. We propose Visual
Prompt Engineering (VPE), which can be seamlessly integrated into such internal frameworks. Specifically, the model first autoregressively generates visual semantic
tokens ($e.g.$, SigLIP 2) as "visual
prompts" that capture the semantic layout, then generates the full image
tokens conditioned on this plan. We validate VPE across class-conditional generation, text-to-image generation, and image editing, covering various
token types and model architectures. Results show that VPE can accelerate convergence, raise quality ceilings, and through internal integration, achieve substantially better editing preservation (PSNR: $26.76$ vs. $19.92$) than a matched external pipeline with semantic-only decoder conditioning, while maintaining competitive editing responsiveness.
Read the full abstract ▾
All authors Liyu Jia, Fengda Zhang, Jiachun Pan, Kesen Zhao, Saining Zhang, Wang Lin, Weijia Wu, Yue Liao, Aojun Zhou, Hanwang Zhang
Computer Vision
Radiomic Feature Selection Using Gradient Loss of Deep Neural Network for Lung Cancer Stage Detection
Hina Shakir, Mohammad Mohatram, Javeed Hussain et al. (+2 more)
Radiomics enables extraction of quantitative imaging biomarkers from medical images and has become an important tool for computer-aided cancer diagnosis. However, radiomics
datasets are typically high-dimensional with limited samples, making feature selection a critical step for building reliable predictive models. This study proposes a Gradient-Loss Recursive Feature Elimination (GL-RFE) framework that integrates gradient sensitivity analysis from a deep neural network to identify the most influential radiomic features for lung cancer stage detection. A total of 106 radiomic features were extracted from chest Computed
Tomography (CT) scans using the PyRadiomics extension of the 3D Slicer platform. The proposed method evaluates feature importance by computing gradients of the network loss with respect to input features and recursively eliminates features with minimal contribution. The resulting top-15 radiomic features are used to train a deep neural network classifier for distinguishing early-stage and advanced-stage lung cancer. The proposed framework achieves strong classification performance, with accuracy of 90.22%, precision of 90.10%, recall of 90.24%, and F1-score of 90.16% on the test
dataset. Visualization analyses, including correlation heat maps and distribution plots, further confirm reduced feature redundancy and improved class separability. Compared to conventional feature selection techniques, GL-RFE effectively captures nonlinear feature interactions and enhances model generalization. The presented protocol provides a reproducible and interpretable methodology for radiomics-based cancer stage detection and is particularly suitable for high-dimensional, small-sample biomedical
datasets, with potential applications in other domains such as genomics and
multimodal clinical analysis.
Read the full abstract ▾
All authors Hina Shakir, Mohammad Mohatram, Javeed Hussain, Syed Rizwan Ali, Muhammad Irfan Memon
Computer Vision
INTACT: Ego-Guided Typed Sparse Evidence Retrieval for Heterogeneous Collaborative Perception
Chen Li, Shengrong Yuan, Jialong Zuo et al. (+3 more)
Collaborative perception extends the perceptual range of autonomous vehicles by sharing information across
agents, but heterogeneous sensors and perception models make intermediate feature fusion difficult to deploy at scale. Existing heterogeneous collaboration methods typically follow a translation-first paradigm: collaborator features must be aligned, adapted, or projected into an ego-compatible space before fusion. Such feature-compatibility contracts improve fixed-system performance, but they couple deployment to collaborator-specific adaptation and make newly joined heterogeneous
agents costly to integrate. To address this gap, we propose INTACT, an ego-guided typed sparse evidence
retrieval framework for heterogeneous collaborative perception. Instead of translating an entire collaborator feature map, INTACT lets the ego vehicle issue typed evidence queries that express suspected objects and evidence-deficient regions. Collaborators respond only with local evidence at queried locations, and the ego selects useful responses through sparse per-query routing and injects them through gated residual write-back. This changes the compatibility requirement from global feature-map interpretability to local, typed response comparability under ego-issued queries, enabling a zero-training heterogeneous insertion protocol in which the ego interface is trained once and new collaborators join through checkpoint merging. Extensive experiments on simulated and real-world heterogeneous collaborative perception
benchmarks validate the effectiveness and deployability of INTACT. On OPV2V-H, INTACT achieves 80.1 AP70 with only 0.52M additional parameters and 18.0 $\log_2$ communication volume, corresponding to about 16$\times$ compression over dense feature transmission. On DAIR-V2X, INTACT achieves 43.8 AP50 under challenging real-world conditions.
Read the full abstract ▾
All authors Chen Li, Shengrong Yuan, Jialong Zuo, Xinzhong Zhu, Nong Sang, Changxin Gao
Computer Vision
3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training
Jiaxin Shi, Xidong Zhang, Fucai Zhu et al. (+3 more)
We propose a 3D-thinking-guided co-training framework that enables
vision-
language-action (VLA) models to perform 3D spatial
reasoning implicitly during action prediction. Our core insight is that 3D geometry perception and 3D spatial
reasoning are distinct capabilities that can be disentangled and injected at different feature hierarchies. During training, three tightly coupled components work in concert primarily within the latent space: (1) To gain
geometric priors, a latent 3D geometry perception module aligns intermediate visual features with a 3D foundation model, acquiring low-level
geometric cues without architectural modifications to the VLM backbone. (2) Complementing this, an online 3D
reasoning distillation module mitigates the
prompt-induced
reasoning gap via a shared
reasoning anchor
token. During 3D VLM co-training, this anchor is emitted as the first output
token to robustly encode spatial priors. During VLA training, it serves as an input
token inserted between the task and action instructions, transferring high-level spatial thinking from explicit teacher
reasoning prompts to student action
prompts without chain-of-thought text generation. (3) These disentangled
geometric and
reasoning features are then united by a spatially augmented action integration, which jointly injects them into the action-query
tokens as hierarchical spatial conditions to prevent action shortcuts. At deployment, our method retains only its lightweight adapters to perform implicit 3D
reasoning, discarding the 3D foundation model and the teacher branch used for supervision. Consequently, it operates purely on 2D images without 3D sensors, external models, or explicit text generation while preventing catastrophic forgetting of the pretrained VLM, achieving state-of-the-art performance on LIBERO, LIBERO-PLUS, SimplerEnv, and real-world manipulation tasks.
Read the full abstract ▾
All authors Jiaxin Shi, Xidong Zhang, Fucai Zhu, Zhe Li, Siyu Zhu, Weihao Yuan
Computer Vision
Niloufar Alipour Talemi, Hossein Kashiani and Fatemeh Afghah
Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for
Multimodal Large
Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks. Despite its flexibility,
multimodal ICL incurs high inference latency and suffers from instability due to sensitivity to demonstration formatting, ordering, and content. To address these limitations, we propose Hyper-ICL, a lightweight, training-based framework for demonstration-free
multimodal ICL that reconstructs demonstration effects directly without requiring ICDs at inference time. Hyper-ICL learns a parameter-efficient low-rank logit-level adapter that calibrates
attention distributions to better match demonstration-induced
attention redistribution. To capture how demonstration influence varies across queries, we introduce a query-adaptive modulation mechanism that adaptively controls intervention strength at
token level across layers and heads based on the current query. Finally, we propose a layer-wise
hyperbolic anchor
distillation loss that aligns intermediate student features to a demonstration-conditioned teacher via Lorentz geodesic distance. This loss encourages the student to reconstruct the demonstration-query relationships induced by ICDs. Extensive experiments across six different
multimodal benchmarks (including VQAv2, OK-VQA, and COCO Caption) demonstrate that Hyper-ICL consistently improves accuracy and stability over vanilla ICL and existing state-of-the-art methods.
Read the full abstract ▾
Computer Vision
Stateful Visual Encoders for Vision-Language Models
Zirui Wang, Junwei Yu, Adam Yala et al. (+3 more)
Vision-
language models (VLMs) are increasingly used in multi-image, multi-turn agentic settings where decisions depend on visual changes. However, in existing open-weight VLMs, visual comparisons happen only inside the
language model, while the visual encoder itself remains stateless: each image is encoded independently, without access to the prior visual context. As a result, small but task-critical changes may be attenuated before the
language model has a chance to compare them, especially when those changes do not affect the high-level semantics of the scene. We introduce a Stateful Visual Encoder, which conditions each visual representation on prior visual features. Under supervised finetuning, VLMs equipped with stateful encoders achieve consistent improvements on controlled tasks involving cross-image spatial aggregation, multi-object visual differencing, and visual trajectory behavior cloning. These improvements are consistent across input resolutions,
language model sizes, and VLM backbones. Finally, we validate our model on real-world tasks, including longitudinal radiology, fine-grained image comparison, and remote sensing, where stateful encoders consistently improve generalist VLM baselines and can match or surpass specialized models in selected domains. Project page: https://statefulvisualencoders.github.io/
Read the full abstract ▾
All authors Zirui Wang, Junwei Yu, Adam Yala, David M. Chan, Joseph E. Gonzalez, Trevor Darrell
Computer Vision
DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation
Thanh-Tung Le, Yunhan Zhao, Menglei Chai et al. (+5 more)
Video diffusion transformers have achieved state-of-the-art visual quality, but their high inference cost remains a major bottleneck for real-time applications. Recent
distillation frameworks produce autoregressive
video diffusion models with reduced latency, yet these models still use a fixed number of denoising steps per frame, wasting computation on predictable frames and under-refining challenging ones. We present DSA, a confidence-guided adaptive computation framework for AR
video diffusion. DSA introduces a lightweight confidence head, trained jointly with the generator under a distribution-matching
distillation objective, to estimate per-frame denoising reliability. At inference, this confidence signal dynamically adjusts the number of
diffusion steps: simple frames terminate early for speed, while complex frames receive additional refinement. Our method requires no extra
video data, no heuristics, and little architectural modification. Experiments show that DSA achieves real-time autoregressive
video generation, reaching 22.63 FPS with sub-second latency on H100 GPUs, while maintaining competitive or superior VBench quality compared to recent autoregressive and bidirectional
video diffusion models. Our results demonstrate that confidence-guided adaptive sampling provides an effective and practical path toward interactive
video generation.
Read the full abstract ▾
All authors Thanh-Tung Le, Yunhan Zhao, Menglei Chai, Zhengyang Shen, Zhe Cao, Danhang Tang, Xiaohui Xie, Deying Kong
Computer Vision
Implicit Fuzzification via Bounded Noise Injection for Robust Medical Image Segmentation
Bisheng Tang, Zhangfeng Ma, Chuchu Zhai et al. (+4 more)
Image
segmentation remains fundamentally limited by boundary ambiguity arising from sampling-induced information loss and inherent uncertainty in pixel-wise labeling. Although encoder-decoder architectures such as U-Net achieve strong performance, they often produce overconfident predictions that fail to capture transition-region ambiguity. To address this issue, we propose NoiseUNet, a simple yet effective framework that injects bounded perturbations into skip connections to regularize cross-scale feature fusion. This mechanism enforces robustness to local feature variations and promotes boundary-aware representations. Theoretically, the perturbation induces an implicit fuzzification effect, yielding soft, data-driven memberships without requiring explicit fuzzy modeling. We further introduce ThyR, a real-world thyroid ultrasound
dataset with inherently ambiguous boundaries. Experiments demonstrate that NoiseUNet consistently improves both
segmentation accuracy and boundary fidelity.
Read the full abstract ▾
All authors Bisheng Tang, Zhangfeng Ma, Chuchu Zhai, Feng Dong, Yaoqun Wu, Ammar Oad, Yifei Peng
Computer Vision · Cross-listed from eess.IV
L-TGVN: Leveraging Longitudinal Priors for Personalized Rapid MRI
Arda Atal{\i}k, Sumit Chopra and Daniel K. Sodickson
MRI provides excellent soft-tissue contrast without ionizing radiation, but long acquisition times increase patient discomfort while also raising exam costs and limiting scanner throughput. A common approach to reduce scan time is to acquire fewer measurements, which yields an ill-posed linear inverse problem; recovering diagnostic-quality images therefore requires incorporating prior knowledge beyond the measured data. In follow-up exams, the most recent prior scan of a patient can provide a highly informative subject-specific context, but practical use is complicated by temporal changes (including pathology progression), misalignment between scans, and protocol drift across acquisitions. In this work, we introduce L-TGVN, a Longitudinal Trust-Guided Variational Network that leverages prior scans as side information to reconstruct the current scan from heavily undersampled measurements. Crucially, L-TGVN constrains the influence of prior scans to be consistent with the acquired measurements. Unlike many existing longitudinal reconstruction methods, it does not require explicit pre-registration between prior and current scans. It further accommodates differences in acquisition protocols across visits (e.g., changes in sequence parameters). We evaluate L-TGVN against matched-capacity baselines, including prior-guided methods and methods that do not use longitudinal priors, and observe consistent improvements in standard quantitative metrics together with better preservation of fine structures at challenging accelerations. Source code is available at github.com/sodicksonlab/L-TGVN.
Read the full abstract ▾
Computer Vision
Motion-Guided Causal Disentanglement for Robust Multi-View Cine Cardiac MRI Diagnosis
Chuankai Xu, Cristiane De Carvalho Singulane, Mohammad Abuannadi et al. (+10 more)
Multi-view cardiac
magnetic resonance (CMR) imaging provides complementary anatomical information and is widely used for noninvasive disease assessment. Recent
transformer-based models have demonstrated strong representation learning capabilities for CMR analysis; however, they typically learn unified latent
embeddings that entangle view-specific anatomical variations with disease-related features. Such
entanglement biases classifiers toward structural attributes rather than view-invariant pathological patterns. This issue is exacerbated in low-data regimes, particularly for underrepresented cardiac conditions, where limited samples increase the susceptibility to shortcut learning and view-dependent decision boundaries. To address this, we propose a Motion-Guided View–Disease Disentanglement framework MoViD built upon a ViT-MAE backbone. The model explicitly factorizes latent representations into view-specific and disease-discriminative components using dual-branch supervised contrastive objectives and a gradient-reversal
adversarial constraint that minimizes disease leakage into the view
embedding. Additionally, an annotation-free temporal motion feature, derived from inter-frame difference maps, is introduced to localize the beating heart region and suppress background artifacts. A focal reweighting mechanism is incorporated into the contrastive loss to mitigate class imbalance. We evaluate the framework on a private clinical venous thrombosis
dataset and two public
benchmarks (M&Ms, M&Ms2). Across disease classification and cardiac
segmentation tasks, our approach consistently outperforms standard
transformer baselines and demonstrates competitive performance against large-scale pretrained foundation models, validating the efficacy of structural disentanglement in medical image analysis.
Read the full abstract ▾
All authors Chuankai Xu, Cristiane De Carvalho Singulane, Mohammad Abuannadi, Stephen Chandler, Jeremy Slivnick, Karolina Zareba, Jane Cao, Vidya Nadig, Fabio Fernandes, Seth Uretsky, Diego Perez de Arenaza, Amit Patel, Jianxin Xie
Computer Vision
Ultra-Fast Neural Video Compression
Jiahao Li, Wenxuan Xie, Zhaoyang Jia et al. (+4 more)
While neural
video codecs (NVCs) have demonstrated superior compression ratio, their prohibitive computational complexity remains a critical barrier to real-world deployment. This paper introduces a chunk-based coding framework designed to significantly improve the rate-distortion-complexity trade-off. Instead of processing frames sequentially, our approach encodes a chunk of multiple frames into a single compact latent representation and decodes them simultaneously. This is enabled by cross-frame interaction modules for joint spatial-temporal modeling and frame-specific decoders for parallel reconstruction. This paradigm not only dramatically enhances coding throughput but also facilitates more effective modeling of long-term temporal correlations. To further boost speed, we propose a streamlined
entropy coding mechanism that consolidates bit-stream interactions into a single step, substantially reducing decoding overhead. Building on these innovations, we present DCVC-UF (Ultra-Fast), a new NVC that sets a new SOTA in performance. Our experiments show that DCVC-UF can achieve ultra-fast encoding and decoding speeds, significantly outperforming previous leading codecs. DCVC-UF serves as a notable landmark in the journey of NVC
evolution. The code is at https://github.com/microsoft/DCVC.
Read the full abstract ▾
All authors Jiahao Li, Wenxuan Xie, Zhaoyang Jia, Bin Li, Zongyu Guo, Xiaoyi Zhang, Yan Lu
Computer Vision
An Empirical Study of Data Scale, Model Complexity, and Input Modalities in Visual Generalization
Yidi Zhouluo
Modern deep neural networks usually have large parameter scales and nonlinear hierarchical structures, and they have achieved strong performance in computer
vision. However, the source of their generalization performance remains difficult to explain using traditional statistical learning theory. Among the factors that may affect visual generalization, data scale, model complexity, and input modalities are fundamental and controllable variables. This study empirically analyzes how these three factors influence model generalization performance. Specifically, in a preliminary experiment, we construct a one-dimensional nonlinear function and vary the number of training samples and the
polynomial degree to observe the effects of data scale and model complexity on model performance. In the main experiments, we compare model performance on CIFAR-10 and CIFAR-100 under different training data scales, model architectures, and input modalities. The experimental results show that increasing the training data scale consistently improves generalization performance, whereas changes in model complexity do not provide stable gains. In addition, removing color information degrades model performance, while explicit prior features such as gradients, edges, and wavelets have inconsistent effects across different model architectures. Overall, this study provides an empirical analysis of the relationships among data scale, model complexity, input modalities, and visual generalization performance. Code and experimental logs are available at: https://github.com/YidiZhouluo/DeepLearning-Empirical-Studies/tree/main/Exp_01.
Read the full abstract ▾
Computer Vision
Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models
Shuwen Yu, Zhanxuan Hu, Yi Zhao et al. (+2 more)
Foundation models have driven rapid progress in computer
vision, yet the two dominant paradigms,
vision-
language foundation models (VLMs) and
vision-only foundation models (VFMs), remain only partially compatible. VLMs offer
language-grounded semantic
alignment but are often visually coarse, while VFMs learn discriminative perceptual geometry but lack semantic grounding. We propose GPUA (Geometry-Preserving Unsupervised
Alignment), a framework that integrates the complementary strengths of VFMs and VLMs. Inspired by cross-lingual
alignment, GPUA treats VFM features as a visual
language and learns an orthogonal mapping that translates the VFM space into the VLM semantic space, preserving geometry and narrowing the modality gap without labels or model parameter updates. GPUA is task-agnostic and requires only feature-level access to pretrained models. Experiments across diverse
benchmarks demonstrate improved cross-model compatibility and strong gains in downstream zero-shot recognition and
segmentation with negligible overhead. Code is available at https://github.com/Yuteam14/GPUA
Read the full abstract ▾
All authors Shuwen Yu, Zhanxuan Hu, Yi Zhao, Yonghang Tai, Huafeng Li
Computer Vision
Selective Coupling of Decoupled Informative Regions: Masked Attention Alignment for Data-Free Quantization of Vision Transformers
Biao Qian, Yang Wang, Yong Wu et al. (+1 more)
Data-Free Quantization (DFQ) addresses data security concerns by synthesizing samples, without accessing real data. It has garnered increasing
attention in the context of
Vision Transformers (ViTs), owing to the superiority of the self-
attention mechanism compared to classical convolutional operation. However, previous DFQ arts for ViTs often suffer from a distribution mismatch between synthetic samples and input distribution expected by quantized models Q, resulting in the suboptimal performance. In this paper, we propose a novel Masked
Attention Alignment approach for Data-Free Quantization of ViTs, named MaskAQ, revealing that: 1) the semantics in the self-
attention mechanism is predominantly localized to a sparse subset of patches, called informative regions; 2) the informative regions dominate the mutual information between synthetic samples and Q's outputs. To these ends, we incorporate differential
entropy maximum over patch similarity of synthetic samples, to decouple informative regions from noisy background. To couple with varied Q, the informative regions are selected to align full-precision models with Q via a masked
attention alignment objective, thus yielding high-quality synthetic samples. Furthermore, a periodic sample refreshing strategy comes up to endow MaskAQ with the capacity to continually adapt to the evolving state of Q throughout the training process, to preserve desirable mutual information with synthetic samples. Extensive experiments verify the merits of MaskAQ over state-of-the-art approaches across multiple backbones and downstream tasks. Our code is available at https://github.com/hfutqian/MaskAQ.
Read the full abstract ▾
All authors Biao Qian, Yang Wang, Yong Wu, Jungong Han
Computer Vision
VT-3DAD: Cross-Category 3D Anomaly Detection via Visual-Text Normal Space Alignment
Zi Wang, Katsuya Hotta, Yawen Zou et al. (+4 more)
Few-shot cross-category 3D anomaly detection aims to determine whether an unknown point cloud belongs to a target normal category using only a few normal references. Existing training-based methods usually require category-wise optimization, while recent training-free methods based on multi-view CLIP visual features mainly rely on visual similarity and may be confused by geometrically similar categories. In this paper, we propose VT-3DAD, a training-free framework for cross-category 3D anomaly detection via Visual-Text Normal Space
Alignment. Given few-shot normal references and a test point cloud, VT-3DAD first generates realistic multi-view depth maps and extracts view-wise features using a frozen CLIP visual encoder. The visual branch measures reference-test deviation in the multi-view feature space. In parallel, depth-aware and 3D-aware
prompts are encoded by the frozen CLIP text encoder to construct textual normal anchors, which provide semantic normality constraints for the target category. The final anomaly score is obtained by fusing visual deviation from normal references and semantic deviation from the textual normal space. Experiments on the ShapeNetPart
dataset demonstrate that VT-3DAD achieves state-of-the-art performance. In particular, VT-3DAD improves the one-shot average AUC-ROC from 92.49% to 94.80% compared with the visual-only baseline, while also reducing the average standard deviation from 5.64 to 3.41.
Read the full abstract ▾
All authors Zi Wang, Katsuya Hotta, Yawen Zou, Koichiro Kamide, Yijin Wei, Chao Zhang, Jun Yu
Computer Vision
Multi-Granularity 3D Kidney Lesion Characterization from CT Volumes
Renjie Liang, Zhengkang Fan, Jinqian Pan et al. (+4 more)
Radiology reports describe kidney lesions by type, size, enhancement, and attenuation, yet existing 3D methods predict only at the patient or organ level. We reformulate kidney CT characterization as a per-lesion set-prediction task: one model emits a variable number of lesions per kidney, each with four clinical attributes. We curated 2,619 CT volumes from 788 patients at one academic medical center, with multi-granularity side- and per-lesion labels, and used KiTS23 (489 cases) for zero-shot external validation. We propose LesionDETR, a DETR-style architecture with size-distance Hungarian matching and a hierarchical loss that aggregates per-slot outputs to side-level objectives. Across four input representations and six encoder initializations, two design choices dominate: a
segmentation mask as an input channel, and same-domain abdominal pretraining (SuPreM); generic large-corpus pretraining is no better than random initialization. LesionDETR reaches bilateral side-level abnormality AUC $0.799 \pm 0.009$ on UF-Health and $0.817 \pm 0.072$ on KiTS23. A count-conditioned variant reaches per-lesion mAP $0.190 \pm 0.083$ on cystic lesions; rare solid-lesion AP stays at the noise floor, pointing to targeted data collection, not architecture, as the next bottleneck. The framework yields verified per-lesion predictions for downstream structured report generation.
Read the full abstract ▾
All authors Renjie Liang, Zhengkang Fan, Jinqian Pan, Chenkun Sun, Jiang Bian, Russell Terry, Jie Xu
Computer Vision
Spatially Grounded Concept Bottleneck Models via Part-Factorized Attention
Dhanesh Ramachandram
Concept bottleneck models (CBMs) predict a layer of human-named attributes before predicting a class, which makes their decisions auditable. On fine-grained recognition tasks, though, the concept heads are usually free to attend anywhere in the image, so a head named for one body region can be satisfied by evidence on another, and the model reaches the right answer for the wrong reason. We propose a part-factorized CBM (PF-CBM) that removes this freedom by construction. A frozen DINOv3
vision transformer feeds a set of part queries, each tied by name to a specific anatomical region through a fixed concept-to-part map, while whole-object attributes such as size and shape are handled separately by a query with no spatial prior, since they are not anchored to any single body part. A learnable Gaussian prior over patch locations, initialized from average keypoint positions, keeps the part queries from collapsing onto the same evidence. On its own this prior spreads the queries apart but does not reliably land them on the correct anatomy. What closes that gap is a lightweight
alignment loss that nudges each part query toward its keypoint, and the central finding of this paper is how little of that supervision is required. Aligning on well under one percent of the training images already moves pointing accuracy from near-chance to roughly three-quarters of what full keypoint supervision achieves, and the gains continue, more slowly, as more annotated images are added. Classification accuracy on CUB-200-2011 barely moves across this entire range and stays within a point of a fully supervised baseline whether the model sees no keypoints at all or every one of them. Grounding a CBM's
attention to the right evidence turns out to be nearly free in accuracy and cheap in annotation, provided the model has the right inductive bias to make efficient use of that small amount of supervision.
Read the full abstract ▾
Computer Vision
Frames2LoRA: Parametric Video Internalization for Vision-Language Models
Manan Suri, Sarvesh Baskar and Dinesh Manocha
Processing
video in
vision-
language models is expensive: each frame occupies hundreds of
tokens, and inference cost scales with every frame and every repeated query. We introduce Frames2LoRA, a method for parametric
video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a
video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Frames2LoRA predicts these weights directly from the
video. Trained for SmolVLM2 500M and 2.2B on
video summarization and captioning, Frames2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual
tokens in its context at query time. Frames2LoRA is statistically non-inferior and equivalent to direct
video-in-context inference across all five captioning
benchmarks at both model scales, and across seven of eight
video question answering
benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct
video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-
token load by up to 1,500x and query TTFT by 6-80x, while preserving
video-faithful outputs. We also find that independently generated adapters for non-overlapping
video segments can compose in rank space, suggesting a path toward chunked long-
video internalization.
Read the full abstract ▾