Computer Vision
MAGIC-SSCIL: Manifold Anchoring and Geometric Incremental Calibration for Semi-Supervised Class Incremental Learning
Yousef Abdi, Mohammad Asadpour and Yousef Seyfari
Semi-supervised Class Incremental Learning (SSCIL) is a severe challenge for neural networks, and it is hardest in the exemplar-free setting where no past data may be stored. Existing methods forget catastrophically due to feature drift, and their pseudo-labels become increasingly unreliable as the label space grows. In this paper, we propose MAGIC (
Manifold Anchoring and
Geometric Incremental Calibration), a framework that stabilizes plasticity without storing exemplars. MAGIC's design centers on two components. The first is Soft-Weighted Geometry Calibration (SWGC), which uses graph-based label propagation on the learner's plastic feature space to weight and calibrate class means and variances computed on the frozen backbone; from these calibrated Gaussians, we sample phantom features that stand in for data from previous tasks. The second is a
Geometric Structural
Alignment (GSA) objective that preserves representation
topology by matching the relational structure of student and teacher heads and aligning feature anchors with the fixed classifier prototypes, locking the orientation of the feature space. Together, these constraints keep the adapter from drifting, so
geometric relations between classes remain stable as new classes arrive. We implement MAGIC with a frozen ResNet-18 backbone and a learnable plastic adapter. Across CIFAR-100, CUB-200, and ImageNet-R, at label ratios of 1%, 5%, and 10%, MAGIC improves average incremental accuracy over most of the supervised CIL methods equipped with FixMatch and native SSCIL baselines; the largest gains occur in the fine-grained, low-label setting, where confidence thresholding fails most clearly.
Read the full abstract ▾
Computer Vision
LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents
Zijian Wang, Junnan Zhu, Rongzhen Li et al. (+7 more)
Long-
video understanding requires models to efficiently acquire and reuse sparse visual evidence from long and redundant
video streams. Recent
video tool-use
agents address this challenge by iteratively invoking visual Tools at different temporal scales, but their Tool-Planner communication typically relies on textual observations. Such text-only interfaces provide lossy summaries of Tool computations, causing previously computed visual evidence not verbalized to be discarded and unavailable for subsequent planning. We identify this limitation as the Tool observation bottleneck and propose Latent Visual Evidence-Enhanced Planning (LAVE), a training-free framework for reusing latent visual evidence from completed Tool calls. LAVE introduces a dual-channel observation interface: the visible channel preserves the original textual trajectory, while the latent channel stores pre-verbal visual updates with their Tool roles, source-frame timestamps, and visual locations. During planning, LAVE retrieves evidence relevant to the current Planner state but not covered by textual observations, and integrates it through bounded timestamp-aligned latent updates with
entropy-constrained frame-time routing. This enables
video agents to reuse existing visual computation without additional training, frame replay, or modifications to the original orchestration. Extensive experiments on
Video-MME, LongVideoBench, and CG-Bench show that LAVE consistently improves
video tool-use
agents across backbones. Under a comparable frame budget, LAVE improves the
Video-MME overall score by 3.76 points over the strongest baseline, demonstrating the effectiveness of latent visual evidence reuse for multi-step
video-
agent planning.
Read the full abstract ▾
All authors Zijian Wang, Junnan Zhu, Rongzhen Li, Xiao Liu, Guohui Xiang, Quan Lu, Lijia Liu, Yining Wang, Jiang Zhong, Kaiwen Wei
Computer Vision
ComplexityWorld: Benchmarking Vision-Language Models on Verifiable Visual Decision Making
Ningxin Pan, Hanyu Li and Yehui Tang
Vision-
language models (VLMs) have made rapid progress in visual perception and increasingly support real-world tasks that depend on images. Many such tasks, however, require more than rec- ognizing what an image contains: a model must use visual evidence to make a complete decision whose parts jointly satisfy global constraints. We introduce COMPLEXITYWORLD, a
benchmark of 390 tasks across 39 domain-inspired visual worlds and 29 decision categories. Each task is generated from a hidden structured specification, rendered as a visual scene, and scored by an exe- cutable verifier that accepts any feasible solution. Under direct inference, all evaluated models ex- cept GPT-5.6-Sol remain below 40% verifier ac- ceptance rate (VAR), while GPT-5.6-Sol reaches 75.6%. Performance improves substantially when the same decision information is made explicit in structured form, yet varies sharply across equiva- lent visual presentations.
Agent scaffolds provide smaller, model-dependent gains. Together, these results reveal a persistent visual-to-decision bot- tleneck that additional inference alone does not remove.
Read the full abstract ▾
Computer Vision
Predictive Failure Detection in Network Hardware Using Thermal Imaging and Deep Learning with Sensor Fusion
Ashly Joseph
Unplanned network hardware malfunctions can interrupt services and result in expensive downtime in data centers. A deep learning-based predictive maintenance strategy is presented that utilizes thermal imaging and power sensor data to detect early indicators of equipment breakdown in routers, switches, and servers. A simulated
dataset was generated comprising annotated thermal pictures and power readings indicative of three operating states: Normal, Warning, and Critical. Three ImageNet-pretrained convolutional neural network (CNN) models ResNet-50, InceptionV3, and VGG16 were assessed together with a multi-modal CNN-LSTM fusion model that integrates visual and sensor time-series information. Experiments were performed with and without pre-processing procedures, including region-of-interest (ROI) extraction and normalization. In the absence of pre-processing, CNNs attained moderate accuracy (e.g., ResNet-50 at 52%), but ROI-based pre-processing significantly enhanced performance (ResNet-50 accuracy reaching 91%). The CNN-LSTM model attained the greatest accuracy of 94%, with precision and recall approaching 95%, illustrating the effectiveness of multi-modal fusion. The results validate that domain-specific pre-processing and sensor fusion substantially improve early failure prediction, providing a potential foundation for proactive maintenance of network hardware through non-intrusive monitoring.
Read the full abstract ▾
Computer Vision
Multi-Branch Policy Optimization for Multimodal Large Language Models
Shuai Lyu, Yuning Gong, Ruiling Gao et al. (+7 more)
Group-based reinforcement learning methods for
multimodal large
language models typically rely on trajectory-level credit assignment that applies a single advantage to all
tokens in a response. However,
multimodal reasoning involves substantially higher perceptual uncertainty than text-only settings, where the model must repeatedly re-examine visual information to verify intermediate interpretations, and different visual groundings can lead to divergent
reasoning paths, making such uniform credit assignment particularly inadequate and causing relative advantages to progressively degenerate toward zero. To address these challenges, we propose Multi-Branch Policy Optimization (MBPO), a tree-based framework that constructs
reasoning trees at
vision-
language decision boundaries, enabling sibling branches to explore diverse visual hypotheses and assigning segment-level credit through branch-relative advantages. We further introduce a temporal replay buffer to reuse informative segments while controlling policy staleness. Experiments on several
multimodal reasoning benchmarks show that MBPO outperforms representative baselines, improving both learning signal quality and optimization efficiency. The code is publicly available at https://github.com/ShuaiLyu0110/MBPO.
Read the full abstract ▾
All authors Shuai Lyu, Yuning Gong, Ruiling Gao, Xiaoran Shang, Zhonghong Ou, Ping Zong, Yifan Zhu, Yuan Sun, Yang Qin, Peng Hu
Computer Vision · Cross-listed from cs.GR
GenGA: Editable and Data-Grounded Graphical Abstract Generation for Academic Papers
Takuro Kawada, Shunsuke Kitada and Hitoshi Iyatomi
Graphical Abstracts (GAs) visually summarize the key findings of academic papers, playing a crucial role in facilitating the understanding of research content. Recently, advancements in
vision-
language models and image generation models have enabled the automatic generation of scientific figures based on paper content. However, most conventional methods output the generated results as raster graphics, making post-editing (e.g., text modification and layout changes) highly difficult. This poses a significant challenge, as they are unsuitable for the iterative figure revision process inherent in paper writing and peer review. To tackle these challenges, we define the novel task of generating editable GAs from paper content and propose GenGA, a new GA generation framework that directly produces figures in vector format. By generating figures as a collection of vector elements with a hierarchical structure, GenGA produces outputs that can be seamlessly imported into existing drawing tools for intuitive, element-level editing. Furthermore, we introduce the Structural Independence Coefficient (SIC), a metric that quantifies the editing simplicity of a figure based on the degree to which local modifications propagate to other elements. Experimental results show that GenGA achieves superior editing simplicity compared to conventional methods, and even surpasses human-authored GAs in conciseness and semantic
alignment. We also validate SIC as an effective metric correlated with manual editing costs. This study fundamentally redefines GA generation as an editable vector graphic generation problem grounded in the practical workflows of researchers, significantly promoting effective scientific communication.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
A Foundational EDM2-Based Generative Model for High-Resolution Synthetic Fetal Ultrasound Imaging from Open Datasets
Harvey Mannering, Yilin Zhang, Ziao Liu et al. (+3 more)
Prenatal ultrasound imaging is key for assessing fetal health, but AI progress is limited by scarce,
privacy-restricted, and hard-to-annotate
datasets. We propose a high-resolution fetal ultrasound synthesis framework based on the EDM2
diffusion architecture, trained on multiple public
datasets to generate 512x512 images across six anatomical classes. Our method achieved improved image quality with lower FID scores and enhanced downstream fetal plane classification, reaching 93.36% ensemble accuracy after fine-tuning, surpassing real-data-only training. Clinical evaluation by an experienced fetal ultrasound specialist (10+ years) on 100 images yielded a mean realism score of 2.67/5, with real images rated higher than synthetic. Artefacts included smoothing, speckle irregularities, and anatomical inconsistencies. Code, data, models and other resources to reproduce this work are available at https://github.com/xfetus/fetal-ultrasound-edm2.
Read the full abstract ▾
All authors Harvey Mannering, Yilin Zhang, Ziao Liu, Zhiwu Huang, Jacqueline Matthew, Miguel Xochicale
Computer Vision
MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generation
Mohammadreza Hami, Mohammadreza Samadi, Chao Gao et al. (+1 more)
Pixel-space
diffusion models avoid the reconstruction ceiling of latent
diffusion models by generating directly in image space. However, their substantially higher
token count makes generation expensive due to the quadratic complexity of self-
attention. Several existing efficiency methods reduce this cost by using larger patches at selected denoising steps, thereby representing the image with fewer
tokens. Yet, each step still uses a single patch size uniformly across the entire image, overlooking that different regions suffer different fidelity losses when coarsened. We introduce MOSAIK, a damage-guided framework that varies patch size across regions and denoising steps. MOSAIK adapts the PixelDiT backbone to generate arbitrary heterogeneous patch layouts, and a lightweight predictor uses intermediate denoising features to estimate the fidelity loss caused by coarsening each region. Given a
token budget, our damage-guided layout predictor assigns fine patches to sensitive regions and coarse patches elsewhere. Remarkably, while reducing FLOPs by 70% and
token count by 83%, MOSAIK matches the full-compute PixelDiT on GenEval and its DPG-Bench score drops by only 1.0 point. Compared to diverse efficiency paradigms, including temporal patch scheduling and feature caching, our approach delivers highly competitive performance at moderate budgets and consistently outperforms these baselines in highly constrained compute regimes.
Read the full abstract ▾
All authors Mohammadreza Hami, Mohammadreza Samadi, Chao Gao, Negar Hassanpour
Computer Vision
Invisible Shortcuts: Why Vision Encoders Know Your Camera
Vladan Stojnić, Ryan Ramos, Giorgos Kordopatis-Zilos et al. (+2 more)
Deep
vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces
Read the full abstract ▾
All authors Vladan Stojnić, Ryan Ramos, Giorgos Kordopatis-Zilos, Noa Garcia, Giorgos Tolias
Computer Vision
Adapting Vision Foundation Models with Cascaded Semantics
Xi Xiao, Xingjian Li, Cheng Han et al. (+8 more)
Prompt tuning, a leading parameter-efficient adaptation paradigm in NLP, has recently been extended to computer
vision. Visual
prompt tuning (VPT) adapts pre-trained
vision transformers (ViTs) by updating a small set of additional
prompt parameters. However, existing visual
prompts are randomly initialized and do not exploit prior knowledge, such as instructions in NLP. We address this gap by injecting two complementary semantic priors into VPT. Fundamental image priors, including color, texture, and shape, are extracted with classical hand-crafted operators and injected into the input space, while self-
attention maps provide instance-aware semantics in the feature space. We further propose a cascaded scheme that integrates both priors throughout ViT adaptation. Experiments on 34 challenging image classification
datasets demonstrate superior downstream adaptation while tuning only 0.74% of ViT parameters. Project page: https://xixiaouab.github.io/Cascaded-Semantics/.
Read the full abstract ▾
All authors Xi Xiao, Xingjian Li, Cheng Han, Tianyang Wang, Lin Zhao, Yunbei Zhang, Guosheng Hu, Runmin Jiang, Xi Li, Xiao Wang, Min Xu
Computer Vision
Text-Guided Refinement of Multi-sequence Glioma Subregion Segmentation with a Vision-Language Foundation Model
Zach Eidex, Yu-nong Lin, Mojtaba Safari et al. (+4 more)
Background: Accurate glioma subregion delineation is important for radiotherapy planning and longitudinal monitoring, but manual contour correction is time-consuming. Models such as nnU-Net may generalize imperfectly and lack clinician-directed text correction. Purpose: We investigated adapting a three-dimensional (3D)
vision-
language foundation model for text-guided
brain tumor
segmentation refinement. Methods: We developed a lightweight VoxTell-based framework. Pretrained VoxTell generated initial masks. Oracle
prompts derived from
segmentation errors encoded target, action, location, imaging evidence, edit size, and preservation constraints. Frozen Qwen/VoxTell
prompt embeddings were injected through trainable projections into its multiscale decoder conditioning; other weights remained frozen. Training, validation, and testing used 901, 100, and 250 BraTS-GLI cases. Cross-
dataset transfer was evaluated on 100 meningioma, metastasis, pediatric tumor, and UPENN-GBM cases. Results: On the internal test set using post-contrast T1-weighted input, correct instructions improved subregion Dice similarity coefficient (DSC; enhancing tumor, edema, and necrotic/non-enhancing core) from $0.774\pm0.158$ to $0.796\pm0.137$. They outperformed blank
prompts ($0.762\pm0.155$; Holm-adjusted $p<0.001$, $d_z=0.71$) and contradictory
prompts ($0.770\pm0.163$; $p<0.001$, $d_z=0.48$). In cross-
dataset testing, correct instructions improved DSC from $0.527\pm0.287$ to $0.550\pm0.278$ and outperformed contradictory instructions ($0.504\pm0.275$; $p<0.001$, $d_z=0.43$). Conclusion: A 3D
vision-
language foundation model can perform instruction-guided refinement of glioma subregion
segmentations. Sensitivity to correct, blank, and contradictory
prompts suggests text-dependent contour editing rather than nonspecific post-processing, supporting further evaluation as a clinician-in-the-loop tool.
Read the full abstract ▾
All authors Zach Eidex, Yu-nong Lin, Mojtaba Safari, Sean Pitroda, Ralph Weichselbaum, Zhen Tian, Xiaofeng Yang
Computer Vision · Cross-listed from cs.RO
World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulation
Yuhao Pan, Haosong Peng, Zhengshen Zhang et al. (+8 more)
Vision-
language-action (VLA) models often treat main-view and wrist-view observations as parallel visual inputs, overlooking their distinct roles in
robot manipulation. Fine-grained manipulation, however, benefits from anticipating how wrist-local interactions may evolve under the global task context. To address this limitation, we present World-to-Wrist VLA (W2-VLA), a VLA model for fine-grained
robot manipulation with task-conditioned future wrist modeling. Given current multi-view observations and a task instruction, W2-VLA contextualizes a set of latent modeling
tokens as a compact interface between the
vision-
language model and the wrist predictor. Conditioned on this interface and the observed wrist history, the predictor forecasts future wrist latents, which are transformed into future-aware context for action prediction. In addition, we introduce W2-CoT, a synthesis pipeline that produces structured annotations describing manipulation progress, physical transition cues, and wrist-local evidence. These annotations provide auxiliary supervision that shapes the task-conditioned latent interface. Experiments on LIBERO, LIBERO-Plus, RoboTwin 2.0, and real-world manipulation tasks demonstrate improved fine-grained and contact-sensitive manipulation across single-arm and bimanual settings, while maintaining real-time action generation above $80$~Hz.
Read the full abstract ▾
All authors Yuhao Pan, Haosong Peng, Zhengshen Zhang, Zhengyang Yan, Yalun Dai, Fushuo Huo, Chujie Wang, Tianyu Qi, Xiucheng Wang, Nan Cheng, Wenchao Xu
Computer Vision
LoDA: A Level of Detection Aware Method and a Multimodal Sensing Benchmark for Object Level Change Detection
Haitian Wang, Xinyu Wang, Sheldon Fung et al. (+2 more)
High-definition 3D
LiDAR maps are important for autonomous driving and smart-city services, which require reliable detection of object-level changes in multi-temporal urban
LiDAR to keep digital maps aligned with the physical world. Existing approaches from raster height differencing to depth image and point-cloud networks often remain tile-based and threshold-driven, yielding per-point scores without explicit detection limits or consistent object-level labels. We propose an object-level 3D change-detection pipeline that integrates detection-limit-aware registration, geometry-driven object proxies with rule-based semantic and instance
segmentation, and displacement cues in height, volume, and surface-normal direction to assign five change labels with confidence. By decoupling registration, geometry, and semantics, the pipeline propagates pose uncertainty into spatially varying detection limits, stabilizes cross-epoch correspondences, and suppresses false changes caused by residual misalignment and density variation. We also present LoDA, a level-of-detection (LoD) aware
benchmark for the Subiaco district with fused multi-temporal vehicle-
LiDAR maps constructed with
LiDAR, GNSS, and IMU support, semantic instances, and object-level annotations. On this
benchmark, our method achieves 95.0% accuracy, 90.8% macro F1, and 83.0% macro IoU, exceeding the best baseline by 8.7 IoU points and 4.4 F1 points. On the public Urb3DCD-V2
benchmark evaluated under the official point-wise protocol, it reaches 96.81% mean accuracy and 89.52% mean change IoU, improving over the strongest reported baselines by 1.36 points in mAcc and 3.18 points in mIoUch.
Read the full abstract ▾
All authors Haitian Wang, Xinyu Wang, Sheldon Fung, Xian Zhang, Zichen Geng
Computer Vision
Positive-Unlabeled Preference Optimization For Chest X-ray Report Generation
Yuta Kobayashi, Pradyun Ramesh, Muhammad Ahmed Chaudhry et al. (+5 more)
Vision-
Language Models (VLMs) for radiology report generation are typically trained on retrospective clinical reports, which suffer from omission noise: clinically present findings are left unreported due to the omission of subtle findings. For example, prior studies show that cardiomegaly may be omitted from ICU chest X-ray reports when the imaging request is focused on monitoring support device placement. As a result, models trained with standard approaches inherit these omissions, learning to under-report findings themselves. We propose PU-DPO, a preference optimization framework to prevent omission noise from corrupting the preference signal. We reformulate the objective under a positive-unlabeled (PU) learning framework, treating absent mentions as unlabeled rather than truly negative. Our framework provides preference supervision using constructed contrastive pairs, generated using edits to model responses, producing variants that explicitly mention or omit a specific finding. Generated responses that mention the finding are naturally preferred in the context of visual evidence. Across semi-synthetic experiments and analyses on real-world chest radiograph
benchmarks where adjudicated labels are available, PU-DPO yields consistent gains in detection rates and recovery of hidden positives across multiple pathologies, and is more robust to omission noise than prior approaches.
Read the full abstract ▾
All authors Yuta Kobayashi, Pradyun Ramesh, Muhammad Ahmed Chaudhry, Vincent Jeanselme, Judy Wawira Gichoya, Sanmi Koyejo, Kathleen Capaccione, Shalmali Joshi
Computer Vision
Context Matters: Support Set Selection and Failure Detection for In-Context Medical Image Segmentation
Youssef Gehad, Emmanuel Zerefa, Krish Kabra et al. (+1 more)
In-context learning (ICL) adapts medical image
segmentation models to unseen structures and modalities without retraining by conditioning on a task-specific support set of image-mask exemplars. Because this support set is the model's only task-specific signal, its composition directly influences
segmentation performance. In this work, we investigate the support set as a controllable determinant of ICL reliability. First, we compare random sampling against similarity-based selection, where exemplars are retrieved based on their visual similarity to the query image. Second, we train a
transformer-based classifier to predict, from the query and support images alone, whether a
segmentation will fall below a specified Intersection-over-Union (IoU) threshold. Using MultiverSeg with DINOv3
embeddings across four
benchmarks and three imaging modalities, we show that similarity-based selection consistently matches or outperforms random sampling, with the largest gains at the smallest support set sizes. Furthermore, our classifier predicts
segmentation failure above chance on all four
benchmarks. Ultimately, these results demonstrate that the reliability of in-context
segmentation can be both improved via informed support selection and anticipated before use, providing practical mechanisms for safer clinical deployment.
Read the full abstract ▾
All authors Youssef Gehad, Emmanuel Zerefa, Krish Kabra, Guha Balakrishnan
Computer Vision · Cross-listed from cs.LG
Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction
Quinn Ledingham, Zhengsen Xu, Yimin Zhu et al. (+6 more)
Prediction of post-wildfire debris flows is critical for mitigating hazards to communities, infrastructure, and resources during intense rainfall in recently burned areas. However, identifying reliable machine learning models is complicated by overlapping debris-flow and non-debris-flow events in feature space, the need for model interpretability, and limited training data. This paper addresses these challenges through a systematic evaluation of machine learning models in terms of predictive performance, feature importance, and synthetic data augmentation. Using basin-scale observations of post-wildfire debris-flow events across the western United States, we compare 15 models, including the Tabular Prior-Data Fitted Network (TabPFN). Repeated stratified cross-validation shows that TabPFN achieves the highest unaugmented performance with a threat score of 0.637, closely followed by the best tree-based models. SHapley Additive exPlanations (SHAP) are used to identify the features driving predictions, revealing that short-duration rainfall intensity and storm accumulation consistently rank highest, while burn severity and terrain features contribute less. We further evaluate synthetic data augmentation using TabPFN-generated samples to address the scarcity of debris-flow observations. Synthetic augmentation improves the performance of all models except CNN, with the largest mean threat score increase of +0.041 among the deep learning models. By combining rigorous model benchmarking, interpretable feature analysis, and synthetic data augmentation, this work provides a comprehensive framework for improving post-wildfire debris-flow prediction.
Read the full abstract ▾
All authors Quinn Ledingham, Zhengsen Xu, Yimin Zhu, Zack Dewis, Mabel Heffring, Saeid Taleghanidoozdoozan, Motasem Alkayid, Megan Greenwood, Lincoln Linlin Xu
Computer Vision
A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval
Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz et al. (+2 more)
Contrastive
vision-
language models such as CLIP and BLIP are typically trained on short image captions, limiting their ability to retrieve images from detailed textual descriptions. While methods such as Long-CLIP extend the
token limit through positional
embedding interpolation, we ask a simpler question: does training text granularity alone determine long-text
retrieval performance? We present a systematic study of supervision ranging from single captions to multi-sentence paragraphs for contrastive image-text
retrieval. Using a synthetic pipeline based on Qwen2-VL and Llama 3.2
Vision, we generate diverse captions, hard negatives, and quality-scored paragraphs for 500K CC3M images. To isolate the effect of text granularity, we fine-tune only the BLIP text encoder while keeping the
vision encoder frozen across 10 training configurations. Our paragraph-supervised models match Long-CLIP-L on ShareGPT4V and outperform it by more than 14 points on DOCCI for image-to-text
retrieval, without architectural changes. We further show that paragraph supervision enables effective use of long
token sequences, whereas caption-only training degrades beyond 60
tokens. Increasing caption diversity improves short-caption
retrieval with diminishing returns, while paragraph supervision consistently benefits long-description
benchmarks and hard negatives prove detrimental in text-only fine-tuning. Evaluations on Flickr30k, COCO, ShareGPT4V, and DOCCI provide a comprehensive analysis of the trade-offs between text granularity,
retrieval direction, and description length.
Read the full abstract ▾
All authors Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz, Binshuai Wang, Peng Wei
Computer Vision
Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AI
Casey Wall, Longwei Wang, Rodrigue Rizk et al. (+1 more)
Gradient-weighted Class Activation Mapping (Grad-CAM) is widely used to visualize model decisions, but it was originally formulated for convolutional neural networks, where spatial feature maps and channel dimensions have clear architectural meanings.
Vision Transformers (ViTs) do not provide the same structure, instead representing images through
tokens,
attention, residual streams, and
multimodal interactions. This paper presents a systematic taxonomy and literature audit of how Grad-CAM and related methods are adapted, justified, and reported for ViT-based architectures. From an initial search of more than 550 papers, we identify 175 papers that apply Grad-CAM or Grad-CAM-adjacent methods to ViTs. We find that most papers do not provide a full mathematical or implementation-level account of how Grad-CAM is adapted to
transformer representations. To characterize this gap, we introduce a descriptive taxonomy of ViT Grad-CAM adaptations that makes explicit the feature locations, gradient targets, spatial reconstruction steps, and aggregation choices that are often left implicit. This taxonomy is not intended to prescribe a single correct adaptation, but to clarify the range of methodological choices being made. The study shows that Grad-CAM on ViTs is often treated as a trivial extension of CNN-based Grad-CAM, despite requiring nontrivial choices that affect rigor, reproducibility, and interpretation.
Read the full abstract ▾
All authors Casey Wall, Longwei Wang, Rodrigue Rizk, KC Santosh
Computer Vision · Cross-listed from cs.AI
WorldClaw: Agentic 3D Open-World Generation at Scale
Chunchao Guo, Jinpeng Li, Yang Li et al. (+1 more)
Generating large-scale, freely explorable 3D worlds from open-ended text remains challenging because a system must jointly maintain global spatial coherence, rich local content, and explicit assets suitable for downstream editing and reuse. We present WorldClaw, a fully agentic, coarse-to-fine framework for open-world 3D scene generation. Planning
agents translate a text
prompt into a structured specification of regions, terrain, assets, materials, and spatial relations. WorldClaw then builds a globally coherent terrain foundation from semantic layouts, reusable assets, generative or procedural materials, and a region-aware height field. For detail-demanding regions, it generates terrain-conditioned compositions, reconstructs editable textured meshes, and recovers their placement on the terrain; render-based
agents further refine terrain, objects, appearance, and contacts. Across diverse open-world
prompts, WorldClaw produces large-scale scenes with coherent spatial organization, visually compelling local content, and editable instance-level assets while preserving a consistent global terrain structure.
Read the full abstract ▾
All authors Chunchao Guo, Jinpeng Li, Yang Li, Zilong Huang
Computer Vision · Cross-listed from cs.LG
Disentangling 3D Modeling from Spatial Reasoning
Haoze Sun, Jiequan Cui, Qingshan Xu et al. (+1 more)
In this work, we explore an alternative paradigm for spatial
reasoning by explicitly disentangling 3D perception from
reasoning, rather than jointly acquiring implicit 3D perception and
reasoning through large-scale training. Our key observation is that modern perception models excel at estimating continuous 3D geometry, whereas large
language models (LLMs) are particularly effective at compositional and symbolic
reasoning. Motivated by these complementary strengths, we propose the Disentangled Spatial Reasoner (DiSR), a simple yet effective framework that reconstructs the physical world into structured 3D evidence using off-the-shelf expert perception models and fine-tunes an LLM with LoRA to perform
reasoning solely over this explicit
geometric evidence. Without large-scale 3D VQA training or complex tool-use policies, DiSR achieves competitive performance on popular spatial
reasoning benchmarks. Beyond its strong performance, DiSR offers improved interpretability, modularity, and computational efficiency, demonstrating that explicit separation of perception and
reasoning is a scalable and effective alternative paradigm to end-to-end modeling for spatial intelligence.
Read the full abstract ▾
All authors Haoze Sun, Jiequan Cui, Qingshan Xu, Richang Hong
Computer Vision
In-Context Forcing: Uncovering Context Effects in Autoregressive Video Diffusion
Lingxiao Yang, Liu Liu, Moran Li et al. (+4 more)
Current few-step autoregressive
video diffusion models depend on previous fully denoised clean frames as context for all denoising steps of the current frame. However, these clean frames leak excessive local details, which causes the model to take shortcuts, resulting in compromised temporal semantics and dynamics. Inspired by the perspective of
diffusion as masking, we explore the impact of noisy contexts on few-step autoregressive generation. Yet, simply applying contexts with the same noise levels provides insufficient guidance, leading to poor temporal consistency. To resolve this dilemma, we introduce In-Context Forcing, a progressive autoregressive paradigm that utilizes contexts with decreasing noise levels. By applying less masking to distant frames and more masking to adjacent ones, this approach provides adaptive guidance, effectively ensuring both robust temporal consistency and high inter-frame dynamics. Furthermore, by decoupling the strict dependence on previous clean frames, our paradigm enables cross-frame parallel denoising, achieving substantial inference acceleration without sacrificing performance. Extensive experiments on VBench demonstrate that our method significantly outperforms state-of-the-art approaches in both visual fidelity and inference speed.
Read the full abstract ▾
All authors Lingxiao Yang, Liu Liu, Moran Li, Han Feng, Wenjian Cao, Jiangning Zhang, Ye Shi
Computer Vision · Cross-listed from cs.AI
Coherence-Oriented Dream Scene Visualisation
Azra Aç{\i}l and Simon Colton
Dreams can be emotionally intense but difficult to communicate. We describe the Dream Scene Visualiser (DSV) system which turns written dream descriptions into a temporal sequence of four panel images visualising the dream. This starts with a large
language model prompted to split a dream description into four chronological parts. Then a text-to-image model produces images for each part with visual coherence maintained across the sequence, and DSV regenerates any image not suitably matching the text. We evaluate DSV over 50 visualisations from dream descriptions in DreamBank, and report quality, fidelity and coherence results via objective measures employing the CLIP, DINOv2 and Qwen2-VL
vision-
language models.
Read the full abstract ▾
Computer Vision
NeuroAdaptTrainer: A Fiji/ImageJ Plugin for YOLO-Based Neuron Segmentation, InteractiveCorrection and Transfer Learning
Daniela Eraso-Casas, Gerard Villarroya-Pique, Esther Serrano-Pertierra et al. (+4 more)
Neuron counting and
segmentation in microscopy images of neuronal cultures is a routine and time-consuming task in neuroscience research, traditionally performed through manual inspection or semi-automatic tools. We present NeuroAdaptTrainer, an open-source Fiji/ImageJ plugin that integrates a YOLO instance-
segmentation model directly into the microscopist's workflow. The plugin allows a user to run automatic
neuron detection on a single image or a batch of images, manually correct the resulting detections from within Fiji, and use those corrections to adapt the model to new imaging conditions via transfer learning. A built-in external validation module allows the base and adapted models to be compared quantitatively on a held-out annotated set. NeuroAdaptTrainer lowers the barrier for non-specialist users to benefit from deep-learning-based
segmentation while keeping expert supervision at the center of the workflow.
Read the full abstract ▾
All authors Daniela Eraso-Casas, Gerard Villarroya-Pique, Esther Serrano-Pertierra, M. Teresa Fernández-Sánchez, Antonello Novellie, Angel Rio-Alvarez, Víctor M. González
Computer Vision · Cross-listed from cs.CR
A Survey of Adversarial Efficiency Degradation for Vision Transformer by Exploiting Input-adaptive Optimization
Anadi Goyal, Nandish Chattopadhyay, Anupam Chattopadhyay et al. (+1 more)
Vision Transformers (ViTs) increasingly rely on input-adaptive inference, such as
token pruning and early halting, to meet energy and latency budgets. This survey examines a recent class of
adversarial efficiency degradation attacks that target these mechanisms to increase computation without necessarily degrading accuracy. We unify and compare two representative attacks, SlowFormer (a universal
adversarial patch) and DeSparsify (per-image perturbations), across three popular
token-
pruning frameworks: A-ViT, ATS, and AdaViT. We standardize reporting using GFLOPs, accuracy loss, and an Attack Success (AS) metric that measures how much of the model's compute savings the attack takes away. Understanding these attacks is crucial for designing countermeasures that not only mitigate risk but also remain lightweight, since deployment often occurs in low-power settings such as mobile or embedded devices. To organize our analysis, we focus on three questions: how input-adaptive optimizations (e.g.,
token pruning and early halting) create attack surfaces for efficiency degradation; how such attacks operate in practice and which optimizations are most vulnerable; and which defenses exist today and whether they meaningfully restore efficiency under attack.
Read the full abstract ▾
All authors Anadi Goyal, Nandish Chattopadhyay, Anupam Chattopadhyay, Chandan Karfa
Computer Vision · Cross-listed from cs.RO
VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances
Jihoon Oh, Kento Kawaharazuka and Kei Okada
Learning manipulation skills from human
videos is promising for scalable
robot learning. However, the embodiment mismatch between humans and
robots makes this challenging. One promising solution is to learn object-centric actionable affordances that are embodiment-agnostic. In this work, we propose a framework that leverages
egocentric human
videos with state-of-the-art 3D Structure-from-Motion and hand mesh reconstruction to extract actionable affordances such as visual, grasp, and trajectory affordances that explicitly encode where to interact, how to grasp, and how to move. We construct EgoAffordance, a large-scale
dataset comprising 204K episodes with 5.6M visual affordances and 11.6M grasp and trajectory affordances. Building on this, we introduce VLAff, a large
vision-
language model-based unified foundation model that learns cross-modal correlations across all actionable affordances. Given a visual observation and instruction, VLAff generates visual affordance heatmaps, grasp poses, and trajectories, which are then converted into directly executable actions by utilizing 3D scene information. Through extensive experiments, we demonstrate that VLAff not only achieves state-of-the-art performance on visual affordance prediction, but can also be effectively applied to real
robot applications such as zero-shot manipulation and affordance-guided
robot learning.
Read the full abstract ▾
Computer Vision
StyleComposer: Training-Free Multi-Reference Style Composition
Sanghyeok Lee, Jihye Kang and Namhyuk Ahn
The style of a painting is not monolithic: color, texture, and structure may come from different sources. Existing reference-guided methods transfer them as one style signal, leaving each attribute's source and strength outside the user's control. We ask where in a
diffusion model one attribute can change while the others hold, and find that no single representation isolates all three. The proposed StyleComposer therefore routes each style attribute through the representation where it separates best and coordinates the routes over denoising time. Without training or inversion, it satisfies three references and the
prompt jointly more closely than prior methods, and exposes one strength slider per attribute. Project page: https://lexxsh.github.io/StyleComposer
Read the full abstract ▾
Computer Vision
Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generation
Ye Leng, Junjie Chu, Yiting Qu et al. (+3 more)
Picture books and comics have long been used to disseminate hateful narratives because they are easily understood even by children, as exemplified by the notorious Nazi propaganda picture book Der Giftpilz. Recently, frontier text-to-image (T2I) systems such as Gemini and GPT-Image have enabled conversational generation with consistent characters and scenes across turns, making hateful visual stories, namely ordered image groups that collectively convey hateful narratives, cheap and scalable to produce. Although prior work has studied hateful content generation by T2I systems, it focuses on individual images, leaving group-level hateful meaning largely unexplored. We aim to address the gap. Concretely, we introduce HatefulStoryPrompts, comprising 330 multi-turn configurations from 55 hateful stories across two
languages and three visual styles, and evaluate five frontier models over 4,950 attempts. Every model completes over 80% of the stories, with the strongest reaching 99.0%. We further evaluate existing moderation systems on HatefulVisualStory, a human-labeled
dataset of 969 hateful image sets and 990 benign controls, and find that they frequently miss group-level hateful meaning: dedicated safety models achieve at most 34.9% recall, while a strong
vision-
language model reaches 67.5%. Finally, we propose complementary proactive and post-generation defenses. An interaction-aware monitor achieves 97.3% recall for
prompt-only sessions and 92.6% when the user supplies the first image, while post-generation methods jointly analyzing completed image groups reach 80.2%. Our work shows that, as image generation evolves from isolated outputs to coherent visual narratives, safety must evolve accordingly, from per-image moderation to stateful
reasoning over interactions and image relationships.
Read the full abstract ▾
All authors Ye Leng, Junjie Chu, Yiting Qu, Mingjie Li, Yun Shen, Yang Zhang
Computer Vision
MapTCL: Temporal Consistency Learning via Bidirectional Alignment for Vectorized HD Map Construction
Hyeonseo Kim, Juyeb Shin, Hyeonjun Jeong et al. (+2 more)
Constructing reliable online HD maps remains challenging in dynamic urban environments due to moving objects and occlusions. While recent works employ feature-level temporal fusion to address this, they rely solely on per-frame ground truth supervision. Consequently, they lack an explicit objective to directly penalize the
geometric noise and temporal jitter between consecutive online HD maps. To address this, we propose MapTCL, an auxiliary training strategy that formulates temporal consistency loss between current and past frames via bidirectional
alignment. Specifically, Bidirectional Vector Consistency Learning (BVCL) models the
geometric and semantic discrepancies between associated past and current vector instances as an auxiliary loss. We also employ Raster map Consistency Learning (RCL) as an additional loss to stabilize dense BEV features. By jointly training with these dual losses, MapTCL improves the temporal stability of generated HD maps. Extensive experiments on two standard
benchmarks demonstrate the effectiveness of our approach. As a versatile plug-and-play module, MapTCL consistently enhances existing baseline models, achieving gains of +3.7 mAP & +2.8 C-mAP on nuScenes and +3.1 mAP & +2.5 C-mAP on Argoverse 2 without additional inference overhead.
Read the full abstract ▾
All authors Hyeonseo Kim, Juyeb Shin, Hyeonjun Jeong, Hiwon Shin, Dongsuk Kum
Computer Vision
CoCo-IR: Contextual Composed Image Retrieval
Shengcao Cao, Tanmaya Shekhar Dabral, Zhongli Ding et al. (+6 more)
Current instruction-based image
retrieval systems are powerful but limited to single-turn interactions, failing to capture the iterative nature of complex, real-world visual searches. To overcome this limitation, we introduce Contextual Composed Image
Retrieval (CoCo-IR), a novel task that enables users to progressively refine search results through interactions. We address this new task by proposing a new model based on a Large
Multimodal Model (LMM) that functions as a context-aware reasoner for CoCo-IR. Our model interprets the entire interaction history to generate Transformable Image
Embeddings (TIE) that evolve across turns. To fuel the model training without expensive human annotations, we develop a fully autonomous, scalable data engine that leverages LMMs to generate high-quality contextual
retrieval data, and uses model-guided verification to mine challenging hard negatives. Extensive experiments demonstrate that our approach establishes new state-of-the-art performance: We achieve 39.4 mAP@5 on the challenging single-turn
benchmark CIRCO; furthermore, on our new CoCo-IR
benchmark, our model maintains robust performance with 44.1 R@1 on 4-turn dialogues, dramatically outperforming existing methods (28.2 4-turn R@1) that fail to handle multi-turn context. Project page: https://CoCo-IR.github.io.
Read the full abstract ▾
All authors Shengcao Cao, Tanmaya Shekhar Dabral, Zhongli Ding, Madhuri Shanbhogue, Kaifeng Chen, Zhe Li, Mojtaba Seyedhosseini, Yu-Xiong Wang, Liang-Yan Gui
Computer Vision
Objects as Audio-Visual Modal Sound Fields
Zisen Shao, Zihao Wei, Derong Jin et al. (+1 more)
While modern 3D reconstruction excels at modeling object geometry and appearance, it largely ignores the rich acoustic cues revealed through physical interaction. Object impact sounds convey material, stiffness, and structural properties that complement
vision, yet existing impact sound modeling approaches either rely on expensive physics-based simulation or require large
datasets to generalize in a purely data-driven manner. We introduce Audio-Visual Modal Sound Field (AV-MSF), a novel object-level acoustic representation reconstructed from multi-view images and only a few impact sound recordings. AV-MSF builds on 3D Gaussian Splatting integrated with dense 3D visual feature to provide a strong geometry-aware prior, and represents the impact sound field using compact, physically meaningful modal parameters, enabling robust few-shot reconstruction. Experiments on two real-world
datasets show that AV-MSF achieves state-of-the-art impact sound rendering, outperforming both physics-based and data-driven baselines. Furthermore, we demonstrate downstream applications enabled by our representation, including contact localization and object sound editing.
Read the full abstract ▾
All authors Zisen Shao, Zihao Wei, Derong Jin, Ruohan Gao