Computer Vision · Cross-listed from cs.CL
The Influence of Iconicity in Transfer Learning for Sign Language Recognition
Keren Artiaga, Conor Lynch, Haithem Afli et al. (+1 more)
Most sign
language recognition research relies on Transfer Learning (TL) from
vision-based
datasets such as ImageNet. Some extend this to alternatively available
language datasets, often focusing on signs with cross-linguistic similarities. This body of work examines the necessity of these likenesses on effective knowledge transfer by comparing TL performance between iconic signs of two different sign
language pairs: Chinese to Arabic and Greek to Flemish. Google Mediapipe was utilised as an input feature extractor, enabling spatial information of these signs to be processed with a Multilayer Perceptron architecture and the temporal information with a Gated Recurrent Unit. Experimental results showed a 7.02% improvement for Arabic and 1.07% for Flemish when conducting iconic TL from Chinese and Greek respectively.
Read the full abstract ▾
All authors Keren Artiaga, Conor Lynch, Haithem Afli, Mohammed Hasanuzzaman
Computer Vision
Fine-Tuning a Large Vision-Language Model for Artwork's Scoring and Critique
Zhehan Zhang, Meihua Qian, Li Luo et al. (+4 more)
Assessing artistic creativity is foundational to creativity research and arts education, yet manual scoring (e.g., Torrance Tests of Creative Thinking) is labor-intensive at scale. Prior machine-learning approaches show promise for visual creativity scoring, but many rely mainly on image features and provide limited or no explanatory feedback. We propose a framework for automated creativity assessment of human paintings by fine-tuning the
vision-
language model Qwen2-VL-7B with multi-task learning. Our
dataset contains 1000 human-created paintings scored on a 1-100 scale and paired with a short human-written description (content or artist explanation). Two expert raters evaluated each work using a five-dimension rubric (originality, color, texture, composition, content) and provided written critiques; we use an 80/20 train-test split. We add a lightweight regression head on the visual encoder output so the model can predict a numerical score and generate rubric-aligned feedback in a single forward pass. By
embedding the structured rubric and the artwork description in the system
prompt, we constrain the generated text to match the quantitative prediction. Experiments show strong accuracy, achieving Pearson r > 0.97 and MAE about 3.95 on the 100-point scale. Qualitative evaluation indicates the generated feedback is semantically close to expert critiques (average SBERT cosine similarity = 0.798). The proposed approach bridges computer
vision and art assessment and offers a scalable tool for creativity research and classroom feedback.
Read the full abstract ▾
All authors Zhehan Zhang, Meihua Qian, Li Luo, Siyu Huang, Chaoyi Zhou, Ripon Saha, Xinxin Song
Computer Vision
WildfireVLM: AI-powered Analysis for Early Wildfire Detection and Risk Assessment Using Satellite Imagery
Aydin Ayanzadeh, Prakhar Dixit, Sadia Kamal et al. (+1 more)
Wildfires are a growing threat to ecosystems, human lives, and infrastructure, with their frequency and intensity rising due to
climate change and human activities. Early detection is critical, yet satellite-based monitoring remains challenging due to faint smoke signals, dynamic weather conditions, and the need for real-time analysis over large areas. We introduce WildfireVLM, an AI framework that combines satellite imagery wildfire detection with
language-driven risk assessment. We construct a labeled wildfire and smoke
dataset using imagery from Landsat-8/9, GOES-16, and other publicly available Earth observation sources, including harmonized products with aligned spectral bands. WildfireVLM employs YOLOv12 to detect fire zones and smoke plumes, leveraging its ability to detect small, complex patterns in satellite imagery. We integrate
Multimodal Large
Language Models (MLLMs) that convert detection outputs into contextualized risk assessments and prioritized response recommendations for disaster management. We validate the quality of risk
reasoning using an LLM-as-judge evaluation with a shared rubric. The system is deployed using a service-oriented architecture that supports real-time processing, visual risk dashboards, and long-term wildfire tracking, demonstrating the value of combining computer
vision with
language-based
reasoning for scalable wildfire monitoring. The code and
dataset are publicly available on GitHub at https://github.com/Ayanzadeh93/_WildfireVLM_.
Read the full abstract ▾
All authors Aydin Ayanzadeh, Prakhar Dixit, Sadia Kamal, Milton Halem
Computer Vision
PCReg-Net: Progressive Contrast-Guided Registration for Cross-Domain Image Alignment
Jiahao Qin
Deformable image registration across heterogeneous domains remains challenging because coupled appearance variation and
geometric misalignment violate the brightness constancy assumption underlying conventional methods. We propose PCReg-Net, a progressive contrast-guided registration framework that performs coarse-to-fine
alignment through four lightweight modules: (1)~a registration U-Net for initial coarse
alignment, (2)~a reference feature extractor capturing multi-scale structural cues from the fixed image, (3)~a multi-scale contrast module that identifies residual misalignment by comparing coarse-registered and reference features, and (4)~a refinement U-Net with feature injection that produces the final high-fidelity output. We evaluate on the FIRE-Reg-256 retinal fundus
benchmark, demonstrating improvements over both traditional and deep learning baselines. Additional experiments on two microscopy
benchmarks further confirm cross-domain applicability. With only 2.56M parameters, PCReg-Net achieves real-time inference at 141 FPS. Code is available at https://github.com/JiahaoQin/PCReg-Net.
Read the full abstract ▾
Computer Vision
Spectral Collapse in Diffusion Inversion
Nicolas Bourriez, Alexandre Verine and Auguste Genovesio
Conditional
diffusion inversion provides a powerful framework for unpaired image-to-image translation. However, we demonstrate through an extensive analysis that standard deterministic inversion (e.g. DDIM) fails when the source domain is spectrally sparse compared to the target domain (e.g., super-resolution, sketch-to-image). In these contexts, the recovered latent from the input does not follow the expected isotropic Gaussian distribution. Instead it exhibits a signal with lower frequencies, locking target sampling to oversmoothed and texture-poor generations. We term this phenomenon spectral collapse. We observe that
stochastic alternatives attempting to restore the noise variance tend to break the semantic link to the input, leading to structural drift. To resolve this structure-texture trade-off, we propose Orthogonal Variance Guidance (OVG), an inference-time method that corrects the ODE dynamics to enforce the theoretical Gaussian noise magnitude within the null-space of the structural gradient. Extensive experiments on microscopy super-resolution (BBBC021) and sketch-to-image (Edges2Shoes) demonstrate that OVG effectively restores photorealistic textures while preserving structural fidelity.
Read the full abstract ▾
Computer Vision
DriveMamba: Task-Centric Scalable State Space Model for Efficient End-to-End Autonomous Driving
Haisheng Su, Wei Wu, Feixiang Song et al. (+3 more)
Recent advances towards End-to-End Autonomous Driving (E2E-AD) have been often devoted on integrating modular designs into a unified framework for joint optimization e.g. UniAD, which follow a sequential paradigm (i.e., perception-prediction-planning) based on separable
Transformer decoders and rely on dense BEV features to encode scene representations. However, such manual ordering design can inevitably cause information loss and cumulative errors, lacking flexible and diverse relation modeling among different modules and sensors. Meanwhile, insufficient training of image backbone and quadratic-complexity of
attention mechanism also hinder the scalability and efficiency of E2E-AD system to handle spatiotemporal input. To this end, we propose DriveMamba, a Task-Centric Scalable paradigm for efficient E2E-AD, which integrates dynamic task relation modeling, implicit view correspondence learning and long-term temporal fusion into a single-stage Unified Mamba decoder. Specifically, both extracted image features and expected task outputs are converted into
token-level sparse representations in advance, which are then sorted by their instantiated positions in 3D space. The linear-complexity operator enables efficient long-context sequential
token modeling to capture task-related inter-dependencies simultaneously. Additionally, a bidirectional trajectory-guided "local-to-global" scan method is designed to preserve spatial locality from ego-perspective, thus facilitating the ego-planning. Extensive experiments conducted on nuScenes and Bench2Drive
datasets demonstrate the superiority, generalizability and great efficiency of DriveMamba.
Read the full abstract ▾
All authors Haisheng Su, Wei Wu, Feixiang Song, Junjie Zhang, Zhenjie Yang, Junchi Yan
Computer Vision
KidMesh: Computational Mesh Reconstruction for Pediatric Congenital Hydronephrosis Using Deep Neural Networks
Haoran Sun, Zhanpeng Zhu, Anguo Zhang et al. (+7 more)
Pediatric congenital hydronephrosis (CH) is a common urinary tract disorder, primarily caused by obstruction at the renal pelvis-ureter junction.
Magnetic resonance urography (MRU) can visualize hydronephrosis, including renal pelvis and calyces, by utilizing the natural contrast provided by water. Existing voxel-based
segmentation approaches can extract CH regions from MRU, facilitating disease diagnosis and prognosis. However, these
segmentation methods predominantly focus on morphological features, such as size, shape, and structure. To enable functional assessments, such as urodynamic simulations, external complex post-processing steps are required to convert these results into mesh-level representations. To address this limitation, we propose an end-to-end method based on deep neural networks, namely KidMesh, which could automatically reconstruct CH meshes directly from MRU. Generally, KidMesh extracts feature maps from MRU images and converts them into feature vertices through grid sampling. It then deforms a template mesh according to these feature vertices to generate the specific CH meshes of MRU images. Meanwhile, we develop a novel schema to train KidMesh without relying on accurate mesh-level annotations, which are difficult to obtain due to the sparsely sampled MRU slices. Experimental results show that KidMesh could reconstruct CH meshes in an average of 0.4 seconds, and achieve comparable performance to conventional methods without requiring post-processing. The reconstructed meshes exhibited no self-intersections, with only 3.7% and 0.2% of the vertices having error distances exceeding 3.2mm and 6.4mm, respectively. After rasterization, these meshes achieved a Dice score of 0.86 against manually delineated CH masks. Furthermore, these meshes could be used in renal urine flow simulations, providing valuable urodynamic information for clinical practice.
Read the full abstract ▾
All authors Haoran Sun, Zhanpeng Zhu, Anguo Zhang, Bo Liu, Zhaohua Lin, Liqin Huang, Mingjing Yang, Lei Liu, Shan Lin, Wangbin Ding
Computer Vision
The Effective Depth Paradox: Topology and Trainability in Deep CNNs
Manfred M. Fischer and Joshua Pitts
This paper presents a controlled comparative study of convolutional neural network (CNN)
topology and image classification performance across the architectural families VGG, ResNet, and GoogLeNet, evaluated on CIFAR-10 under a unified training protocol. We formalize the distinction between nominal depth ($D_{\mathrm{nom}}$), the physical count of weight-bearing layers, and effective depth ($D_{\mathrm{eff}}$), an operational metric quantifying the expected length of forward information paths, extending the path-ensemble interpretation of residual networks introduced by Veit et al. (2016) into closed-form, pre-training proxies spanning sequential, residual, and multi-branch
topologies. We validate this proxy against a gradient-weighted variant computed from observed backpropagation signal. Across eight representative models (VGG-11/13/16/19, ResNet-18/34/50, GoogLeNet), plain VGG-style stacks show early accuracy saturation as $D_{\mathrm{eff}}$ increases, whereas ResNet and GoogLeNet continue to benefit from added depth by keeping $D_{\mathrm{eff}}$ low relative to $D_{\mathrm{nom}}$ - a pattern we term the "Effective Depth Paradox". A pooled correlation analysis shows both $D_{\mathrm{nom}}$ and $D_{\mathrm{eff}}$ are strongly, significantly associated with accuracy (r = 0.94 and r = 0.93; both p < 0.01); given the small family-clustered sample, this alone cannot cleanly separate the two metrics, so we treat gradient-norm evidence as complementary mechanistic support rather than decisive statistical proof. We conclude that architectural
topology, not layer count alone, governs trainability and scaling efficiency in deep CNNs. All claims are scoped to CIFAR-10-scale training of the three families studied; we do not claim validation at ImageNet scale or generalization to modern architectures such as EfficientNet, ConvNeXt, or
Vision Transformers, which we identify as necessary future work.
Read the full abstract ▾
Computer Vision
Conditional Generative Models for High-Resolution Range Profiles: Capturing Geometry-Driven Trends in a Large-Scale Maritime Dataset
Edwyn Brient, Santiago Velasco-Forero and Rami Kassab
High-resolution range profiles (HRRPs) enable fast onboard processing for radar automatic target recognition, but their strong sensitivity to acquisition conditions limits robustness across operational scenarios. Conditional HRRP generation can mitigate this issue, yet prior studies are constrained by small, highly specific
datasets. We study HRRP synthesis on a largescale maritime database representative of coastal surveillance variability. Our analysis indicates that the fundamental scenario drivers are
geometric: ship dimensions and the desired aspect angle. Conditioning on these variables, we train generative models and show that the synthesized signatures reproduce the expected line-of-sight
geometric trend observed in real data. These results highlight the central role of acquisition geometry for robust HRRP generation.
Read the full abstract ▾
Computer Vision
MFN Decomposition and Related Metrics for High-Resolution Range Profiles Generative Models
Edwyn Brient, Santiago Velasco-Forero and Rami Kassab
High-resolution range profile (HRRP ) data are in vogue in radar automatic target recognition (RATR). With the interest in classifying models using HRRP, filling gaps in
datasets using generative models has recently received promising contributions. Evaluating generated data is a challenging topic, even for explicit data like face images. However, the evaluation methods used in the state-ofthe-art of HRRP generation rely on classification models. Such models, called ”black-box”, do not allow either explainability on generated data or multi-level evaluation. This work focuses on decomposing HRRP data into three components: the mask, the features, and the noise. Using this decomposition, we propose two metrics based on the physical interpretation of those data. We take profit from an expensive
dataset to evaluate our metrics on a challenging task and demonstrate the discriminative ability of those.
Read the full abstract ▾
Computer Vision
VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction
Jiarong Liang, Max Ku, Ka-Hei Hui et al. (+2 more)
Evaluating whether
Multimodal Large
Language Models (MLLMs) genuinely reason about physical dynamics remains challenging. Most existing
benchmarks rely on recognition-style protocols such as Visual Question Answering (VQA) and Violation of Expectation (VoE), which can often be answered without committing to an explicit, testable physical hypothesis. We propose VisPhyWorld, an execution-based framework that evaluates physical
reasoning by requiring models to generate executable simulator code from visual observations. By producing runnable code, the inferred world representation is directly inspectable, editable, and falsifiable. This separates physical
reasoning from rendering. Building on this framework, we introduce VisPhyBench, comprising 209 evaluation scenes derived from 108 physical templates and a systematic protocol that evaluates how well models reconstruct appearance and reproduce physically plausible motion. Our pipeline produces valid reconstructed
videos in 97.7% of
benchmark runs before fallback. Experiments show that while state-of-the-art MLLMs achieve strong semantic scene understanding, they struggle to accurately infer physical parameters and to simulate consistent physical dynamics. Our code is available https://github.com/TIGER-AI-Lab/VisPhyWorld
Read the full abstract ▾
All authors Jiarong Liang, Max Ku, Ka-Hei Hui, Ping Nie, Wenhu Chen
Computer Vision
NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Driving
Xiaoxu Peng, Dong Zhou, Jianwen Zhang et al. (+3 more)
Vision Language Models (VLMs) have advanced perception in autonomous driving (AD), but they remain vulnerable to
adversarial threats. These risks range from localized physical patches to imperceptible global perturbations. Existing defense methods for VLMs remain limited and often fail to reconcile robustness with clean-sample performance. To bridge these gaps, we propose NutVLM, a comprehensive self-adaptive defense framework designed to secure the entire perception-decision lifecycle. Specifically, we first employ NutNet++ as a sentinel, which is a unified detection-purification mechanism. It identifies benign samples, local patches, and global perturbations through three-way classification. Subsequently, localized threats are purified via efficient grayscale masking, while global perturbations trigger Expert-guided
Adversarial Prompt Tuning (EAPT). Instead of the costly parameter updates of full-model fine-tuning, EAPT generates "corrective driving
prompts" via gradient-based latent optimization and discrete projection. These
prompts refocus the VLM's
attention without requiring exhaustive full-model retraining. Evaluated on the Dolphins
benchmark, our NutVLM yields a 4.89% improvement in overall metrics (e.g., Accuracy,
Language Score, and GPT Score). These results validate NutVLM as a scalable security solution for intelligent transportation. Our code is available at https://github.com/PXX/NutVLM.
Read the full abstract ▾
All authors Xiaoxu Peng, Dong Zhou, Jianwen Zhang, Guanghui Sun, Anh Tu Ngo, Anupam Chattopadhyay
Computer Vision · Cross-listed from eess.IV
Data-driven registration and modeling of brain deformation for image-guided neurosurgery
Tiago Assis, Colin P. Galvin, Joshua P. Castillo et al. (+12 more)
Accurate compensation of
brain deformation is critical for reliable image-guided neurosurgery.
Surgical manipulation and tumor resection induce tissue motion, causing preoperative planning images to become misaligned with the intraoperative anatomy. In this review, we examine data-driven methods developed between 2020 and 2025 for
brain deformation registration and modeling, with a particular focus on learning-based approaches. A comprehensive literature search was conducted in PubMed, IEEE Xplore, Scopus, and Web of Science using predefined inclusion and exclusion criteria for computational methods addressing
brain deformation in neurosurgical imaging, resulting in 46 eligible studies. We provide a unified analysis of methodological strategies, including deep learning-based image registration, direct deformation field regression, synthesis-driven
multimodal alignment, resection-aware architectures for handling missing correspondences, and hybrid models integrating biomechanical priors. We also examine
dataset utilization, evaluation metrics, validation protocols, and the assessment of uncertainty and generalization across studies. While learning-based methods demonstrate promising accuracy and computational efficiency, current approaches remain limited by out-of-distribution robustness, standardized benchmarking, interpretability, and readiness for clinical deployment. Our review highlights these gaps and outlines future directions toward more robust, generalizable, and clinically translatable solutions for neurosurgical guidance. By organizing recent advances and critically assessing evaluation practices, this work provides a comprehensive reference for researchers and clinicians working on data-driven registration and modeling of
brain deformation.
Read the full abstract ▾
All authors Tiago Assis, Colin P. Galvin, Joshua P. Castillo, Nazim Haouchine, Marta Kersten-Oertel, Zeyu Gao, Mireia Crispin-Ortuzar, Stephen J. Price, Thomas Santarius, Yangming Ou, Sarah Frisken, Nuno C. Garcia, Alexandra J. Golby, Reuben Dorent, Ines P. Machado
Computer Vision
VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understanding
Rongcan Pei, Huan Li, Fang Guo et al. (+1 more)
While
Vision-
Language Models (VLMs) have shown promise in textual understanding, they face significant challenges when handling long context and complex
reasoning tasks. In this paper, we dissect the internal mechanisms governing long-context processing in VLMs to understand their performance bottlenecks. Through the lens of
attention analysis, we identify specific Visual Evidence
Retrieval (VER) Heads - a sparse, dynamic set of
attention heads critical for locating visual cues during
reasoning, distinct from static OCR heads. We demonstrate that these heads are causal to model performance; masking them leads to significant degradation. Leveraging this discovery, we propose VERA (Visual Evidence
Retrieval Augmentation), a training-free framework that detects model uncertainty (i.e.,
entropy) to trigger the explicit verbalization of visual evidence attended by VER heads. Comprehensive experiments demonstrate that VERA significantly improves long-context understanding of open-source VLMs: it yields an average relative improvement of 21.3% on Qwen3-VL-8B-Instruct and 20.1% on GLM-4.1V-Thinking across five
benchmarks.
Read the full abstract ▾
All authors Rongcan Pei, Huan Li, Fang Guo, Qi Zhu
Computer Vision
MPA: Multimodal Prototype Augmentation for Few-Shot Learning
Liwen Wu, Wei Wang, Lei Zhao et al. (+5 more)
Recently, few-shot learning (FSL) has become a popular task that aims to recognize new classes from only a few labeled examples and has been widely applied in fields such as natural science, remote sensing, and medical images. However, most existing methods focus only on the visual modality and compute prototypes directly from raw support images, which lack comprehensive and rich
multimodal information. To address these limitations, we propose a novel
Multimodal Prototype Augmentation FSL framework called MPA, including LLM-based Multi-Variant Semantic Enhancement (LMSE), Hierarchical Multi-View Augmentation (HMA), and an Adaptive Uncertain Class Absorber (AUCA). LMSE leverages large
language models to generate diverse paraphrased category descriptions, enriching the support set with additional semantic cues. HMA exploits both natural and multi-view augmentations to enhance feature diversity (e.g., changes in viewing distance, camera angles, and lighting conditions). AUCA models uncertainty by introducing uncertain classes via interpolation and Gaussian sampling, effectively absorbing uncertain samples. Extensive experiments on four single-domain and six cross-domain FSL
benchmarks demonstrate that MPA achieves superior performance compared to existing state-of-the-art methods across most settings. Notably, MPA surpasses the second-best method by 12.29% and 24.56% in the single-domain and cross-domain setting, respectively, in the 5-way 1-shot setting.
Read the full abstract ▾
All authors Liwen Wu, Wei Wang, Lei Zhao, Zhan Gao, Qika Lin, Shaowen Yao, Zuozhu Liu, Bin Pu
Computer Vision
Nikita Starodubcev, Daniil Pakhomov, Zongze Wu et al. (+6 more)
Diffusion transformers typically incorporate textual information via
attention layers and a modulation mechanism using a pooled text
embedding. Nevertheless, recent approaches discard modulation-based text conditioning and rely exclusively on
attention. In this paper, we address whether modulation-based text conditioning is necessary and whether it can provide any performance advantage. Our analysis shows that, in its conventional usage, the pooled
embedding contributes little to overall performance, suggesting that
attention alone is generally sufficient for faithfully propagating
prompt information. However, we reveal that the pooled
embedding can provide significant gains when used from a different perspective-serving as guidance and enabling controllable shifts toward more desirable properties. This approach is training-free, simple to implement, incurs negligible runtime overhead, and can be applied to various
diffusion models, bringing improvements across diverse tasks, including text-to-image/
video generation and image editing.
Read the full abstract ▾
All authors Nikita Starodubcev, Daniil Pakhomov, Zongze Wu, Ilya Drobyshevskiy, Yuchen Liu, Zhonghao Wang, Yuqian Zhou, Zhe Lin, Dmitry Baranchuk
Computer Vision
VLM-Guided Iterative Refinement for Surgical Image Segmentation with Foundation Models
Ange Lou, Yamin Li, Qi Chang et al. (+4 more)
Surgical image
segmentation is essential for
robot-assisted surgery and intraoperative guidance. However, existing methods are constrained to predefined categories, produce one-shot predictions without adaptive refinement, and lack mechanisms for clinician interaction. We propose IR-SIS, an iterative refinement system for
surgical image
segmentation that accepts natural
language descriptions. IR-SIS leverages a fine-tuned SAM3 for initial
segmentation, employs a
Vision-
Language Model to detect instruments and assess
segmentation quality, and applies an agentic workflow that adaptively selects refinement strategies. The system supports clinician-in-the-loop interaction through natural
language feedback. We also construct a multi-granularity
language-annotated
dataset from EndoVis2017 and EndoVis2018
benchmarks. Experiments demonstrate state-of-the-art performance on both in-domain and out-of-distribution data, with clinician interaction providing additional improvements. Our work establishes the first
language-based
surgical segmentation framework with adaptive self-refinement capabilities.
Read the full abstract ▾
All authors Ange Lou, Yamin Li, Qi Chang, Nan Xi, Luyuan Xie, Zichao Li, Tianyu Luan
Computer Vision · Cross-listed from cs.HC
Towards Human-AI Accessibility Mapping in India: VLM-Guided Annotations and POI-Centric Analysis in Chandigarh
Varchita Lalwani, Utkarsh Agarwal, Michael Saugstad et al. (+3 more)
Project Sidewalk is a web-based platform that enables crowdsourcing accessibility of sidewalks at city-scale by virtually walking through city streets using Google Street View. The tool has been used in 40 cities across the world, including the US, Mexico, Chile, and Europe. In this paper, we describe adaptation efforts to enable deployment in Chandigarh, India, including modifying annotation types, provided examples, and integrating VLM-based mission guidance, which adapts instructions based on a street scene and metadata analysis. Our evaluation with 3 annotators indicates the utility of AI-mission guidance with an average score of 4.66. Using this adapted Project Sidewalk tool, we conduct a Points of Interest (POI)-centric accessibility analysis for three sectors in Chandigarh with very different land uses, residential, commercial and institutional covering about 40 km of sidewalks. Across 40 km of roads audited in three sectors and around 230 POIs, we identified 1,644 of 2,913 locations where infrastructure improvements could enhance accessibility.
Read the full abstract ▾
All authors Varchita Lalwani, Utkarsh Agarwal, Michael Saugstad, Manish Kumar, Jon E. Froehlich, Anupam Sobti
Computer Vision
VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models
Chenyu Wang, Tianle Chen, H. M. Sabbir Ahmad et al. (+2 more)
Uncertainty quantification (UQ) is vital for ensuring that
vision-
language models (VLMs) behave safely and reliably. A central challenge is to localize uncertainty to its source, determining whether it arises from the image, the text, or misalignment between the two. We introduce VLM-UQBench, a
benchmark for modality-specific and cross-modal data uncertainty in VLMs, It consists of 600 real-world samples drawn from the VizWiz
dataset, curated into clean, image-, text-, and cross-modal uncertainty subsets, and a scalable perturbation pipeline with 8 visual, 5 textual, and 3 cross-modal perturbations. We further propose two simple metrics that quantify the sensitivity of UQ scores to these perturbations and their correlation with
hallucinations, and use them to evaluate a range of UQ methods across four VLMs and three
datasets. Empirically, we find that: (i) existing UQ methods exhibit strong modality-specific specialization and substantial dependence on the underlying VLM, (ii) modality-specific uncertainty frequently co-occurs with
hallucinations while current UQ scores provide only weak and inconsistent risk signals, and (iii) although UQ methods can rival
reasoning-based chain-of-thought baselines on overt, group-level ambiguity, they largely fail to detect the subtle, instance-level ambiguity introduced by our perturbation pipeline. These results highlight a significant gap between current UQ practices and the fine-grained, modality-aware uncertainty required for reliable VLM deployment.
Read the full abstract ▾
All authors Chenyu Wang, Tianle Chen, H. M. Sabbir Ahmad, Kayhan Batmanghelich, Wenchao Li
Computer Vision
Wearable environmental sensing to forecast how legged systems will interact with upcoming terrain
Michael D. Murray, James Tung and Richard W. Nuckols
Computer-
vision (CV) has been used for environmental classification during gait and is often used to inform control in assistive systems; however, the ability to predict how the foot will contact a changing environment is underexplored. We evaluated the feasibility of
forecasting the anterior-posterior (AP) foot center-of-pressure (COP) and time-of-impact (TOI) prior to foot-strike on a level-ground to stair-ascent transition. Eight subjects wore an RGB-D camera on their right shank and instrumented insoles while performing the task of stepping onto the stairs. We trained a CNN-RNN to forecast the COP and TOI continuously within a 250ms window prior to foot-strike, termed the forecast horizon (FH). The COP mean-absolute-error (MAE) at 150, 100, and 50ms FH was 29.42mm, 26.82, and 23.72mm respectively. The TOI MAE was 21.14, 20.08, and 17.73ms for 150, 100, and 50ms respectively. While torso velocity had no effect on the error in either task, faster toe-swing speeds prior to foot-strike were found to improve the prediction accuracy in the COP case, however, was insignificant in the TOI case. Further, more anterior foot-strikes were found to reduce COP prediction accuracy but did not affect the TOI prediction accuracy. We also found that our lightweight model was capable at running at 60 FPS on either a consumer grade laptop or an edge computing device. This study demonstrates that
forecasting COP and TOI from visual data was feasible using a lightweight model, which may have important implications for anticipatory control in assistive systems.
Read the full abstract ▾
Computer Vision
All-in-One Conditioning for Text-to-Image Synthesis
Hirunima Jayasekara, Chuong Huynh, Yixuan Ren et al. (+2 more)
Accurate interpretation and visual representation of complex
prompts involving multiple objects, attributes, and spatial relationships is a critical challenge in text-to-image synthesis. Despite recent advancements in generating photorealistic outputs, current models often struggle with maintaining semantic fidelity and structural coherence when processing intricate textual inputs. We propose a novel approach that grounds text-to-image synthesis within the framework of scene graph structures, aiming to enhance the compositional abilities of existing models. Eventhough, prior approaches have attempted to address this by using pre-defined layout maps derived from
prompts, such rigid constraints often limit compositional flexibility and diversity. In contrast, we introduce a zero-shot, scene graph-based conditioning mechanism that generates soft visual guidance during inference. At the core of our method is the Attribute-Size-Quantity-Location (ASQL) Conditioner, which produces visual conditions via a lightweight
language model and guides
diffusion-based generation through inference-time optimization. This enables the model to maintain text-image
alignment while supporting lightweight, coherent, and diverse image synthesis.
Read the full abstract ▾
All authors Hirunima Jayasekara, Chuong Huynh, Yixuan Ren, Christabel Acquaye, Abhinav Shrivastava
Computer Vision
Decoding Future Risk: Deep Learning Analysis of Tubular Adenoma Whole-Slide Images
Ahmed Rahu, Brian Shula, Brandon Combs et al. (+4 more)
Colorectal cancer (CRC) remains a significant cause of cancer-related mortality, despite the widespread implementation of prophylactic initiatives aimed at detecting and removing precancerous polyps. Although screening effectively reduces incidence, a notable portion of patients initially diagnosed with low-grade adenomatous polyps will still develop CRC later in life, even without the presence of known high-risk syndromes. Identifying which low-risk patients are at higher risk of progression is a critical unmet need for tailored surveillance and preventative therapeutic strategies. Traditional histological assessment of adenomas, while fundamental, may not fully capture subtle architectural or cytological features indicative of malignant potential. Advancements in digital pathology and machine learning provide an opportunity to analyze whole-slide images (WSIs) comprehensively and objectively. This study investigates whether machine learning
algorithms, specifically convolutional neural networks (CNNs), can detect subtle histological features in WSIs of low-grade tubular adenomas that are predictive of a patient's long-term risk of developing colorectal cancer.
Read the full abstract ▾
All authors Ahmed Rahu, Brian Shula, Brandon Combs, Aqsa Sultana, Surendra P. Singh, Vijayan K. Asari, Derrick Forchetti
Computer Vision
A Hybrid Deterministic Framework for Named Entity Extraction in Broadcast News Video
Andrea Filiberto Lucas and Dylan Seychell
The growing volume of
video-based news content has heightened the need for transparent and reliable methods to extract on-screen information. Yet the variability of graphical layouts, typographic conventions, and platform-specific design patterns renders manual indexing impractical. This work presents a comprehensive framework for automatically detecting and extracting personal names from broadcast and social-media-native news
videos. It introduces a curated and balanced corpus of annotated frames capturing the diversity of contemporary news graphics and proposes an interpretable, modular extraction pipeline designed to operate under deterministic and auditable conditions. The pipeline is evaluated against a contrasting class of generative
multimodal methods, revealing a clear trade-off between deterministic auditability and
stochastic inference. The underlying detector achieves 95.8% mAP@0.5, demonstrating operationally robust performance for graphical element localisation. While generative systems achieve marginally higher raw accuracy (F1: 84.18% vs 77.08%), they lack the transparent data lineage required for journalistic and analytical contexts. The proposed pipeline delivers balanced precision (79.9%) and recall (74.4%), avoids
hallucination, and provides full traceability across each processing stage. Complementary user findings indicate that 59% of respondents report difficulty reading on-screen names in fast-paced broadcasts, underscoring the practical relevance of the task. The results establish a methodologically rigorous and interpretable baseline for hybrid
multimodal information extraction in modern news media.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes
Nicholas Pfaff, Thomas Cohn, Sergey Zakharov et al. (+2 more)
Simulation has become a key tool for training and evaluating home
robots at scale, yet existing environments fail to capture the diversity and physical complexity of real indoor spaces. Current scene synthesis methods produce sparsely furnished rooms that lack the dense clutter, articulated furniture, and physical properties essential for robotic manipulation. We introduce SceneSmith, a hierarchical agentic framework that generates simulation-ready indoor environments from natural
language prompts. SceneSmith constructs scenes through successive stages$\unicode{x2013}$from architectural layout to furniture placement to small object population$\unicode{x2013}$each implemented as an interaction among VLM
agents: designer, critic, and orchestrator. The framework tightly integrates asset generation through text-to-3D synthesis for static objects,
dataset retrieval for articulated objects, and physical property estimation. SceneSmith generates 3-6x more objects than prior methods, with <2% inter-object collisions and 96% of objects remaining stable under physics simulation. In a user study with 205 participants, it achieves 92% average realism and 91% average
prompt faithfulness win rates against baselines. We further demonstrate that these environments can be used in an end-to-end pipeline for automatic
robot policy evaluation.
Read the full abstract ▾
All authors Nicholas Pfaff, Thomas Cohn, Sergey Zakharov, Rick Cory, Russ Tedrake
Computer Vision
SemanticMoments: Training-Free Motion Similarity via Third Moment Features
Saar Huberman, Kfir Goldberg, Or Patashnik et al. (+2 more)
Retrieving
videos based on semantic motion is a fundamental, yet unsolved, problem. Existing
video representation approaches overly rely on static appearance and scene context rather than motion dynamics, a bias inherited from their training data and objectives. Conversely, traditional motion-centric inputs like optical flow lack the semantic grounding needed to understand high-level motion. To demonstrate this inherent bias, we introduce the SimMotion
benchmarks, combining controlled synthetic data with a new human-annotated real-world
dataset. We show that existing models perform poorly on these
benchmarks, often failing to disentangle motion from appearance. To address this gap, we propose SemanticMoments, a simple, training-free method that computes temporal statistics (specifically, higher-order moments) over features from pre-trained semantic models. Across our
benchmarks, SemanticMoments consistently outperforms existing RGB, flow, and text-supervised methods. This demonstrates that temporal statistics in a semantic feature space provide a scalable and perceptually grounded foundation for motion-centric
video understanding.
Read the full abstract ▾
All authors Saar Huberman, Kfir Goldberg, Or Patashnik, Sagie Benaim, Ron Mokady
Computer Vision · Cross-listed from cs.LG
Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide
Hossam Amer, Rezaul Karim, Ali Pourranjbar et al. (+3 more)
With the rapid growth of large
language models (LLMs), a wide range of methods have been developed to distribute computation and
memory across hardware devices for efficient training and inference. While existing surveys provide descriptive overviews of these techniques, systematic analysis of their benefits and trade offs and how such insights can inform principled methodology for designing optimal distributed systems remain limited. This paper offers a comprehensive review of collective operations and distributed parallel strategies, complemented by mathematical formulations to deepen theoretical understanding. We further examine hybrid parallelization designs, emphasizing communication computation overlap across different stages of model deployment, including both training and inference. Recent advances in automated search for optimal hybrid parallelization strategies using cost models are also discussed. Moreover, we present case studies with mainstream architecture categories to reveal empirical insights to guide researchers and practitioners in parallelism strategy selection. Finally, we highlight open challenges and limitations of current LLM training paradigms and outline promising directions for the next generation of large scale model development.
Read the full abstract ▾
All authors Hossam Amer, Rezaul Karim, Ali Pourranjbar, Weiwei Zhang, Walid Ahmed, Boxing Chen
Computer Vision
Agent Banana: High-Fidelity Image Editing with Agentic Thinking and Tooling
Ruijie Ye, Jiayi Zhang, Zhuoxin Liu et al. (+10 more)
We study instruction-based image editing under professional workflows and identify three persistent challenges: (i) editors often over-edit, modifying content beyond the user's intent; (ii) existing models are largely single-turn, while multi-turn edits can alter object faithfulness; and (iii) evaluation at around 1K resolution is misaligned with real workflows that often operate on ultra high-definition images (e.g., 4K). We propose
Agent Banana, a hierarchical agentic planner-executor framework for high-fidelity, object-aware, deliberative editing.
Agent Banana introduces two key mechanisms: (1) Context Folding, which compresses long interaction histories into structured
memory for stable long-horizon control; and (2) Image Layer Decomposition, which performs localized layer-based edits to preserve non-target regions while enabling native-resolution outputs. To support rigorous evaluation, we build HDD-Bench, a high-definition, dialogue-based
benchmark featuring verifiable stepwise targets and native 4K images (11.8M pixels) for diagnosing long-horizon failures. On HDD-Bench,
Agent Banana achieves the best multi-turn consistency and background fidelity (e.g., IC 0.871, SSIM-OM 0.84, LPIPS-OM 0.12) while remaining competitive on instruction following, and also attains strong performance on standard single-turn editing
benchmarks. We hope this work advances reliable, professional-grade agentic image editing and its integration into real workflows.
Read the full abstract ▾
All authors Ruijie Ye, Jiayi Zhang, Zhuoxin Liu, Zihao Zhu, Siyuan Yang, Li Li, Tianfu Fu, Franck Dernoncourt, Yue Zhao, Jiacheng Zhu, Ryan Rossi, Wenhao Chai, Zhengzhong Tu
Computer Vision
UI-Venus-1.5 Technical Report
Venus Team, Changlong Gao, Zhangxuan Gu et al. (+24 more)
GUI
agents have emerged as a powerful paradigm for automating interactions in digital environments, yet achieving both broad generality and consistently strong task performance remains challenging. In this report, we present UI-Venus-1.5, a unified, end-to-end GUI
Agent designed for robust real-world applications. The proposed model family comprises two dense variants (2B and 8B) and one mixture-of-experts variant (30B-A3B) to meet various downstream application scenarios. Compared to our previous version, UI-Venus-1.5 introduces three key technical advances: (1) a comprehensive Mid-Training stage leveraging 10 billion
tokens across 30+
datasets to establish foundational GUI semantics; (2) Online Reinforcement Learning with full-trajectory rollouts, aligning training objectives with long-horizon, dynamic
navigation in large-scale environments; and (3) a single unified GUI
Agent constructed via Model Merging, which synthesizes domain-specific models (grounding, web, and mobile) into one cohesive checkpoint. Extensive evaluations demonstrate that UI-Venus-1.5 establishes new state-of-the-art performance on
benchmarks such as ScreenSpot-Pro (69.6%), VenusBench-GD (75.0%), and AndroidWorld (77.6%), significantly outperforming previous strong baselines. In addition, UI-Venus-1.5 demonstrates robust
navigation capabilities across a variety of Chinese mobile apps, effectively executing user instructions in real-world scenarios. Code: https://github.com/inclusionAI/UI-Venus; Model: https://huggingface.co/collections/inclusionAI/ui-venus
Read the full abstract ▾
All authors Venus Team, Changlong Gao, Zhangxuan Gu, Yulin Liu, Xinyu Qiu, Shuheng Shen, Yue Wen, Tianyu Xia, Zhenyu Xu, Zhengwen Zeng, Beitong Zhou, Xingran Zhou, Weizhi Chen, Sunhao Dai, Jingya Dou, Yichen Gong, Yuan Guo, Zhenlin Guo, Feng Li, Qian Li, Jinzhen Lin, Yuqi Zhou, Linchao Zhu, Liang Chen, Zhenyu Guo, Changhua Meng, Weiqiang Wang
Computer Vision
Autoregressive Image Generation with Masked Bit Modeling
Qihang Yu, Qihao Liu, Ju He et al. (+4 more)
This paper challenges the dominance of continuous pipelines in visual generation. We systematically investigate the performance gap between discrete and continuous methods. Contrary to the belief that discrete tokenizers are intrinsically inferior, we demonstrate that the disparity arises primarily from the total number of bits allocated in the latent space (i.e., the compression ratio). We show that scaling up the codebook size effectively bridges this gap, allowing discrete tokenizers to match or surpass their continuous counterparts. However, existing discrete generation methods struggle to capitalize on this insight, suffering from performance degradation or prohibitive training costs with scaled codebook. To address this, we propose masked Bit AutoRegressive modeling (BAR), a scalable framework that supports arbitrary codebook sizes. By equipping an autoregressive
transformer with a masked bit modeling head, BAR predicts discrete
tokens through progressively generating their constituent bits. BAR achieves a new state-of-the-art gFID of 0.99 on ImageNet-256, outperforming leading methods across both continuous and discrete paradigms, while significantly reducing sampling costs and converging faster than prior continuous approaches. Project page is available at https://bar-gen.github.io/
Read the full abstract ▾
All authors Qihang Yu, Qihao Liu, Ju He, Xinyang Zhang, Yang Liu, Liang-Chieh Chen, Xi Chen
Computer Vision
WorldCompass: Reinforcement Learning for Long-Horizon World Models
Zehan Wang, Tengfei Wang, Haiyu Zhang et al. (+9 more)
This work presents WorldCompass, a novel Reinforcement Learning (RL) post-training framework for the long-horizon, interactive
video-based world models, enabling them to explore the world more accurately and consistently based on interaction signals. To effectively "steer" the world model's exploration, we introduce three core innovations tailored to the autoregressive
video generation paradigm: 1) Clip-level rollout Strategy: We generate and evaluate multiple samples at a single target clip, which significantly boosts rollout efficiency and provides fine-grained reward signals. 2) Complementary Reward Functions: We design reward functions for both interaction-following accuracy and visual quality, which provide direct supervision and effectively suppress reward-hacking behaviors. 3) Efficient RL
Algorithm: We employ the negative-aware fine-tuning strategy coupled with various efficiency optimizations to efficiently and effectively enhance model capacity. Evaluations on the SoTA open-source world model, WorldPlay, demonstrate that WorldCompass significantly improves interaction accuracy and visual fidelity across various scenarios.
Read the full abstract ▾
All authors Zehan Wang, Tengfei Wang, Haiyu Zhang, Xuhui Zuo, Junta Wu, Haoyuan Wang, Wenqiang Sun, Zhenwei Wang, Chenjie Cao, Hengshuang Zhao, Chunchao Guo, Zhou Zhao