Computer Vision
A Computer Vision Framework for Multi-Class Detection and Tracking in Soccer Broadcast Footage
Daniel Tshiani
Clubs with access to expensive multi-camera setups or GPS tracking systems gain a competitive advantage through detailed data, whereas lower-budget teams are often unable to collect similar information. This paper examines whether such data can instead be extracted directly from standard broadcast footage using a single-camera computer
vision pipeline. This project develops an end-to-end system that combines a YOLO object detector with the ByteTrack tracking
algorithm to identify and track players, referees, goalkeepers, and the ball throughout a match. Experimental results show that the pipeline achieves high performance in detecting and tracking players and officials, with strong precision, recall, and mAP50 scores, while ball detection remains the primary challenge. Despite this limitation, our findings demonstrate that AI can extract meaningful player-level spatial information from a single broadcast camera. By reducing reliance on specialized hardware, the proposed approach enables colleges, academies, and amateur clubs to adopt scalable, data-driven analysis methods previously accessible only to professional teams, highlighting the potential for affordable computer
vision-based soccer analytics.
Read the full abstract ▾
Computer Vision
Mitigating Shortcut Learning via Feature Disentanglement in Medical Imaging: A Benchmark Study
Sarah Müller and Philipp Berens
Although deep learning models in medical imaging often achieve excellent classification performance, they can rely on shortcut learning, exploiting spurious correlations or confounding factors that are not causally related to the target task. This poses risks in clinical settings, where models must generalize across institutions, populations, and acquisition conditions. Feature disentanglement is a promising approach to mitigate shortcut learning by separating task-relevant information from confounder-related features in latent representations. In this study, we systematically evaluated feature disentanglement methods for mitigating shortcuts in medical imaging, including
adversarial learning and latent space splitting based on dependence minimization. We assessed classification performance and disentanglement quality using latent space analyses across one artificial and two medical
datasets with natural and synthetic confounders. We also examined robustness under varying levels of confounding and compared computational efficiency across methods. We found that shortcut mitigation methods improved classification performance under strong spurious correlations during training. Latent space analyses revealed differences in representation quality not captured by classification metrics, highlighting the strengths and limitations of each method. Model reliance on shortcuts depended on the degree of confounding in the training data. The best-performing models combine data-centric rebalancing with model-centric disentanglement, achieving stronger and more robust shortcut mitigation than rebalancing alone while maintaining similar computational efficiency. The project code is publicly available at https://github.com/berenslab/medical-shortcut-mitigation.
Read the full abstract ▾
Computer Vision
Scaling Ultrasound Volumetric Reconstruction via Mobile Augmented Reality
Kian Wei Ng, Yujia Gao, Deborah Khoo et al. (+7 more)
Accurate volumetric characterization of lesions is essential for oncologic diagnosis, risk stratification, and treatment planning. While imaging modalities such as Computed
Tomography provide high-quality 3D data, 2D ultrasound (2D-US) remains the preferred first-line modality for breast and thyroid imaging due to cost, portability, and safety factors. However, volume estimates derived from 2D-US suffer from high inter-user variability even among experienced clinicians. Existing 3D ultrasound (3D-US) solutions use specialized probes or external tracking hardware, but such configurations increase costs and diminish portability, constraining widespread clinical use. To address these limitations, we present Mobile Augmented Reality Volumetric Ultrasound (MARVUS), a resource-efficient system designed to increase accessibility to accurate and reproducible volumetric assessment. MARVUS is interoperable with conventional ultrasound (US) systems, using a foundation model to enhance cross-specialty generalization while minimizing hardware requirements relative to current 3D-US solutions. In a user study involving experienced clinicians performing measurements on breast phantoms, MARVUS yielded a substantial improvement in volume estimation accuracy (mean difference: 0.469 cm3) with reduced inter-user variability (mean difference: 0.417 cm3). Additionally, we prove that augmented reality (AR) visualizations enhance objective performance metrics and clinician-reported usability. Collectively, our findings suggests that MARVUS can enhance US-based cancer screening, diagnostic workflows, and treatment planning in a scalable, cost-conscious, and resource-efficient manner. Usage
video demonstration available (https://youtu.be/m4llYcZpqmM).
Read the full abstract ▾
All authors Kian Wei Ng, Yujia Gao, Deborah Khoo, Ying Zhen Tan, Chengzheng Mao, Haojie Cheng, Andrew Makmur, Kee Yuan Ngiam, Serene Goh, Eng Tat Khoo
Computer Vision
LGQ: Learnable Geometric Quantization for Image Tokenization
Idil Bilge Altun, Mert Onur Cakiroglu, Elham Buxton et al. (+2 more)
Recent collapse-free quantizers such as FSQ achieve stable training by replacing the learnable codebook with an engineered geometry: a fixed scalar grid whose structure is dictated by the codebook size K. We show this trade-off is unnecessary. We introduce Learnable
Geometric Quantization (LGQ), which retains a learnable codebook of codes and performs soft-to-hard assignment via temperature annealing, regularized by two cheap terms: a diversity term scaled by codebook size that penalizes concentrated batch-average usage is the primary driver of collapse resistance, complemented by a peakedness term that sharpens each
token's soft-assignment toward one-hot; together they prevent codebook collapse without EMA, reset heuristics, or codebook reparameterization. Under a fixed VQ-GAN backbone, we
benchmark LGQ against RotVQ, FSQ, LFQ, SimVQ, and IBQ on ImageNet 256x256 at K = 16,384, and sweep LGQ over K in {4096, ..., 65,536} without any per-K hyperparameter tuning. LGQ attains the best reconstruction FID at K = 16,384 while maintaining 100% codebook utilization, and continues to improve as the codebook grows to K = 65,536, holding 100% utilization at every K. Training MaskGIT on the frozen tokenizers, LGQ further attains the best class-conditional generation among the compared quantizers, leading on reconstruction and generation alike. Code is available at https://anonymous.4open.science/r/lgq-anon-E12C/.
Read the full abstract ▾
All authors Idil Bilge Altun, Mert Onur Cakiroglu, Elham Buxton, Mehmet Dalkilic, Hasan Kurban
Computer Vision · Cross-listed from cs.LG
Extracting and Analyzing Rail Crossing Behavior Signatures from Videos using Tensor Methods
Dawon Ahn, Het Patel, Aemal Khattak et al. (+2 more)
Railway crossings present complex safety challenges where driver behavior varies by location, time, and conditions. Traditional approaches analyze crossings individually, limiting the ability to identify shared behavioral patterns across locations. We propose a multi-view tensor decomposition framework that captures behavioral similarities across three temporal phases: Approach (warning activation to gate lowering), Waiting (gates down to train passage), and Clearance (train passage to gate raising). We analyze railway crossing
videos from multiple locations using TimeSformer
embeddings to represent each phase. By constructing phase-specific similarity
matrices and applying non-negative symmetric CP decomposition, we discover latent behavioral components with distinct temporal signatures. Our tensor analysis reveals that crossing location appears to be a stronger determinant of behavior patterns than time of day, and that approach-phase behavior provides particularly discriminative signatures. Visualization of the learned component space confirms location-based clustering, with certain crossings forming distinct behavioral clusters. This automated framework enables scalable pattern discovery across multiple crossings, providing a foundation for grouping locations by behavioral similarity to inform targeted safety interventions.
Read the full abstract ▾
All authors Dawon Ahn, Het Patel, Aemal Khattak, Jia Chen, Evangelos E. Papalexakis
Computer Vision
MedProbCLIP: Probabilistic Adaptation of Vision-Language Foundation Model for Reliable Radiograph-Report Retrieval
Ahmad Elallaf, Yu Zhang, Yuktha Priya Masupalli et al. (+4 more)
Vision-
language foundation models have emerged as powerful general-purpose representation learners with strong potential for
multimodal understanding, but their deterministic
embeddings often fail to provide the reliability required for high-stakes biomedical applications. This work introduces MedProbCLIP, a probabilistic
vision-
language learning framework for chest X-ray and radiology report representation learning and bidirectional
retrieval. MedProbCLIP models image and text representations as Gaussian
embeddings through a probabilistic contrastive objective that explicitly captures uncertainty and many-to-many correspondences between radiographs and clinical narratives. A variational information bottleneck mitigates overconfident predictions, while MedProbCLIP employs multi-view radiograph encoding and multi-section report encoding during training to provide fine-grained supervision for clinically aligned correspondence, yet requires only a single radiograph and a single report at inference. Evaluated on the MIMIC-CXR
dataset, MedProbCLIP outperforms deterministic and probabilistic baselines, including CLIP, CXR-CLIP, and PCME++, in both
retrieval and zero-shot classification. Beyond accuracy, MedProbCLIP demonstrates superior calibration, risk-coverage behavior, selective
retrieval reliability, and robustness to clinically relevant corruptions, underscoring the value of probabilistic
vision-
language modeling for improving the trustworthiness and safety of radiology image-text
retrieval systems.
Read the full abstract ▾
All authors Ahmad Elallaf, Yu Zhang, Yuktha Priya Masupalli, Jeong Yang, Young Lee, Zechun Cao, Gongbo Liang
Computer Vision
BTReport: A Framework for Brain Tumor Radiology Report Generation with Clinically Relevant Features
Juampablo E. Heras Rivera, Dickson T. Chen, Tianyi Ren et al. (+4 more)
Recent advances in radiology report generation (RRG) have been driven by large paired image-text
datasets; however, progress in neuro-oncology has been limited due to a lack of open paired image-report
datasets. Here, we introduce BTReport, an open-source framework for
brain tumor RRG that constructs natural
language radiology reports using deterministically extracted imaging features. Unlike existing approaches that rely on large general-purpose or fine-tuned
vision-
language models for both image interpretation and report composition, BTReport performs deterministic feature extraction for image analysis and uses large
language models only for syntactic structuring and narrative formatting. By separating RRG into a deterministic feature extraction step and a report generation step, the generated reports are completely interpretable and less prone to
hallucinations. We show that the features used for report generation are predictive of key clinical outcomes, including survival and IDH mutation status, and reports generated by BTReport are more closely aligned with reference clinical reports than existing baselines for RRG. Finally, we introduce BTReport-BraTS, a companion
dataset that augments BraTS imaging with synthetically generated radiology reports produced with BTReport. Code for this project can be found at https://github.com/KurtLabUW/BTReport.
Read the full abstract ▾
All authors Juampablo E. Heras Rivera, Dickson T. Chen, Tianyi Ren, Daniel K. Low, Asma Ben Abacha, Alberto Santamaria-Pang, Mehmet Kurt
Computer Vision
SAM 3D Body: Robust Full-Body Human Mesh Recovery
Xitong Yang, Devansh Kukreja, Don Pinkus et al. (+11 more)
We introduce SAM 3D Body (3DB), a promptable model for single-image full-body 3D human mesh recovery (HMR) that demonstrates state-of-the-art performance, with strong generalization and consistent accuracy in diverse in-the-wild conditions. 3DB estimates the human pose of the body, feet, and hands. It is the first model to use a new parametric mesh representation, Momentum Human Rig (MHR), which decouples skeletal structure and surface shape. 3DB employs an encoder-decoder architecture and supports auxiliary
prompts, including 2D keypoints and masks, enabling user-guided inference similar to the SAM family of models. We derive high-quality annotations from a multi-stage annotation pipeline that uses various combinations of manual keypoint annotation, differentiable optimization, multi-view geometry, and dense keypoint detection. Our data engine efficiently selects and processes data to ensure data diversity, collecting unusual poses and rare imaging conditions. We present a new evaluation
dataset organized by pose and appearance categories, enabling nuanced analysis of model behavior. Our experiments demonstrate superior generalization and substantial improvements over prior methods in both qualitative user preference studies and traditional quantitative analysis. Both 3DB and MHR are open-source.
Read the full abstract ▾
All authors Xitong Yang, Devansh Kukreja, Don Pinkus, Anushka Sagar, Taosha Fan, Jinhyung Park, Soyong Shin, Jinkun Cao, Jiawei Liu, Nicolas Ugrinovic, Matt Feiszli, Jitendra Malik, Piotr Dollar, Kris Kitani
Computer Vision · Cross-listed from eess.IV
Automated Assessment of Kidney Ureteroscopy Exploration for Training
Fangjie Li, Nicholas Kavoussi, Charan Mohan et al. (+2 more)
Purpose: Kidney ureteroscopic
navigation is challenging with a steep learning curve. However, current clinical training has major deficiencies, as it requires one-on-one feedback from experts and occurs in the operating room (OR). Therefore, there is a need for a phantom training system with automated feedback to greatly \revision{expand} training opportunities. Methods: We propose a novel, purely ureteroscope
video-based scope localization framework that automatically identifies calyces missed by the trainee in a phantom kidney exploration. We use a slow, thorough, prior exploration
video of the kidney to generate a reference reconstruction. Then, this reference reconstruction can be used to localize any exploration
video of the same phantom. Results: In 15 exploration
videos, a total of 69 out of 74 calyces were correctly classified. We achieve < 4mm camera pose localization error. Given the reference reconstruction, the system takes 10 minutes to generate the results for a typical exploration (1-2 minute long). Conclusion: We demonstrate a novel camera localization framework that can provide accurate and automatic feedback for kidney phantom explorations. We show its ability as a valid tool that enables out-of-OR training without requiring supervision from an expert.
Read the full abstract ▾
All authors Fangjie Li, Nicholas Kavoussi, Charan Mohan, Matthieu Chabanas, Jie Ying Wu
Computer Vision
LAND: A Longitudinal Analysis of Neuromorphic Datasets
Gregory Cohen and Alexandre Marcireau
Neuromorphic engineering has a data problem. Despite the meteoric rise in the number of neuromorphic
datasets published over the past ten years, the conclusion of a significant portion of neuromorphic research papers still states that there is a need for yet more data and even larger
datasets. Whilst this need is driven in part by the sheer volume of data required by modern deep learning approaches, it is also fuelled by the current state of the available neuromorphic
datasets and the difficulties in finding them, understanding their purpose, and determining the nature of their underlying task. This is further compounded by practical difficulties in downloading and using these
datasets. This review starts by capturing a snapshot of the existing neuromorphic
datasets, covering over 423
datasets, and then explores the nature of their tasks and the underlying structure of the presented data. Analysing these
datasets shows the difficulties arising from their size, the lack of standardisation, and difficulties in accessing the actual data. This paper also highlights the growth in the size of individual
datasets and the complexities involved in working with the data. However, a more important concern is the rise of synthetic
datasets, created by either simulation or
video-to-events methods. This review explores the benefits of simulated data for testing existing
algorithms and applications, highlighting the potential pitfalls for exploring new applications of neuromorphic technologies. This review also introduces the concepts of meta-
datasets, created from existing
datasets, as a way of both reducing the need for more data, and to remove potential bias arising from defining both the
dataset and the task.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
B-DENSE: Branching For Dense Ensemble Network Supervision Efficiency
Cherish Puniani, Tushar Kumar, Arnav Bendre et al. (+2 more)
Inspired by non-
equilibrium thermodynamics,
diffusion models have achieved state-of-the-art performance in generative modeling. However, their iterative sampling nature results in high inference latency. While recent
distillation techniques accelerate sampling, they discard intermediate trajectory steps. This sparse supervision leads to a loss of structural information and introduces significant discretization errors. To mitigate this, we propose B-DENSE, a novel framework that leverages multi-branch trajectory
alignment. We modify the student architecture to output $K$-fold expanded channels, where each subset corresponds to a specific branch representing a discrete intermediate step in the teacher's trajectory. By training these branches to simultaneously map to the entire sequence of the teacher's target timesteps, we enforce dense intermediate trajectory
alignment. Consequently, the student model learns to navigate the solution space from the earliest stages of training, demonstrating superior image generation quality compared to baseline
distillation frameworks.
Read the full abstract ▾
All authors Cherish Puniani, Tushar Kumar, Arnav Bendre, Gaurav Kumar, Shree Singhi
Computer Vision
Non-Contact Physiological Monitoring in Pediatric Intensive Care Units via Adaptive Masking and Self-Supervised Learning
Mohamed Khalil Ben Salah, Philippe Jouvet and Rita Noumeir
Continuous monitoring of vital signs in Pediatric Intensive Care Units (PICUs) is essential for early detection of clinical deterioration and effective clinical decision-making. However, contact-based sensors such as pulse oximeters may cause skin irritation, increase infection risk, and lead to patient discomfort. Remote photoplethysmography (rPPG) offers a contactless alternative to monitor heart rate using facial
video, but remains underutilized in PICUs due to motion artifacts, occlusions, variable lighting, and domain shifts between laboratory and clinical data. We introduce a self-supervised pretraining framework for rPPG estimation in the PICU setting, based on a progressive curriculum strategy. The approach leverages the VisionMamba architecture and integrates an adaptive masking mechanism, where a lightweight Mamba-based controller assigns spatiotemporal importance scores to guide probabilistic patch sampling. This strategy dynamically increases reconstruction difficulty while preserving physiological relevance. To address the lack of labeled clinical data, we adopt a teacher-student
distillation setup. A supervised expert model, trained on public
datasets, provides latent physiological guidance to the student. The curriculum progresses through three stages: clean public
videos, synthetic occlusion scenarios, and unlabeled
videos from 500 pediatric patients. Our framework achieves a 42% reduction in mean absolute error relative to standard masked autoencoders and outperforms PhysFormer by 31%, reaching a final MAE of 3.2 bpm. Without explicit region-of-interest extraction, the model consistently attends to pulse-rich areas and demonstrates robustness under clinical occlusions and noise.
Read the full abstract ▾
Computer Vision
Automated Re-Identification of Holstein-Friesian Cattle in Dense Crowds
Phoenix Yu, Tilo Burghardt, Andrew W Dowsey et al. (+1 more)
Holstein-Friesian detection and re-identification (Re-ID) methods capture individuals well when targets are spatially separate. However, existing approaches, including YOLO-based species detection, break down when cows group closely together. This is particularly prevalent for species which have outline-breaking coat patterns. To boost both effectiveness and transferability in this setting, we propose a new detect-segment-identify pipeline that leverages the Open-Vocabulary Weight-free Localisation and the Segment Anything models as pre-processing stages alongside Re-ID networks. To evaluate our approach, we publish a collection of nine days CCTV data filmed on a working dairy farm. Our methodology overcomes detection breakdown in dense animal groupings, resulting in a 98.93% accuracy. This significantly outperforms current oriented bounding box-driven, as well as SAM species detection baselines with accuracy improvements of 47.52% and 27.13%, respectively. We show that unsupervised contrastive learning can build on this to yield 94.82% Re-ID accuracy on our test data. Our work demonstrates that Re-ID in crowded scenarios is both practical as well as reliable in working farm settings with no manual intervention. Code and
dataset are provided for reproducibility.
Read the full abstract ▾
All authors Phoenix Yu, Tilo Burghardt, Andrew W Dowsey, Neill W Campbell
Computer Vision
Deformation-Free Cross-Domain Image Registration via Position-Encoded Temporal Attention
Yiwen Wang and Jiahao Qin
We address the problem of cross-domain image registration, where paired images exhibit coupled
geometric misalignment and domain-specific appearance shift. We formalize this as a factorization problem: decomposing each image into a domain-invariant scene representation and a global appearance statistic, such that registration reduces to recombining the scene structure of the moving image with the appearance of the fixed image via Adaptive Instance Normalization (AdaIN). This factorization eliminates the need for explicit deformation field estimation. To exploit temporal coherence in sequential acquisitions, we introduce a position-encoded cross-frame
attention mechanism that fuses learnable and sinusoidal position
embeddings with multi-head
attention over a sliding window of neighboring frames, enriching the scene representation with inter-frame context. We instantiate this framework as GPEReg-Net and evaluate on two
benchmarks: FIRE-Reg-256 (retinal fundus, semi-rigid) and HPatches-Reg-256 (synthetic textured patches, affine). GPEReg-Net achieves state-of-the-art performance on both
benchmarks (FIRE: SSIM = 0.928, PSNR = 33.47 dB; HPatches: SSIM = 0.450, PSNR = 21.01 dB), surpassing all baselines, including deformation-based methods, while running 1.87x faster than SAS-Net. Code: https://github.com/JiahaoQin/GPEReg-Net.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splitting
Md Mofijul Islam, Md Sirajus Salekin, Nivedha Balakrishnan et al. (+6 more)
Document understanding in real-world applications often requires processing heterogeneous, multi-page document packets containing multiple documents stitched together. Despite recent advances in visual document understanding, the fundamental task of document packet splitting, which involves separating a document packet into individual units, remains largely unaddressed. We present the first comprehensive
benchmark dataset, DocSplit, along with novel evaluation metrics for assessing the document packet splitting capabilities of large
language models. DocSplit comprises five
datasets of varying complexity, covering diverse document types, layouts, and
multimodal settings. We formalize the DocSplit task, which requires models to identify document boundaries, classify document types, and maintain correct page ordering within a document packet. The
benchmark addresses real-world challenges, including out-of-order pages, interleaved documents, and documents lacking clear demarcations. We conduct extensive experiments evaluating
multimodal LLMs on our
datasets, revealing significant performance gaps in current models' ability to handle complex document splitting tasks. The DocSplit
benchmark datasets and proposed novel evaluation metrics provide a systematic framework for advancing document understanding capabilities essential for legal, financial, healthcare, and other document-intensive domains. We release the
datasets to facilitate future research in document packet processing.
Read the full abstract ▾
All authors Md Mofijul Islam, Md Sirajus Salekin, Nivedha Balakrishnan, Vincil C. Bishop III, Niharika Jain, Spencer Romo, Bob Strahan, Boyi Xie, Diego A. Socolinsky
Computer Vision
Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families
Yuval Levental
We present a simple experiment that exposes a fundamental limitation in
vision-
language models (VLMs): the inability to accurately localize filled
cells in binary grids when those
cells lack textual identity. We generate fifteen 15x15 grids with varying density (10.7%-41.8% filled
cells) and render each as two image types – text symbols (. and #) and filled squares without gridlines – then ask three frontier VLMs (Claude Opus, ChatGPT 5.2, and Gemini 3 Thinking) to transcribe them. In the text-symbol condition, Claude and ChatGPT achieve approximately 91%
cell accuracy and 84% F1, while Gemini achieves 84% accuracy and 63% F1. In the filled-squares condition, all three models collapse to 60-73% accuracy and 29-39% F1. Critically, all conditions pass through the same visual encoder – the text symbols are images, not tokenized text. The text-vs-squares F1 gap ranges from 34 to 54 points across models, demonstrating that VLMs behave as if they possess a high-fidelity text-recognition pathway for spatial
reasoning that dramatically outperforms their native visual pathway. Each model exhibits a distinct failure mode in the squares condition – systematic under-counting (Claude), massive over-counting (ChatGPT), and template
hallucination (Gemini) – but all share the same underlying deficit: severely degraded spatial localization for non-textual visual elements.
Read the full abstract ▾
Computer Vision
Visual Memory Injection Attacks for Multi-Turn Conversations
Christian Schlarmann and Matthias Hein
Generative large
vision-
language models (LVLMs) have recently achieved impressive performance gains, and their user base is growing rapidly. However, the security of LVLMs, in particular in a long-context multi-turn setting, is largely underexplored. In this paper, we consider the realistic scenario in which an attacker uploads a manipulated image to the web/social media. A benign user downloads this image and uses it as input to the LVLM. Our novel stealthy Visual
Memory Injection (VMI) attack is designed such that on normal
prompts the LVLM exhibits nominal behavior, but once the user gives a triggering
prompt, the LVLM outputs a specific prescribed target message to manipulate the user, e.g. for
adversarial marketing or political persuasion. Compared to previous work that focused on single-turn attacks, VMI is effective even after a long multi-turn conversation with the user. We demonstrate our attack on several recent open-weight LVLMs. This article thereby shows that large-scale manipulation of users is feasible with perturbed images in multi-turn conversation settings, calling for better robustness of LVLMs against these attacks. We release the source code at https://github.com/chs20/visual-
memory-injection
Read the full abstract ▾
Computer Vision
A Study on Real-time Object Detection using Deep Learning
Ankita Bose, Jayasravani Bhumireddy and Naveen N
Object detection has compelling applications over a range of domains, including human-computer interfaces, security and
video surveillance,
navigation and road traffic monitoring, transportation systems, industrial automation healthcare, the world of Augmented Reality (AR) and Virtual Reality (VR), environment monitoring and activity identification. Applications of real time object detection in all these areas provide dynamic analysis of the visual information that helps in immediate decision making. Furthermore, advanced deep learning
algorithms leverage the progress in the field of object detection providing more accurate and efficient solutions. There are some outstanding deep learning
algorithms for object detection which includes, Faster R CNN(Region-based Convolutional Neural Network),Mask R-CNN, Cascade R-CNN, YOLO (You Only Look Once), SSD (Single Shot Multibox Detector), RetinaNet etc. This article goes into great detail on how deep learning
algorithms are used to enhance real time object recognition. It provides information on the different object detection models available, open
benchmark datasets, and studies on the use of object detection models in a range of applications. Additionally, controlled studies are provided to compare various strategies and produce some illuminating findings. Last but not least, a number of encouraging challenges and approaches are offered as suggestions for further investigation in both relevant deep learning approaches and object recognition.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
World Action Models are Zero-shot Policies
Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng et al. (+33 more)
State-of-the-art
Vision-
Language-Action (VLA) models excel at semantic generalization but struggle to generalize to unseen physical motions in novel environments. We introduce DreamZero, a World Action Model (WAM) built upon a pretrained
video diffusion backbone. Unlike VLAs, WAMs learn physical dynamics by predicting future world states and actions, using
video as a dense representation of how the world evolves. By jointly modeling
video and action, DreamZero learns diverse skills effectively from heterogeneous
robot data without relying on repetitive demonstrations. This results in over 2x improvement in generalization to new tasks and environments compared to state-of-the-art VLAs in real
robot experiments. Crucially, through model and system optimizations, we enable a 14B autoregressive
video diffusion model to perform real-time closed-loop control at 7Hz. Finally, we demonstrate two forms of cross-embodiment transfer:
video-only demonstrations from other
robots or humans yield a relative improvement of over 42% on unseen task performance with just 10-20 minutes of data. More surprisingly, DreamZero enables few-shot embodiment adaptation, transferring to a new embodiment with only 30 minutes of play data while retaining zero-shot generalization.
Read the full abstract ▾
All authors Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, Ayaan Malik, Kyungmin Lee, William Liang, Nadun Ranawaka, Jiasheng Gu, Yinzhen Xu, Guanzhi Wang, Fengyuan Hu, Avnish Narayan, Johan Bjorck, Jing Wang, Gwanghyun Kim, Dantong Niu, Ruijie Zheng, Yuqi Xie, Jimmy Wu, Qi Wang, Ryan Julian, Danfei Xu, Yilun Du, Yevgen Chebotar, Scott Reed, Jan Kautz, Yuke Zhu, Linxi "Jim" Fan, Joel Jang
Computer Vision
EarthSpatialBench: Benchmarking Spatial Reasoning Capabilities of Multimodal LLMs on Earth Imagery
Zelin Xu, Yupu Zhang, Saugat Adhikari et al. (+6 more)
Benchmarking spatial
reasoning in
multimodal large
language models (MLLMs) has attracted growing interest in computer
vision due to its importance for
embodied AI and other agentic systems that require precise interaction with the physical world. However, spatial
reasoning on Earth imagery has lagged behind, as it uniquely involves grounding objects in georeferenced images and quantitatively
reasoning about distances, directions, and topological relations using both visual cues and vector geometry coordinates (e.g., 2D bounding boxes, polylines, and polygons). Existing
benchmarks for Earth imagery primarily focus on 2D spatial grounding, image captioning, and coarse spatial relations (e.g., simple directional or proximity cues). They lack support for quantitative direction and distance
reasoning, systematic topological relations, and complex object geometries beyond bounding boxes. To fill this gap, we propose EarthSpatialBench, a comprehensive
benchmark for evaluating spatial
reasoning in MLLMs on Earth imagery. The
benchmark contains over 325K question-answer pairs spanning: (1) qualitative and quantitative
reasoning about spatial distance and direction; (2) systematic topological relations; (3) single-object queries, object-pair queries, and compositional aggregate group queries; and (4) object references expressed via textual descriptions, visual overlays, and explicit geometry coordinates, including 2D bounding boxes, polylines, and polygons. We conducted extensive experiments on both open-source and proprietary models to identify limitations in the spatial
reasoning of MLLMs.
Read the full abstract ▾
All authors Zelin Xu, Yupu Zhang, Saugat Adhikari, Saiful Islam, Tingsong Xiao, Zibo Liu, Shigang Chen, Da Yan, Zhe Jiang
Computer Vision · Cross-listed from eess.IV
ROIX-Comp: Optimizing X-ray Computed Tomography Imaging Strategy for Data Reduction and Reconstruction
Amarjit Singh, Kento Sato, Kohei Yoshida et al. (+4 more)
In high-performance computing (HPC) environments, particularly in synchrotron radiation facilities, vast amounts of X-ray images are generated. Processing large-scale X-ray Computed
Tomography (X-CT)
datasets presents significant computational and storage challenges due to their high dimensionality and data volume. Traditional approaches often require extensive storage capacity and high transmission bandwidth, limiting real-time processing capabilities and workflow efficiency. To address these constraints, we introduce a region-of-interest (ROI)-driven extraction framework (ROIX-Comp) that intelligently compresses X-CT data by identifying and retaining only essential features. Our work reduces data volume while preserving critical information for downstream processing tasks. At pre-processing stage, we utilize error-bounded quantization to reduce the amount of data to be processed and therefore improve computational efficiencies. At the compression stage, our methodology combines object extraction with multiple state-of-the-art lossless and lossy compressors, resulting in significantly improved compression ratios. We evaluated this framework against seven X-CT
datasets and observed a relative compression ratio improvement of 12.34x compared to the standard compression.
Read the full abstract ▾
All authors Amarjit Singh, Kento Sato, Kohei Yoshida, Kentaro Uesugi, Yasumasa Joti, Takaki Hatsui, Andrès Rubio Proaño
Computer Vision
MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering
Xianwei Mao, Kai Ye, Sheng Zhou et al. (+4 more)
Knowledge-based Visual Question Answering (KB-VQA) requires models to answer questions by integrating visual information with external knowledge. However, retrieved knowledge is often noisy, partially irrelevant, or misaligned with the visual content, while internal model knowledge is difficult to control and interpret. Naive aggregation of these sources limits
reasoning effectiveness and reduces answer accuracy. To address this, we propose MaS-VQA, a selection-driven framework that tightly couples explicit knowledge filtering with implicit knowledge
reasoning. MaS-VQA first retrieves candidate passages and applies a Mask-and-Select mechanism to jointly prune irrelevant image regions and weakly relevant knowledge fragments, producing compact, high-signal
multimodal knowledge . This filtered knowledge then guides the activation of internal knowledge in a constrained semantic space, enabling complementary co-modeling of explicit and implicit knowledge for robust answer prediction. Experiments on Encyclopedic-VQA and InfoSeek demonstrate consistent performance gains across multiple MLLM backbones, and ablations verify that the selection mechanism effectively reduces noise and enhances knowledge utilization.
Read the full abstract ▾
All authors Xianwei Mao, Kai Ye, Sheng Zhou, Nan Zhang, Haikuan Huang, Bin Li, Jiajun Bu
Computer Vision · Cross-listed from eess.IV
Foundation Models for Medical Imaging: Status, Challenges, and Directions
Chuang Niu, Pengwei Wu, Bruno De Man et al. (+1 more)
Foundation models (FMs) are rapidly reshaping medical imaging, shifting the field from narrowly trained, task-specific networks toward large, general-purpose models that can be adapted across modalities, anatomies, and clinical tasks. In this review, we synthesize the emerging landscape of medical imaging FMs along three major axes: principles of FM design, applications of FMs, and forward-looking challenges and opportunities. Taken together, this review provides a technically grounded, clinically aware, and future-facing roadmap for developing FMs that are not only powerful and versatile but also trustworthy and ready for responsible translation into clinical practice.
Read the full abstract ▾
All authors Chuang Niu, Pengwei Wu, Bruno De Man, Ge Wang
Computer Vision · Cross-listed from cs.RO
Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulation
Yuxuan Kuang, Sungjae Park, Katerina Fragkiadaki et al. (+1 more)
Learning generalist policies capable of accomplishing a wide range of everyday tasks remains an open challenge in dexterous manipulation. In particular, collecting large-scale manipulation data via real-world teleoperation is expensive and difficult to scale. While learning in simulation provides a feasible alternative, designing multiple task-specific environments and rewards for training is similarly challenging. We propose Dex4D, a framework that instead leverages simulation for learning task-agnostic dexterous skills that can be flexibly recomposed to perform diverse real-world manipulation tasks. Specifically, Dex4D learns a domain-agnostic 3D point track conditioned policy capable of transforming a rigid object from a current pose to a target pose. We train this 'Anypose-to-Anypose' policy in simulation across thousands of objects with diverse pose configurations, covering a broad space of
robot-object interactions that can be composed at test time. At deployment, this policy can be zero-shot transferred to real-world tasks without finetuning, simply by prompting it with desired object-centric point tracks extracted from generated
videos. During execution, Dex4D uses online point tracking for closed-loop perception and control. Extensive experiments in simulation and on real
robots show that our method enables zero-shot deployment for diverse dexterous manipulation tasks and yields consistent improvements over prior baselines. Furthermore, we demonstrate strong generalization to novel objects, scene layouts, backgrounds, and trajectories, highlighting the robustness and scalability of the proposed framework. Project page: https://dex4d.github.io/.
Read the full abstract ▾
All authors Yuxuan Kuang, Sungjae Park, Katerina Fragkiadaki, Shubham Tulsiani
Computer Vision
VideoSketcher: Sequential Sketch Generation Using Video Model Priors
Hui Ren, Yuval Alaluf, Omer Bar Tal et al. (+3 more)
Sketching is inherently sequential: strokes are drawn progressively to explore and refine ideas. Yet most generative approaches treat sketches as static images, ignoring the temporal process underlying creative exploration. Modeling this sequential structure remains challenging: prior methods either rely on large-scale human-drawn
datasets with limited diversity, or use large
language models (LLMs) to produce drawing instructions, often at the cost of visual fidelity. We present VideoSketcher, a method for generating high-quality sketching processes by adapting pretrained text-to-
video diffusion models to the sparse, continuous nature of sketch formation. Our key insight is that LLMs and
video diffusion models offer complementary strengths: LLMs act as semantic planners that decompose concepts into step-by-step instructions, while
video diffusion models serve as powerful "renderers" that translate them into temporally coherent sketch sequences. We introduce a two-stage fine-tuning strategy that decouples temporal structure from visual appearance: stroke ordering is learned from synthetic shape compositions, while style is distilled from as few as seven hand-drawn examples. Despite minimal supervision, our method can generate diverse, high-quality sequential sketches that faithfully follow specified drawing orders. Our framework naturally extends to brush style control and autoregressive generation, supporting artistic applications.
Read the full abstract ▾
All authors Hui Ren, Yuval Alaluf, Omer Bar Tal, Alexander Schwing, Antonio Torralba, Yael Vinker
Computer Vision
CARL-CXR: Continual Adapter-Based Routing for Task-Unknown Chest Radiograph Classification
Muthu Subash Kavitha, Anas Zafar, Amgad Muneer et al. (+1 more)
Clinical deployment of chest radiograph classifiers requires models that can be updated as new
datasets become available without retraining on previously observed data or degrading validated performance. We study a task-incremental continual learning setting for chest radiograph classification under task-unknown inference, where heterogeneous chest X-ray
datasets arrive sequentially and task identity is unavailable at deployment time. We propose CARL-CXR, a continual adapter-based routing framework that maintains a fixed high-capacity backbone while incrementally introducing lightweight task-specific adapters and classifier heads. A latent task selector operates on adapter-conditioned features to dynamically route each input to the most relevant task pathway, leveraging compact task prototypes and feature-level experience replay to preserve task identity across sequential updates without storing raw images. Experiments on MIMIC-CXR and CheXpert two large-scale
datasets with distinct patient populations, imaging devices, and annotation pipelines demonstrate that CARL-CXR achieves minimal catastrophic forgetting (0.012 AUROC drop), representing a 6X and 11X reduction over established continual learning baselines LwF and EWC respectively, while maintaining competitive diagnostic performance (AUROC 0.74). Under task unknown deployment, CARL-CXR outperforms joint training by 12.5 points in routing accuracy (75.0% vs. 62.5%): unlike LwF and EWC, which require explicit task identifiers at inference and provide no routing mechanism.
Read the full abstract ▾
All authors Muthu Subash Kavitha, Anas Zafar, Amgad Muneer, Jia Wu
Computer Vision
Context-aware Skin Cancer Epithelial Cell Classification with Scalable Graph Transformers
Lucas Sancéré, Noémie Moreau and Katarzyna Bozek
Whole-slide images (WSIs) from cancer patients contain rich information that can be used for medical diagnosis or to follow treatment progress. To automate their analysis, numerous deep learning methods based on convolutional neural networks and
Vision Transformers have been developed and have achieved strong performance in
segmentation and classification tasks. However, due to the large size and complex cellular organization of WSIs, these models rely on patch-based representations, losing vital tissue-level context. We propose using scalable Graph
Transformers on a full-WSI
cell graph for classification. We evaluate this methodology on a challenging task: the classification of healthy versus tumor epithelial
cells in cutaneous squamous
cell carcinoma (cSCC), where both
cell types exhibit very similar morphologies and are therefore difficult to differentiate for image-based approaches. We first compared image-based and graph-based methods on a single WSI. Graph
Transformer models SGFormer and DIFFormer achieved balanced accuracies of $85.2 \pm 1.5$ ($\pm$ standard error) and $85.1 \pm 2.5$ in 3-fold cross-validation, respectively, whereas the best image-based method reached $81.2 \pm 3.0$. By evaluating several node feature configurations, we found that the most informative representation combined morphological and texture features as well as the
cell classes of non-epithelial
cells, highlighting the importance of the surrounding cellular context. We then extended our work to train on several WSIs from several patients. To address the computational constraints of image-based models, we extracted four $2560 \times 2560$ pixel patches from each image and converted them into graphs. In this setting, DIFFormer achieved a balanced accuracy of $83.6 \pm 1.9$ (3-fold cross-validation), while the state-of-the-art image-based model CellViT256 reached $78.1 \pm 0.5$.
Read the full abstract ▾
Computer Vision
Meteorological data and Sky Images meets Neural Models for Photovoltaic Power Forecasting
Ines Montoya-Espinagosa and Antonio Agudo
Due to the rise in the use of renewable energies as an alternative to traditional ones, and especially
solar energy, there is increasing interest in studying how to address photovoltaic
forecasting in the face of the challenge of variability in photovoltaic energy production, using different methodologies. This work develops a hybrid approach for short and long-term
forecasting based on two studies with the same purpose. A
multimodal approach that combines images of the sky and photovoltaic energy history with meteorological data is proposed. The main goal is to improve the accuracy of ramp event prediction, increase the robustness of forecasts in cloudy conditions, and extend capabilities beyond nowcasting, to support more efficient operation of the power grid and better management of
solar variability. Deep neural models are used for both nowcasting and
forecasting solutions, incorporating individual and multiple meteorological variables, as well as an analytical
solar position. The results demonstrate that the inclusion of meteorological data, particularly the surface long-wave, radiation downwards, and the combination of wind and
solar position, significantly improves current predictions in both nowcasting and
forecasting tasks, especially on cloudy days. This study highlights the importance of integrating diverse data sources to improve the reliability and interpretability of
solar energy prediction models.
Read the full abstract ▾
Computer Vision
NeRFscopy: Neural Radiance Fields for in-vivo Time-Varying Tissues from Endoscopy
Laura Salort-Benejam and Antonio Agudo
Endoscopy is essential in medical imaging, used for diagnosis, prognosis and treatment. Developing a robust dynamic 3D reconstruction pipeline for endoscopic
videos could enhance visualization, improve diagnostic accuracy, aid in treatment planning, and guide surgery procedures. However, challenges arise due to the deformable nature of the tissues, the use of
monocular cameras, illumination changes, occlusions and unknown camera trajectories. Inspired by neural rendering, we introduce NeRFscopy, a self-supervised pipeline for novel view synthesis and 3D reconstruction of deformable endoscopic tissues from a
monocular video. NeRFscopy includes a deformable model with a canonical radiance field and a time-dependent deformation field parameterized by SE(3) transformations. In addition, the color images are efficiently exploited by introducing sophisticated terms to learn a 3D implicit model without assuming any template or pre-trained model, solely from data. NeRFscopy achieves accurate results in terms of novel view synthesis, outperforming competing methods across various challenging endoscopy scenes.
Read the full abstract ▾
Computer Vision
Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models
Sen Ye, Mengde Xu, Shuyang Gu et al. (+3 more)
Current research in
multimodal models faces a key challenge where enhancing generative capabilities often comes at the expense of understanding, and vice versa. We analyzed this trade-off and identify the primary cause might be the potential conflict between generation and understanding, which creates a competitive dynamic within the model. To address this, we propose the Reason-Reflect-Refine (R3) framework. This innovative
algorithm re-frames the single-step generation task into a multi-step process of "generate-understand-regenerate". By explicitly leveraging the model's understanding capability during generation, we successfully mitigate the optimization dilemma, achieved stronger generation results and improved understanding ability which are related to the generation process. This offers valuable insights for designing next-generation unified
multimodal models. Code is available at https://github.com/sen-ye/R3.
Read the full abstract ▾
All authors Sen Ye, Mengde Xu, Shuyang Gu, Di He, Liwei Wang, Han Hu