Computer Vision
Motion Semantics Guided Normalizing Flow for Privacy-Preserving Video Anomaly Detection
Yang Liu, Boan Chen, Yuanyuan Meng et al. (+5 more)
As
embodied perception systems increasingly bridge digital and physical realms in interactive multimedia applications, the need for
privacy-preserving approaches to understand human activities in physical environments has become paramount.
Video anomaly detection is a critical task in such
embodied multimedia systems for intelligent surveillance and forensic analysis. Skeleton-based approaches have emerged as a
privacy-preserving alternative that processes physical world information through abstract human pose representations while discarding sensitive visual attributes such as identity and facial features. However, existing skeleton-based methods predominantly model continuous motion trajectories in a monolithic manner, failing to capture the hierarchical nature of human activities composed of discrete semantic primitives and fine-grained kinematic details, which leads to reduced discriminability when anomalies manifest at different abstraction levels. In this regard, we propose Motion Semantics Guided Normalizing Flow (MSG-Flow) that decomposes skeleton-based VAD into hierarchical motion semantics modeling. It employs vector quantized variational auto-encoder to discretize continuous motion into interpretable primitives, an autoregressive
Transformer to model semantic-level temporal dependencies, and a conditional normalizing flow to capture detail-level pose variations. Extensive experiments on
benchmarks (HR-ShanghaiTech & HR-UBnormal) demonstrate that MSG-Flow achieves state-of-the-art performance with 88.1% and 75.8% AUC respectively.
Read the full abstract ▾
All authors Yang Liu, Boan Chen, Yuanyuan Meng, Jing Liu, Zhengliang Guo, Wei Zhou, Peng Sun, Hong Chen
Computer Vision
CNMBI: Determining the Number of Clusters Using Center Pairwise Matching and Boundary Filtering
Ruilin Zhang, Haiyang Zheng and Hongpeng Wang
One of the main challenges in data mining is choosing the optimal number of clusters without prior information. Notably, existing methods are usually in the philosophy of cluster validation and hence have underlying assumptions on data distribution, which prevents their application to complex data such as large-scale images and high-dimensional data from the real world. In this regard, we propose an approach named CNMBI. Leveraging the distribution information inherent in the data space, we map the target task as a dynamic comparison process between cluster centers regarding positional behavior, without relying on the complete clustering results and designing the complex validity index as before. Bipartite graph theory is then employed to efficiently model this process. Additionally, we find that different samples have different confidence levels and thereby actively remove low-confidence ones, which is, for the first time to our knowledge, considered in cluster number determination. CNMBI is robust and allows for more flexibility in the dimension and shape of the target data (e.g., CIFAR-10 and STL-10). Extensive comparison studies with state-of-the-art competitors on various challenging
datasets demonstrate the superiority of our method.
Read the full abstract ▾
Computer Vision
Steering Sparse Autoencoder Latents to Control Dynamic Head Pruning in Vision Transformers (Student Abstract)
Yousung Lee and Dongsoo Har
Dynamic head
pruning in
Vision Transformers (ViTs) improves efficiency by removing redundant
attention heads, but existing
pruning policies are often difficult to interpret and control. In this work, we propose a novel framework by integrating Sparse Autoencoders (SAEs) with dynamic
pruning, leveraging their ability to disentangle dense
embeddings into interpretable and controllable sparse latents. Specifically, we train an SAE on the final-layer residual
embedding of the ViT and amplify the sparse latents with different strategies to alter
pruning decisions. Among them, per-class steering reveals compact, class-specific head subsets that preserve accuracy. For example, bowl improves accuracy (76% to 82%) while reducing head usage (0.72 to 0.33) via heads h2 and h5. These results show that sparse latent features enable class-specific control of dynamic
pruning, effectively bridging
pruning efficiency and mechanistic interpretability in ViTs.
Read the full abstract ▾
Computer Vision
Language-Conditioned World Modeling for Visual Navigation
Yifei Dong, Fengyi Wu, Yilong Dai et al. (+8 more)
Goal-conditioned visual
navigation has been a long-standing testbed for
embodied AI. We study a natural
language-conditioned variant,
language-conditioned visual
navigation (LCVN), in which an
embodied agent must follow a natural
language instruction given only an initial
egocentric observation. Without access to goal images, the
agent must rely on
language to shape its perception and continuous control. We introduce the LCVN
Dataset, a
benchmark of 39,016 trajectories and 117,048 human-verified instructions spanning diverse environments and instruction styles. Building on this
benchmark, we study two complementary paradigms: (i) latent-imagination policy learning, in which a
diffusion-based world model (LCVN-WM) imagines future observations and an actor-critic
agent (LCVN-AC) learns its policy entirely within the imagined latent space; and (ii) unified autoregressive prediction, in which a single
multimodal backbone (LCVN-Uni) jointly predicts actions and observations in one forward pass over a shared
token sequence. Experiments show that two paradigms offer complementary strengths: latent imagination produces more temporally coherent rollouts, whereas unified prediction generalizes better to unseen environments. Targeted ablations further isolate the contributions of
language guidance, conditioning signals, and instruction style, clarifying when
language grounding versus dynamics modeling is the performance bottleneck. Together, these findings position LCVN as a testbed for studying how
language, imagination, and decision-making interact in
embodied agents.
Read the full abstract ▾
All authors Yifei Dong, Fengyi Wu, Yilong Dai, Lingdong Kong, Guangyu Chen, Yetong Sha, Qiyu Hu, Feng Liu, Siyu Huang, Qi Dai, Zhi-Qi Cheng
Computer Vision · Cross-listed from cs.CR
CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training
Yuxi Chen, Haoyu Zhai, Chenkai Wang et al. (+4 more)
GUI
agents are rapidly shifting from multi-module pipelines to end-to-end, native
vision-
language models (VLMs) that perceive raw screenshots and directly interact with digital devices. Despite rapid progress on general GUI tasks, CAPTCHA solving remains a major challenge. On the other hand, although specialized CAPTCHA solving pipelines exist, they cannot handle general GUI tasks. To address this gap, we introduce ReCAP: a CAPTCHA-capable native GUI
agent that solves modern, interactive CAPTCHA challenges while retaining general GUI-
agent performance. We first develop a dynamic CAPTCHA system spanning seven representative CAPTCHA types, designed to stress primitive and complementary capabilities for CAPTCHA solving. Then, we develop an automated data collection and curation pipeline that generates large-scale CAPTCHA interaction trajectories paired with
reasoning traces. As CAPTCHA solving often requires multi-step interaction and recovery from intermediate mistakes, we further leverage failed trajectories to construct self-correction data, training
agents to reflect on errors and correct their actions online. Across synthetic and real-world test sets, ReCAP substantially improves CAPTCHA-solving success over its base
agents, while maintaining strong performance on general GUI-
agent benchmarks.
Read the full abstract ▾
All authors Yuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, Huan Zhang
Computer Vision
Drop-In Perceptual Optimization for 3D Gaussian Splatting
Ezgi Ozyilkan, Zhiqi Chen, Oren Rippel et al. (+2 more)
Despite their output being ultimately consumed by human viewers, 3D Gaussian Splatting (3DGS) methods often rely on ad-hoc combinations of pixel-level losses, resulting in blurry renderings. To address this, we systematically explore perceptual optimization strategies for 3DGS by searching over a diverse set of distortion losses. We conduct the first-of-its-kind large-scale human subjective study on 3DGS, involving 39,320 pairwise ratings across several
datasets and 3DGS frameworks. A regularized version of Wasserstein Distortion, which we call WD-R, emerges as the clear winner, excelling at recovering fine textures without incurring a higher splat count. WD-R is preferred by raters more than $2.3\times$ over the original 3DGS loss, and $1.5\times$ over the current best method Perceptual-GS. WD-R also consistently achieves state-of-the-art LPIPS, DISTS, and FID scores across various
datasets, and generalizes across recent frameworks, such as Mip-Splatting and Scaffold-GS, where replacing the original loss with WD-R consistently enhances perceptual quality within a similar resource budget (number of splats for Mip-Splatting, model size for Scaffold-GS), and leads to reconstructions being preferred by human raters $1.8\times$ and $3.6\times$, respectively. We also find that this carries over to the task of 3DGS scene compression, with $\approx 50\%$ bitrate savings for comparable perceptual metric performance.
Read the full abstract ▾
All authors Ezgi Ozyilkan, Zhiqi Chen, Oren Rippel, Jona Ballé, Kedar Tatwawadi
Computer Vision
MAGICIAN: Efficient Long-Term Planning with Imagined Gaussians for Active Mapping
Shiyao Li, Antoine Guédon, Shizhe Chen et al. (+1 more)
Active mapping aims to determine how an
agent should move to efficiently reconstruct unknown environments. Most existing approaches rely on greedy next-best-view prediction, resulting in inefficient exploration and incomplete reconstruction. To address this, we introduce MAGICIAN, a novel long-term planning framework that maximizes accumulated surface coverage gain through Imagined Gaussians, a scene representation based on 3D Gaussian Splatting, derived from a pre-trained occupancy network with strong structural priors. This representation enables efficient coverage gain computation for any novel viewpoint via fast volumetric rendering, allowing its integration into a tree-search
algorithm for long-horizon planning. We update Imagined Gaussians and refine the trajectory in a closed loop. Our method achieves state-of-the-art performance across indoor and outdoor
benchmarks with varying action spaces, highlighting the advantage of long-term planning in active mapping.
Read the full abstract ▾
All authors Shiyao Li, Antoine Guédon, Shizhe Chen, Vincent Lepetit
Computer Vision
Pretext Matters: An Empirical Study of SSL Methods in Medical Imaging
Vedrana Ivezić, Mara Pleasure, Ashwath Radhachandran et al. (+7 more)
Though self-supervised learning (SSL) has demonstrated incredible ability to learn robust representations from unlabeled data, the choice of optimal SSL strategy can lead to vastly different performance outcomes in specialized domains. Joint
embedding architectures (JEAs) and joint
embedding predictive architectures (JEPAs) have shown robustness to noise and strong semantic feature learning compared to pixel reconstruction-based SSL methods, leading to widespread adoption in medical imaging. However, no prior work has systematically investigated which SSL objective is better aligned with the spatial organization of clinically relevant signal. In this work, we empirically investigate how the choice of SSL method impacts the learned representations in medical imaging. We select two representative imaging modalities characterized by unique noise profiles: ultrasound and histopathology. When informative signal is spatially localized, as in histopathology, JEAs are more effective due to their view-invariance objective. In contrast, when diagnostically relevant information is globally structured, such as the macroscopic anatomy present in liver ultrasounds, JEPAs are optimal. These differences are especially evident in the clinical relevance of the learned features, as independently validated by board-certified radiologists and pathologists. Together, our results provide a framework for matching SSL objectives to the structural and noise properties of medical imaging modalities.
Read the full abstract ▾
All authors Vedrana Ivezić, Mara Pleasure, Ashwath Radhachandran, Saarang Panchavati, Shreeram Athreya, Vivek Sant, Benjamin Emert, Gregory Fishbein, Corey Arnold, William Speier
Computer Vision
Q-Tacit: Image Quality Assessment via Latent Visual Reasoning
Yuxuan Jiang, Yixuan Li, Hanwei Zhu et al. (+3 more)
Vision-
Language Model (VLM)-based image quality assessment (IQA) has been significantly advanced by incorporating Chain-of-Thought (CoT)
reasoning. Recent work has refined image quality
reasoning by applying reinforcement learning (RL) and leveraging active visual tools. However, such strategies are typically
language-centric, with visual information being treated as static preconditions. Quality-related visual cues often cannot be abstracted into text in extenso due to the gap between discrete textual
tokens and quality perception space, which in turn restricts the
reasoning effectiveness for visually intensive IQA tasks. In this paper, we revisit this by asking the question, "Is natural
language the ideal space for quality
reasoning?" and, as a consequence, we propose Q-Tacit, a new paradigm that elicits VLMs to reason beyond natural
language in the latent quality space. Our approach follows a synergistic two-stage process: (i) injecting structural visual quality priors into the latent space, and (ii) calibrating latent
reasoning trajectories to improve quality assessment ability. Extensive experiments demonstrate that Q-Tacit can effectively perform quality
reasoning with significantly fewer
tokens than previous
reasoning-based methods, while achieving strong overall performance. This paper validates the proposition that
language is not the only compact representation suitable for visual quality, opening possibilities for further exploration of effective latent
reasoning paradigms for IQA. Source code will be released to support future research.
Read the full abstract ▾
All authors Yuxuan Jiang, Yixuan Li, Hanwei Zhu, Siyue Teng, Fan Zhang, David Bull
Computer Vision
CAM3R: Camera-Agnostic Model for 3D Reconstruction
Namitha Guruprasad, Abhay Yadav, Cheng Peng et al. (+1 more)
Recovering dense 3D geometry from unposed images remains a foundational challenge in computer
vision. Current state-of-the-art models are predominantly trained on perspective
datasets, which implicitly constrains them to a standard pinhole camera geometry. As a result, these models suffer from significant
geometric degradation when applied to wide-angle imagery captured via non-rectilinear optics, such as fisheye or panoramic sensors. To address this, we present CAM3R, a Camera-Agnostic, feed-forward Model for 3D Reconstruction capable of processing images from wide-angle camera models without prior calibration. Our framework consists of a two-view network which is bifurcated into a Ray Module (RM) to estimate per-pixel ray directions and a Cross-view Module (CVM) to infer radial distance with confidence maps, pointmaps, and relative poses. To unify these pairwise predictions into a consistent 3D scene, we introduce a Ray-Aware Global
Alignment framework for pose refinement and scale optimization while strictly preserving the predicted local geometry. Extensive experiments on various camera model
datasets, including panorama, fisheye and pinhole imagery, demonstrate that CAM3R establishes a new state-of-the-art in pose estimation and reconstruction.
Read the full abstract ▾
All authors Namitha Guruprasad, Abhay Yadav, Cheng Peng, Rama Chellappa
Computer Vision · Cross-listed from eess.IV
Single-Subject Multi-View MRI Super-Resolution via Implicit Neural Representations
Heejong Kim, Abhishek Thanki, Roel van Herten et al. (+2 more)
Clinical MRI frequently acquires anisotropic volumes with high in-plane resolution and low through-plane resolution to reduce acquisition time. Multiple orientations are therefore acquired to provide complementary anatomical information. Conventional integration of these views relies on registration followed by interpolation, which can degrade fine structural details. Recent deep learning-based super-resolution (SR) approaches have demonstrated strong performance in enhancing single-view images. However, their clinical reliability is often limited by the need for large-scale training
datasets, resulting in increased dependence on cohort-level priors. Self-supervised strategies offer an alternative by learning directly from the target scans. Prior work either neglects the existence of multi-view information or assumes that in-plane information can supervise through-plane reconstruction under the assumption of pre-
alignment between images. However, this assumption is rarely satisfied in clinical settings. In this work, we introduce Single-Subject Implicit Multi-View Super-Resolution for MRI (SIMS-MRI), a framework that operates solely on anisotropic multi-view scans from a single patient without requiring pre- or post-processing. Our method combines a multi-resolution hash-encoded implicit representation with learned inter-view
alignment to generate a spatially consistent isotropic reconstruction. We validate the SIMS-MRI pipeline on both simulated
brain and clinical prostate MRI
datasets. Code will be made publicly available for reproducibility: https://github.com/abhshkt/SIMS-MRI
Read the full abstract ▾
All authors Heejong Kim, Abhishek Thanki, Roel van Herten, Daniel Margolis, Mert R Sabuncu
Computer Vision
PIVM: Diffusion-Based Prior-Integrated Variation Modeling for Anatomically Precise Abdominal CT Synthesis
Dinglun He, Baoming Zhang, Xu Wang et al. (+3 more)
Abdominal CT data are limited by high annotation costs and
privacy constraints, which hinder the development of robust
segmentation and diagnostic models. We present a Prior-Integrated Variation Modeling (PIVM) framework, a
diffusion-based method for anatomically accurate CT image synthesis. Instead of generating full images from noise, PIVM predicts voxel-wise intensity variations relative to organ-specific intensity priors derived from
segmentation labels. These priors and labels jointly guide the
diffusion process, ensuring spatial
alignment and realistic organ boundaries. Unlike latent-space
diffusion models, our approach operates directly in image space while preserving the full Hounsfield Unit (HU) range, capturing fine anatomical textures without smoothing. Source code is available at https://github.com/BZNR3/PIVM.
Read the full abstract ▾
All authors Dinglun He, Baoming Zhang, Xu Wang, Yao Hao, Deshan Yang, Ye Duan
Computer Vision
Toward Faithful Segmentation Attribution via Benchmarking and Dual-Evidence Fusion
Abu Noman Md Sakib, OFM Riaz Rahman Aranya, Kevin Desai et al. (+1 more)
Attribution maps for semantic
segmentation are almost always judged by visual plausibility. Yet looking convincing does not guarantee that the highlighted pixels actually drive the model's prediction, nor that attribution credit stays within the target region. These questions require a dedicated evaluation protocol. We introduce a reproducible
benchmark that tests intervention-based faithfulness, off-target leakage, perturbation robustness, and runtime on Pascal VOC and SBD across three pretrained backbones. To further demonstrate the
benchmark, we propose Dual-Evidence Attribution (DEA), a lightweight correction that fuses gradient evidence with region-level intervention signals through agreement-weighted fusion. DEA increases emphasis where both sources agree and retains causal support when gradient responses are unstable. Across all completed runs, DEA consistently improves deletion-based faithfulness over gradient-only baselines and preserves strong robustness, at the cost of additional compute from intervention passes. The
benchmark exposes a faithfulness-stability tradeoff among attribution families that is entirely hidden under visual evaluation, providing a foundation for principled method selection in
segmentation explainability. Code is available at https://github.com/anmspro/DEA.
Read the full abstract ▾
All authors Abu Noman Md Sakib, OFM Riaz Rahman Aranya, Kevin Desai, Zijie Zhang
Computer Vision
To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models
OFM Riaz Rahman Aranya and Kevin Desai
Vision-
language models (VLMs) adapted to the medical domain have shown strong performance on visual question answering
benchmarks, yet their robustness against two critical failure modes,
hallucination and sycophancy, remains poorly understood, particularly in combination. We evaluate six VLMs (three general-purpose, three medical-specialist) on three medical VQA
datasets and uncover a grounding-sycophancy tradeoff: models with the lowest
hallucination propensity are the most sycophantic, while the most pressure-resistant model hallucinates more than all medical-specialist models. To characterize this tradeoff, we propose three metrics: L-VASE, a logit-space reformulation of VASE that avoids its double-normalization; CCS, a confidence-calibrated sycophancy score that penalizes high-confidence capitulation; and Clinical Safety Index (CSI), a unified safety index that combines grounding, autonomy, and calibration via a
geometric mean. Across 1,151 test cases, no model achieves a CSI above 0.35, indicating that none of the evaluated 7-8B parameter VLMs is simultaneously well-grounded and robust to social pressure. Our findings suggest that joint evaluation of both properties is necessary before these models can be considered for clinical use. Code is available at https://github.com/UTSA-VIRLab/AgreeOrRight
Read the full abstract ▾
Computer Vision
A Vision Language Model for Generating Procedural Plant Architecture Representations from Simulated Images
Heesup Yun, Isaac Kazuo Uyehara, Ioannis Droutsas et al. (+4 more)
Three-dimensional (3D) procedural plant architecture models have emerged as an important tool for simulation-based studies of plant structure and function, extracting plant architectural parameters from field measurements, and for generating realistic plants in computer graphics. However, measuring the architectural parameters and nested structures for these models at the field scales remains prohibitively labor-intensive. We present a novel
algorithm that generates a 3D plant architecture from an image, creating a functional structural plant model that reflects organ-level
geometric and topological parameters and provides a more comprehensive representation of the plant's architecture. Instead of using 3D sensors or processing multi-view images with computer
vision to obtain the 3D structure of plants, we proposed a method that generates
token sequences that encode a procedural definition of plant architecture. This work used only synthetic images for training and testing, with exact architectural parameters known, allowing testing of the hypothesis that organ-level architectural parameters could be extracted from image data using a
vision-
language model (VLM). A synthetic
dataset of cowpea plant images was generated using the Helios 3D plant simulator, with the detailed plant architecture encoded in XML files. We developed a plant architecture tokenizer for the XML file defining plant architecture, converting it into a
token sequence that a
language model can predict. The model achieved a
token F1 score of 0.73 during teacher-forced training. Evaluation of the model was performed through autoregressive generation, achieving a BLEU-4 score of 94.00% and a ROUGE-L score of 0.5182. This led to the conclusion that such plant architecture model generation and parameter extraction were possible from synthetic images; thus, future work will extend the approach to real imagery data.
Read the full abstract ▾
All authors Heesup Yun, Isaac Kazuo Uyehara, Ioannis Droutsas, Earl Ranario, Christine H. Diepenbrock, Brian N. Bailey, J. Mason Earles
Computer Vision
Dress-ED: Instruction-Guided Editing for Virtual Try-On and Try-Off
Davide Lobba, Fulvio Sanguigni, Bin Ren et al. (+3 more)
Recent advances in Virtual Try-On (VTON) and Virtual Try-Off (VTOFF) have greatly improved photo-realistic fashion synthesis and garment reconstruction. However, existing
datasets remain static, lacking instruction-driven editing for controllable and interactive fashion generation. In this work, we introduce the Dress Editing
Dataset (Dress-ED), the first large-scale
benchmark that unifies VTON, VTOFF, and text-guided garment editing within a single framework. Each sample in Dress-ED includes an in-shop garment image, the corresponding person image wearing the garment, their edited counterparts, and a natural-
language instruction of the desired modification. Built through a fully automated
multimodal pipeline that integrates MLLM-based garment understanding,
diffusion-based editing, and LLM-guided verification, Dress-ED comprises over 146k verified quadruplets spanning three garment categories and seven edit types, including both appearance (e.g., color, pattern, material) and structural (e.g., sleeve length, neckline) modifications. Based on this
benchmark, we further propose a unified
multimodal diffusion framework that jointly reasons over linguistic instructions and visual garment cues, serving as a strong baseline for instruction-driven VTON and VTOFF.
Dataset and code available at this link: https://github.com/aimagelab/Dress-ED
Read the full abstract ▾
All authors Davide Lobba, Fulvio Sanguigni, Bin Ren, Marcella Cornia, Rita Cucchiara, Nicu Sebe
Computer Vision
TrajLoom: Dense Future Trajectory Generation from Video
Zewei Zhang, Jia Jun Cheng Xian, Kaiwen Liu et al. (+4 more)
Predicting future motion is crucial in
video understanding and controllable
video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future
evolution from observed
video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and
video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified
benchmark spanning real and synthetic
videos with a standardized setup aligned with
video-generation
benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across
datasets. The predicted trajectories directly support downstream
video generation and editing. Code, model checkpoints, and
datasets are available at https://trajloom.github.io/.
Read the full abstract ▾
All authors Zewei Zhang, Jia Jun Cheng Xian, Kaiwen Liu, Ming Liang, Hang Chu, Jun Chen, Renjie Liao
Computer Vision
Language Models Can Explain Visual Features via Steering
Javier Ferrando, Enrique Lopez-Cuena, Pablo Agustin Martin-Torres et al. (+3 more)
Sparse Autoencoders uncover thousands of features in
vision models, yet explaining these features without requiring human intervention remains an open challenge. While previous work has proposed generating correlation-based explanations based on top activating input examples, we present a fundamentally different alternative based on causal interventions. We leverage the structure of
Vision-
Language Models and steer individual SAE features in the
vision encoder after providing an empty image. Then, we
prompt the
language model to explain what it “sees”, effectively eliciting the visual concept represented by each feature. Results show that Steering offers an scalable alternative that complements traditional approaches based on input examples, serving as a new axis for automated interpretability in
vision models. Moreover, the quality of explanations improves consistently with the scale of the
language model, highlighting our method as a promising direction for future research. Finally, we propose Steering-informed Top-k, a hybrid approach that combines the strengths of causal interventions and input-based approaches to achieve state-of-the-art explanation quality without additional computational cost.
Read the full abstract ▾
All authors Javier Ferrando, Enrique Lopez-Cuena, Pablo Agustin Martin-Torres, Daniel Hinjos, Anna Arias-Duart, Dario Garcia-Gasulla
Computer Vision
A vision-language model and platform for temporally mapping surgery from video
Dani Kiyasseh
Mapping surgery is fundamental to developing operative guidelines and enabling autonomous robotic surgery. Recent advances in artificial intelligence (AI) have shown promise in mapping the behaviour of surgeons from
videos, yet current models remain narrow in scope, capturing limited behavioural components within single procedures, and offer limited translational value, as they remain inaccessible to practising surgeons. Here we introduce Halsted, a
vision-
language model trained on the Halsted
Surgical Atlas (HSA), one of the most comprehensive annotated
video libraries grown through an iterative self-labelling framework and encompassing over 650,000
videos across eight
surgical specialties. To facilitate benchmarking, we publicly release HSA-27k, a subset of the Halsted
Surgical Atlas. Halsted surpasses previous state-of-the-art models in mapping
surgical activity while offering greater comprehensiveness and computational efficiency. To bridge the longstanding translational gap of
surgical AI, we develop the Halsted web platform (https://halstedhealth.ai/) to provide surgeons anywhere in the world with the previously-unavailable capability of automatically mapping their own procedures within minutes. By standardizing unstructured
surgical video data and making these capabilities directly accessible to surgeons, our work brings
surgical AI closer to clinical deployment and helps pave the way toward autonomous robotic surgery.
Read the full abstract ▾
Computer Vision
FullCircle: Effortless 3D Reconstruction from Casual 360$^\circ$ Captures
Yalda Foroutan, Ipek Oztas, Daniel Rebain et al. (+4 more)
Radiance fields have emerged as powerful tools for 3D scene reconstruction. However, casual capture remains challenging due to the narrow field of view of perspective cameras, which limits viewpoint coverage and feature correspondences necessary for reliable camera calibration and reconstruction. While commercially available 360$^\circ$ cameras offer significantly broader coverage than perspective cameras for the same capture effort, existing 360$^\circ$ reconstruction methods require special capture protocols and pre-processing steps that undermine the promise of radiance fields: effortless workflows to capture and reconstruct 3D scenes. We propose a practical pipeline for reconstructing 3D scenes directly from raw 360$^\circ$ camera captures. We require no special capture protocols or pre-processing, and exhibit robustness to a prevalent source of reconstruction errors: the human operator that is visible in all 360$^\circ$ imagery. To facilitate evaluation, we introduce a multi-tiered
dataset of scenes captured as raw dual-fisheye images, establishing a
benchmark for robust casual 360$^\circ$ reconstruction. Our method significantly outperforms not only vanilla 3DGS for 360$^\circ$ cameras but also robust perspective baselines when perspective cameras are simulated from the same capture, demonstrating the advantages of 360$^\circ$ capture for casual reconstruction. Additional results are available at: https://theialab.github.io/fullcircle
Read the full abstract ▾
All authors Yalda Foroutan, Ipek Oztas, Daniel Rebain, Aysegul Dundar, Kwang Moo Yi, Lily Goli, Andrea Tagliasacchi
Computer Vision
CanViT: Toward Active-Vision Foundation Models
Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand et al. (+1 more)
Active computer
vision promises efficient, biologically plausible perception through sequential, localized glimpses, but lacks scalable general-purpose architectures and pretraining pipelines, leaving Active-
Vision Foundation Models (AVFMs) underexplored. We introduce CanViT, the first task- and policy-agnostic AVFM. CanViT uses scene-relative RoPE to bind a retinotopic
Vision Transformer backbone and a spatiotopic scene-wide latent workspace, the canvas. Efficient interaction with this high-capacity working
memory is supported by Canvas
Attention, a novel asymmetric cross-
attention mechanism. We decouple thinking (backbone-level) and
memory (canvas-level), eliminating canvas-side self-
attention and fully-connected layers to achieve fast sequential inference and scalability to high output resolutions. We propose a label-free active
vision pretraining scheme, policy-agnostic passive-to-active dense latent
distillation: reconstructing scene-wide DINOv3
embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths. We pretrain CanViT-B from a random initialization on 13.2 million ImageNet-21k scenes–an order of magnitude more than previous active models–and 1 billion random glimpses, in 166 hours on a single H100. On ADE20K
segmentation, a frozen CanViT-B achieves 38.5% mIoU in a single low-resolution glimpse, outperforming the best active model's 27.6% with 20x fewer inference FLOPs as well as its FLOP- or input-matched DINOv3 teacher. Given additional glimpses, CanViT-B reaches 45.9% ADE20K mIoU. On ImageNet-1k classification, CanViT-B also sets a new active-
vision state of the art, with 84.5% top-1 accuracy after fine-tuning. CanViT generalizes to longer rollouts, larger scenes, and new policies. Our work narrows the wide gap between passive and active computer
vision, demonstrating the potential of task- and policy-agnostic AVFM pretraining.
Read the full abstract ▾
All authors Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand, B. Suresh Krishna
Computer Vision
Generalized multi-object classification and tracking with sparse feature resonator networks
Lazar Supic, Alec Mullen and E. Paxon Frady
In visual scene understanding tasks, it is essential to capture both invariant and equivariant structure. While neural networks are frequently trained to achieve invariance to transformations such as translation, this often comes at the cost of losing access to equivariant information - e.g., the precise location of an object. Moreover, invariance is not naturally guaranteed through supervised learning alone, and many architectures generalize poorly to input transformations not encountered during training. Here, we take an approach based on analysis-by-synthesis and factoring using resonator networks. A generative model describes the construction of simple scenes containing MNIST digits and their transformations, like color and position. The resonator network inverts the generative model, and provides both invariant and equivariant information about particular objects. Sparse features learned from training data act as a basis set to provide flexibility in representing variable shapes of objects, allowing the resonator network to handle previously unseen digit shapes from the test set. The modular structure provides a shape module which contains information about the object shape with translation factored out, allowing a simple classifier to operate on centered digits. The classification layer is trained solely on centered data, requiring much less training data, and the network as a whole can identify objects with arbitrary translations without data augmentation. The natural
attention-like mechanism of the resonator network also allows for analysis of scenes with multiple objects, where the network dynamics selects and centers only one object at a time. Further, the specific position information of a particular object can be extracted from the translation module, and we show that the resonator can be designed to track multiple moving objects with precision of a few pixels.
Read the full abstract ▾
Computer Vision
UrbanVGGT: Scalable Sidewalk Width Estimation from Street View Images
Kaizhen Tan and Fan Zhang
Sidewalk width is an important indicator of pedestrian accessibility, comfort, and network quality, yet large-scale width data remain scarce in most cities. Existing approaches typically rely on costly field surveys, high-resolution overhead imagery, or simplified
geometric assumptions that limit scalability or introduce systematic error. To address this gap, we present UrbanVGGT, a measurement pipeline for estimating metric sidewalk width from a single street-view image. The method combines semantic
segmentation, feed-forward 3D reconstruction, adaptive ground-plane fitting, camera-height-based scale calibration, and directional width measurement on the recovered plane. On a ground-truth
benchmark from Washington, D.C., UrbanVGGT achieves a mean absolute error of 0.252 m, with 95.5% of estimates within 0.50 m of the reference width. Ablation experiments show that metric scale calibration is the most critical component, and controlled comparisons with alternative geometry backbones support the effectiveness of the overall design. As a feasibility demonstration, we further apply the pipeline to three cities and generate SV-SideWidth, a prototype sidewalk-width
dataset covering 527 OpenStreetMap street segments. The results indicate that street-view imagery can support scalable generation of candidate sidewalk-width attributes, while broader cross-city validation and local ground-truth
auditing remain necessary before deployment as authoritative planning data.
Read the full abstract ▾
Computer Vision
Shoubin Yu, Lei Shu, Antoine Yang et al. (+6 more)
Multimodal AI
agents are increasingly automating complex real-world workflows that involve online web execution. However, current web-
agent benchmarks suffer from a critical limitation: they focus entirely on web-based interaction and perception, lacking grounding in the user's real-world physical surroundings. This limitation prevents evaluation in crucial scenarios, such as when an
agent must use
egocentric visual perception (e.g., via AR glasses) to recognize an object in the user's surroundings and then complete a related task online. To address this gap, we introduce Ego2Web, the first
benchmark designed to bridge
egocentric video perception and web
agent execution. Ego2Web pairs real-world first-person
video recordings with web tasks that require visual understanding, web task planning, and interaction in an online environment for successful completion. We utilize an automatic data-generation pipeline combined with human verification and refinement to curate well-constructed, high-quality
video-task pairs across diverse web task types, including e-commerce, media
retrieval, knowledge lookup, etc. To facilitate accurate and scalable evaluation for our
benchmark, we also develop a novel LLM-as-a-Judge automatic evaluation method, Ego2WebJudge, which achieves approximately 84% agreement with human judgment, substantially higher than existing evaluation methods. Experiments with diverse SoTA
agents on our Ego2Web show that their performance is weak, with substantial headroom across all task categories. We also conduct a comprehensive ablation study on task design, highlighting the necessity of accurate
video understanding in the proposed task and the limitations of current
agents. We hope Ego2Web can be a critical new resource for developing truly capable AI assistants that can seamlessly see, understand, and act across the physical and digital worlds.
Read the full abstract ▾
All authors Shoubin Yu, Lei Shu, Antoine Yang, Yao Fu, Srinivas Sunkara, Maria Wang, Jindong Chen, Mohit Bansal, Boqing Gong
Computer Vision · Cross-listed from cs.RO
Learning Sidewalk Autopilot from Multi-Scale Imitation with Corrective Behavior Expansion
Honglin He, Yukai Ma, Brad Squicciarini et al. (+2 more)
Sidewalk micromobility is a promising solution for last-mile transportation, but current learning-based control methods struggle in complex urban environments. Imitation learning (IL) learns policies from human demonstrations, yet its reliance on fixed offline data often leads to compounding errors, limited robustness, and poor generalization. To address these challenges, we propose a framework that advances IL through corrective behavior expansion and multi-scale imitation learning. On the data side, we augment teleoperation
datasets with diverse corrective behaviors and sensor augmentations to enable the policy to learn to recover from its own mistakes. On the model side, we introduce a multi-scale IL architecture that captures both short-horizon interactive behaviors and long-horizon goal-directed intentions via horizon-based trajectory clustering and hierarchical supervision. Real-world experiments show that our approach significantly improves robustness and generalization in diverse sidewalk scenarios.
Read the full abstract ▾
All authors Honglin He, Yukai Ma, Brad Squicciarini, Wayne Wu, Bolei Zhou
Computer Vision
High Resolution Flood Extent Detection Using Deep Learning with Random Forest Derived Training Labels
Azizbek Nuriddinov, Ebrahim Ahmadisharaf and Mohammad Reza Alizadeh
Validation of flood models, used to support risk mitigation strategies, remains challenging due to limited observations during extreme events. High-frequency, high-resolution optical imagery (~3 m), such as PlanetScope, offers new opportunities for flood mapping, although applications remain limited by cloud cover and the lack of labeled training data during disasters. To address this, we develop a flood mapping framework that integrates PlanetScope optical imagery with topographic features using machine learning (ML) and deep learning (DL)
algorithms. A Random Forest model was applied to expert-annotated flood masks to generate training labels for DL models, U-Net. Two U-Net models with ResNet18 backbone were trained using optical imagery only (4 bands) and optical imagery combined with Height Above Nearest Drainage (HAND) and topographic slope (6 bands). Hurricane Ida (September 2021), which caused catastrophic flooding across the eastern United States, including the New York City metropolitan area, was used as an example to evaluate the framework. Results demonstrate that the U-Net model with topographic features achieved very close performance to the optical-only configuration (F1=0.92 and IoU=0.85 by both modeling scenarios), indicating that HAND and slope provide only marginal value to inundation extent detection. The proposed framework offers a scalable and label-efficient approach for mapping inundation extent that enables modeling under data-scarce flood scenarios.
Read the full abstract ▾
Computer Vision
Sketch2CT: Multimodal Diffusion for Structure-Aware 3D Medical Volume Generation
Delin An and Chaoli Wang
Diffusion probabilistic models have demonstrated significant potential in generating high-quality, realistic medical images, providing a promising solution to the persistent challenge of data scarcity in the medical field. Nevertheless, producing 3D medical volumes with anatomically consistent structures under
multimodal conditions remains a complex and unresolved problem. We introduce Sketch2CT, a
multimodal diffusion framework for structure-aware 3D medical volume generation, jointly guided by a user-provided 2D sketch and a textual description that captures 3D
geometric semantics. The framework initially generates 3D
segmentation masks of the target organ from random noise, conditioned on both modalities. To effectively align and fuse these inputs, we propose two key modules that refine sketch features with localized textual cues and integrate global sketch-text representations. Built upon a capsule-
attention backbone, these modules leverage the complementary strengths of sketches and text to produce anatomically accurate organ shapes. The synthesized
segmentation masks subsequently guide a latent
diffusion model for 3D CT volume synthesis, enabling realistic reconstruction of organ appearances that are consistent with user-defined sketches and descriptions. Extensive experiments on public CT
datasets demonstrate that Sketch2CT achieves superior performance in generating
multimodal medical volumes. Its controllable, low-cost generation pipeline enables principled, efficient augmentation of medical
datasets. Code is available at https://github.com/adlsn/Sketch2CT.
Read the full abstract ▾
Computer Vision
Tiny Inference-Time Scaling with Latent Verifiers
Davide Bucciarelli, Evelyn Turri, Lorenzo Baraldi et al. (+3 more)
Inference-time scaling has emerged as an effective way to improve generative models at test time by using a verifier to score and select candidate outputs. A common choice is to employ
Multimodal Large
Language Models (MLLMs) as verifiers, which can improve performance but introduce substantial inference-time cost. Indeed,
diffusion pipelines operate in an autoencoder latent space to reduce computation, yet MLLM verifiers still require decoding candidates to pixel space and re-encoding them into the visual
embedding space, leading to redundant and costly operations. In this work, we propose Verifier on Hidden States (VHS), a verifier that operates directly on intermediate hidden representations of
Diffusion Transformer (DiT) single-step generators. VHS analyzes generator features without decoding to pixel space, thereby reducing the per-candidate verification cost while improving or matching the performance of MLLM-based competitors. We show that, under tiny inference budgets with only a small number of candidates per
prompt, VHS enables more efficient inference-time scaling reducing joint generation-and-verification time by 63.3%, compute FLOPs by 51% and VRAM usage by 14.5% with respect to a standard MLLM verifier, achieving a +2.7% improvement on GenEval at the same inference-time budget.
Read the full abstract ▾
All authors Davide Bucciarelli, Evelyn Turri, Lorenzo Baraldi, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Computer Vision
Color When It Counts: Grayscale-Guided Online Triggering for Always-On Streaming Video Sensing
Weitong Cai, Hang Zhang, Yukai Huang et al. (+5 more)
Always-on sensing is essential for next-generation edge/wearable AI systems, yet continuous high-fidelity RGB
video capture remains prohibitively expensive for resource-constrained mobile and edge platforms. We present a new paradigm for efficient streaming
video understanding: grayscale-always, color-on-demand. Through preliminary studies, we discover that color is not always necessary. Sparse RGB frames suffice for comparable performance when temporal structure is preserved via continuous grayscale streams. Building on this insight, we propose ColorTrigger, an online training-free trigger that selectively activates color capture based on windowed grayscale affinity analysis. Designed for real-time edge deployment, ColorTrigger uses lightweight quadratic programming to detect chromatic redundancy causally, coupled with credit-budgeted control and dynamic
token routing to jointly reduce sensing and inference costs. On streaming
video understanding
benchmarks, ColorTrigger achieves 91.6% of full-color baseline performance while using only 8.1% RGB frames, demonstrating substantial color redundancy in natural
videos and enabling practical always-on
video sensing on resource-constrained devices.
Read the full abstract ▾
All authors Weitong Cai, Hang Zhang, Yukai Huang, Shitong Sun, Jiankang Deng, Songcen Xu, Jifei Song, Zhensong Zhang
Computer Vision
MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding
Hejun Dong, Junbo Niu, Bin Wang et al. (+3 more)
Optical character recognition (OCR) has evolved from line-level transcription to structured document parsing, requiring models to recover long-form sequences containing layout, tables, and formulas. Despite recent advances in
vision-
language models, most existing systems rely on autoregressive decoding, which introduces sequential latency and amplifies error propagation in long documents. In this work, we revisit document OCR from an inverse rendering perspective, arguing that left-to-right causal generation is an artifact of serialization rather than an intrinsic property of the task. Motivated by this insight, we propose MinerU-
Diffusion, a unified
diffusion-based framework that replaces autoregressive sequential decoding with parallel
diffusion denoising under visual conditioning. MinerU-
Diffusion employs a block-wise
diffusion decoder and an uncertainty-driven curriculum learning strategy to enable stable training and efficient long-sequence inference. Extensive experiments demonstrate that MinerU-
Diffusion consistently improves robustness while achieving up to 3.2x faster decoding compared to autoregressive baselines. Evaluations on the proposed Semantic Shuffle
benchmark further confirm its reduced dependence on linguistic priors and stronger visual OCR capability.
Read the full abstract ▾
All authors Hejun Dong, Junbo Niu, Bin Wang, Weijun Zeng, Wentao Zhang, Conghui He