Computer Vision · Cross-listed from cs.IR
DocAnnot – Accelerating the Creation of Key Information Extraction Datasets with GenAI-Powered Auto-annotation
Siddartha Reddy, Harikrishnan P M, Goutham Vignesh et al. (+2 more)
Key Information Extraction (KIE) is vital for many document applications, but creating training
datasets is traditionally a time-consuming manual process. We introduce DocAnnot, a framework that significantly accelerates KIE
dataset generation. DocAnnot leverages a Large
Vision Language Model (LVLM) for label value extraction, OCR for text/bounding box detection, and a novel Spatially Informed Contextual Matching (SICM)
algorithm. SICM improves label-value association by combining spatial relationships and proximity analysis with textual matching. We evaluate our framework on the CORD and SROIE
benchmarks, demonstrating its ability to auto-generate annotations with F1-scores of 0.679 and 0.846, respectively. Furthermore, we investigate the effectiveness of using auto-annotated data for fine-tuning downstream KIE models. While human-annotated data remains superior, models trained exclusively on DocAnnot's outputs attain respectable performance (e.g., LayoutLMv3 achieving an F1-score of 0.6765 on CORD). These results show that while our framework significantly reduces reliance on manual effort, it does not yet fully eliminate the need for human intervention. However, by automating the process to a point where reviewers can efficiently refine outputs, our system enables near-perfect annotations with much greater efficiency than manual annotation from scratch. This approach offers substantial time and cost savings, making it valuable for resource-constrained settings and rapid model prototyping.
Read the full abstract ▾
All authors Siddartha Reddy, Harikrishnan P M, Goutham Vignesh, Varun V, Vishal Vaddina
Computer Vision · Cross-listed from cs.CL
ReportMedSAM: Guiding Segmentation Through Radiology Reports
Anghong Du, Theodoros N. Arvanitis, Colin Watts et al. (+2 more)
Free-form radiology reports contain rich clinical descriptions, yet converting them for reliable
segmentation remains challenging due to the inherent variability of natural
language. Existing pipelines often rely on predefined organ phrases or brittle rule-based inference-time extraction, which limits their scalability to novel anatomical structures and makes them sensitive to linguistic variations. To address this, we propose ReportMedSAM, a report-driven framework that replaces discrete extraction with a learnable concept bank. By leveraging a frozen medical
vision-
language encoder (BiomedCLIP), we align organ-level concept
embeddings with large-scale clinical corpora through contrastive learning, establishing mutually orthogonal semantic anchors. Our approach explicitly mitigates organ-level semantic collapse and ensures high robustness against diverse clinical synonyms (e.g., "renal" vs. "kidney" ). During inference, a clinical report is embedded and matched against this concept bank to dynamically activate task-specific Mixture-of-Experts (MoE) modules. This decoupled design allows new concepts and experts to be added without retraining existing components, providing a parameter-isolated extension mechanism while keeping previously learned experts unchanged. Evaluated on the AbdomenAtlas 3.0
dataset, ReportMedSAM effectively interprets free-form reports, achieves competitive
segmentation accuracy, and demonstrates seamless, non-interfering extension to novel clinical tasks.
Read the full abstract ▾
All authors Anghong Du, Theodoros N. Arvanitis, Colin Watts, Alejandro F. Frangi, Le Zhang
Computer Vision · Cross-listed from cs.AI
DialogueVPR: Towards Conversational Visual Place Recognition
Yukun Song, Changwei Wang, Xingtian Pei et al. (+8 more)
Inspired by how humans communicate spatial information,
language-guided geo-localization has gained significant traction for its intuitive and practical value. Despite this progress, most methods still rely on a static, one-shot
retrieval paradigm, which fails to handle the ambiguity and incompleteness inherent in real-world natural
language descriptions. We propose a paradigm shift to
reasoning retrieval and introduce Dialogue Place Recognition (DlgPR), which casts localization as an interactive, dialogue-driven
reasoning process. To support this new task, we present DlgQuest-Cities, the first large-scale dialogue-based
benchmark for place recognition, and a unified
reasoning framework that couples a cross-modal multi-level retriever with an intelligent questioner, DQ-pilot. DQ-pilot is trained in a curriculum: supervised fine-tuning on a curated DQ-cities-20k subset followed by reinforcement refinement on a harder DQ-cities-10k split via GRPO. Two task-aligned metrics guide learning: a Discriminative Difficulty Index (DDI) for curriculum sampling and a Positional
Retrieval Gain (PRG) reward that directly measures
retrieval improvement induced by a question. Experiments show this
reasoning-based approach significantly outperforms baselines. The code and model are available at https://github.com/Graysonggg/DlgPR.
Read the full abstract ▾
All authors Yukun Song, Changwei Wang, Xingtian Pei, Shibiao Xu, Wenhao Xu, Shunpeng Chen, Yu Zhang, Ke Zhang, Rongtao Xu, Xuxiang Feng, Pengyang Wang
Computer Vision · Cross-listed from eess.IV
A Task-Agnostic Algebraic Integrity Metric for Event-Camera Streams Toward SOTIF-Compliant Perception using Pearson Correlation Coefficient
Arthur de Miranda Neto
Event cameras have emerged as a high-bandwidth, low-latency sensing modality for safety-critical perception in automated driving systems (ADS), offering microsecond temporal resolution, 120-140 dB dynamic range, and intrinsic absence of motion blur. However, no task-agnostic quality metric currently operates directly on the asynchronous event stream: state-of-the-art proxies require a downstream task (e.g., detection accuracy, tracking error) to assess stream integrity, which is incompatible with the certification requirements of ISO 21448 (SOTIF) and ISO/PAS 8800:2024. The recent BiasBench
benchmark (CVPR 2025) explicitly identifies this gap. This work proposes a unified algebraic framework that lifts the Pearson Correlation Coefficient (PCC), historically used in two prior works for redundancy filtering and ROI selection on frame-based images, to the three standard event representations: Time Surface, Event Frame, and Voxel Grid. The framework yields three metrics: (i) r-TS for stream integrity monitoring against an ego-motion-predicted Time Surface, (ii) r2-EF for adaptive ROI selection requiring only integer comparisons, and (iii) r-VG for temporal redundancy gating. A structural isomorphism is established between the contrast-threshold mechanism of the event camera (|Delta L| >= C) and the PCC-based change criterion, the three lifted metrics are formalized, and pipeline latency and information loss are analyzed symmetrically against the raw stream. Illustrative behavior of each metric is demonstrated on a procedural-synthetic event stream, generated by direct simulation of the emission model rather than drawn from any real or
video-derived
dataset, including a tunnel-dip integrity-anomaly scenario in which r_C drops from 0.93 (coherent flow) to below 0 (alarm). An explicit epistemic convention ([ESTABLISHED], [SOLID], [HYPOTH.], [OPEN]) delineates the status of every contribution.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.GR
Generative 3D Gaussians with Learned Density Control
Runjie Yan, Yan-Pei Cao, Peng Wang et al. (+2 more)
We present Density-Sampled Gaussians (DeG), a novel 3D representation designed to bridge the gap between adaptive rendering primitives and scalable generative modeling. Unlike existing approaches that constrain 3D Gaussians to fixed voxel grids or arrays, DeG models Gaussian centers as samples from a learnable probability density function defined over an octree. This formulation provides a rigorous mathematical framework for adaptive density control: by jointly optimizing the spatial density and Gaussian attributes under rendering supervision, our model naturally concentrates primitives in regions of high
geometric complexity. We achieve this via a new render loss contribution gradient that serves as a fully differentiable analogue to the discrete densification and
pruning heuristics used in standard Gaussian Splatting. The resulting representation is highly flexible, supporting variable-resolution decoding from a single latent code by simply adjusting the sampling budget. To enable generative synthesis, we train a latent
diffusion model on DeG. We identify a critical challenge in applying
diffusion to unordered set-structured latents, which can significantly slow convergence, and propose VecSeq, a canonical re-indexing mechanism that anchors latent
tokens to a deterministic 3D Sobol sequence. This transforms the ambiguous set-generation problem into a robust sequence modeling task. Extensive experiments demonstrate that our pipeline achieves state-of-the-art quality in single-image-to-3D generation, combining the structural adaptivity of unstructured primitives with the training stability of grid-based methods.
Read the full abstract ▾
All authors Runjie Yan, Yan-Pei Cao, Peng Wang, Ding Liang, Yuan-Chen Guo
Computer Vision
StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMs
Chang Che, Ziqi Wang, Hui Ma et al. (+2 more)
Continual Visual Instruction Tuning (CVIT) enables
Multimodal Large
Language Models to incrementally acquire new abilities. However, existing CVIT methods operate under a restrictive task-incremental setting, where each training phase corresponds to a single, predefined task. This does not reflect real-world conditions, where data arrives as a continuous stream of interleaved and dynamically evolving tasks. To bridge this gap, we introduce Streaming CVIT (StrCVIT), a more general and realistic setting where models learn from a stream of data chunks containing a dynamic mixture of tasks. In StrCVIT, a model must simultaneously acquire new abilities, reinforce recurring abilities, and mitigate forgetting. Existing CVIT methods fail here as they cannot reliably distinguish or adapt to the heterogeneous task samples within each chunk. We therefore propose StrLoRA, a regularized two-stage expert routing framework. StrLoRA first performs task-aware expert selection using the textual instruction to activate a sparse subset of relevant experts, reducing cross-task interference. It then applies
token-wise expert weighting within this subset, where contribution weights are computed via cross-modal
attention between local visual
tokens and the global instruction representation. To maintain stability across the non-stationary stream, a routing-stability regularization aligns current routing distributions with a historical exponential moving average reference. Extensive experiments on a newly developed StrCVIT
benchmark show that StrLoRA substantially outperforms existing methods, effectively enhancing model's abilities from continuously evolving data streams. The code is available at https://github.com/chanceche/StrCVIT.
Read the full abstract ▾
All authors Chang Che, Ziqi Wang, Hui Ma, Cheems Wang, Zenglin Shi
Computer Vision
MicroDiffuse3D: A Foundation Model for 3D Microscopy Imaging Restoration
Yongkang Li, Brian Wong, King Wai Chiu et al. (+5 more)
Chemical imaging enables label-free visualization of
cells, tissues and living systems while providing direct biochemical information that is difficult to obtain with conventional fluorescence microscopy. Despite its promise in applications ranging from intraoperative diagnosis to drug-response analysis, its broader use remains limited by slow data acquisition, particularly for three-dimensional imaging. Here we present MicroDiffuse3D, a pretrained foundation model for 3D microscopy image restoration that recovers high-quality volumetric structure from degraded low-resolution measurements acquired at substantially higher throughput. We evaluated MicroDiffuse3D across three challenging restoration settings, including 3D super-resolution under 16-fold volumetric sparsity, joint degradation in resolution and noise, and 3D denoising in the low signal-to-noise ratio (SNR) regime, where the model delivered clear gains over strong baselines. Under the sparse 3D super-resolution setting, MicroDiffuse3D produced clearer continuity across depth with fewer artifacts and improved
segmentation quality by 10.58% and line-profile concordance by 15.59%. Together, our results establish pretrained 3D restoration as a broadly applicable strategy for overcoming the throughput and SNR limitations in volumetric chemical imaging, enabling high-resolution analysis at scales and speeds that were previously difficult to achieve.
Read the full abstract ▾
All authors Yongkang Li, Brian Wong, King Wai Chiu, Hanwen Xu, Tangqi Fang, Erin Dunnington, Dan Fu, Sheng Wang
Computer Vision · Cross-listed from cs.AI
Biological Plausibility and Representational Alignment of Feedback Alignment in Convolutional Networks
Jake Lance and Larry Kieu
The feedback
alignment (FA)
algorithm offers a biologically plausible alternative to backpropagation (BP) for training neural networks yet notably fails to scale to convolutional architectures. Modifications have been proposed to address this limitation, but at questionable cost to biological plausibility. In this paper, we evaluate five learning
algorithms including modified FA and standard BP, applied to the same convolutional architecture with the CIFAR-10
dataset. We provide a tripartite comparative analysis focusing on biological plausibility, interpretability, and computational complexity. Our results indicate that modified FA
algorithms converge on internal representations that are structurally similar to those produced by backpropagation. In particular, it appears the functional success of modified FA
algorithms may be rooted in their ability to mimic the representational geometry of backpropagation, converging on similar representations despite relying on fundamentally different weight update mechanisms.
Read the full abstract ▾
Computer Vision
ZAYA1-VL-8B Technical Report
Hassan Shapourian, Kasra Hejazi, Olabode M. Sule et al. (+1 more)
We present ZAYA1-VL-8B, a compact mixture-of-experts
vision-
language model built upon our in-house
language model, ZAYA1-8B. Despite its compact size, ZAYA1-VL achieves performance competitive with leading base models such as Molmo2-4B and InternVL3.5-4B, while surpassing models including Qwen2.5-VL-3B, PLM-3B, and MolmoE-1B across a range of image understanding,
reasoning, and counting
benchmarks. The architecture incorporates two key innovations: (1)
vision-specific LoRA adapters integrated into the LLM to increase modality-specific capacity without increasing the number of experts, and (2) bidirectional
attention over image
tokens within the LLM to enhance visual understanding. We detail the full training pipeline including data composition at each stage, sequence packing, and the
attention masking scheme. The model comprises 9.2B total parameters, with 1.4B active parameters including the
vision encoder, and is publicly available at https://huggingface.co/Zyphra/ZAYA1-VL.
Read the full abstract ▾
All authors Hassan Shapourian, Kasra Hejazi, Olabode M. Sule, Beren Millidge
Computer Vision
MC-RFM: Geometry-Aware Few-Shot Adaptation via Mixed-Curvature Riemannian Flow Matching
Salim Khazem, Ibrahim Mohamed Serouis and Zakaria Ezzahed
Parameter-efficient adaptation of pretrained
vision models is commonly performed through linear probes,
prompts, low-rank updates, or lightweight residual modules. While effective, these methods usually treat adaptation as a discrete Euclidean perturbation of frozen representations, without explicitly modeling the geometry of the task-induced feature displacement. We propose MC-RFM, a mixed-
curvature Riemannian flow-matching framework for few-shot adaptation of frozen visual backbones. The key idea is to represent adapted features on a product
manifold combining a
hyperbolic factor, which captures hierarchy-sensitive semantic structure, and a Euclidean factor, which preserves locally discriminative visual variation. Adaptation is formulated as a task-conditioned continuous transport from frozen features to support-set prototypes, trained with a flow-matching objective and coupled to a hybrid prototype-linear classifier. The method is lightweight, backbone-agnostic, and operates entirely on cached frozen features. Across seven visual recognition
benchmarks, five frozen backbones, and 1/4/16-shot regimes, MC-RFM is the best-performing method in a majority of evaluated settings, with the strongest gains on
Transformer backbones and fine-grained
datasets. Ablations show that the mixed-
curvature head, task conditioning, adaptive branch gating, prototype shrinkage, and discriminative supervision each contribute to performance. These results suggest that few-shot adaptation benefits not only from deciding which parameters to update, but also from modeling how representations should move through a geometry matched to the structure of the downstream task.
Read the full abstract ▾
Computer Vision
A Two-Stage Shape- and Motion-Aware Framework for mmWave-based Human Mesh Recovery
Hoang Hai Pham, Shuntian Zheng, Jiaqi Li et al. (+1 more)
Millimeter-wave (mmWave) radar has emerged as a promising sensing modality for human perception, offering robustness under challenging environmental conditions and strong
privacy preservation. However, recovering accurate 3D human meshes from radar is difficult for two reasons: radar tensors contain significant environmental clutter, and each frame observes only part of the body surface. These challenges demand opposite operations, removing measured energy that does not belong to the body versus inferring geometry the sensor never measured. Yet previous works typically entangle both within a single network, with the mesh objective as the main supervision. We instead reformulate radar-based mesh recovery as two explicitly supervised sub-problems. A signal-level separation stage isolates body-related voxels through coarse-to-fine localization and voxel-wise
segmentation, trained with visibility derived voxel labels. A structure-level reconstruction stage then recovers the body from this partial observation using a dual-branch network that jointly models per-frame shape and inter-frame motion. Experiments show that our method reduces mesh error by up to 9.1% over the strongest baseline while using over 10 $\times$ fewer parameters. Source code can be found via https://github.com/haipham03/mmTSM-Mesh
Read the full abstract ▾
All authors Hoang Hai Pham, Shuntian Zheng, Jiaqi Li, Yu Guan
Computer Vision
Geometric Flood Depth Estimation: Fusing Transformer-Based Segmentation with Digital Elevation Models
Nhut Le, Ehsan Karimi and Maryam Rahnemoonfar
Post-disaster situational awareness relies heavily on understanding both the extent and the volume of floodwaters. While 2D semantic
segmentation provides accurate flood masking, it lacks the vertical dimension required to assess navigability and structural risk. This paper presents a
geometric "Water Surface Elevation" approach for estimating flood depth from
monocular aerial imagery. Our pipeline utilizes Mask2Former, a state-of-the-art
transformer-based
segmentation model, to generate precise 2D flood masks. These masks are fused with Digital Elevation Models (DEMs) to identify the water-land boundary, calculate a global water surface elevation ($Z_{water}$), and compute per-pixel depth based on the principle of local hydrostatic
equilibrium. We evaluate this workflow using the FloodNet and CRASAR-U-DROIDS
datasets, demonstrating how high-performance
segmentation can be leveraged to extract 3D volumetric data from 2D imagery without the latency of hydrodynamic simulations.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
A Deep Risk Estimator for Known Operator Learning
Andreas Maier, Md Hasan, Paulina Conrad et al. (+1 more)
We describe an approach for estimating the statistical risk of deep networks that contain a mix of learned and known operators. Building on the maximal training error bounds previously established for known operator learning, we derive a deep risk
estimator that connects the expected error of a layered network to the size of the training sample. The
estimator decomposes the total risk into a sum over learned layers; every known operator contributes zero to this sum, while every learned layer adds an approximation term inspired by Barron's classic work and an estimation term that decreases with the number of training samples. We are able to show that the bound shrinks whenever a learned layer is replaced by a known operator and that the corresponding sample requirement scales with the number of trainable parameters of the layer that is replaced. As an application, we use computed
tomography as an example and compare an operator-aware filtered backprojection network with a fully connected substitute that collapses the entire reconstruction pipeline into a single learned dense
matrix. The predicted parameter ratio coincides with the structural sparsity that the analytic decomposition into a circulant filter and a sparse backprojection exposes. We confirm the predicted scaling on CPU at small image scale and on GPU at medium image scale, all on the same scaling law. Beyond CT reconstruction, the
estimator applies to physics-informed neural networks that hardcode a known physical operation in its architecture, and we expect the result to be of interest for a broad community working on operator-aware deep learning. Calibrating the per-layer constants on each sweep yields a bound that tracks the empirical test MSE within a factor of two at every training-set size, so the
estimator can be inverted to predict how many training samples are required to reach a target error.
Read the full abstract ▾
All authors Andreas Maier, Md Hasan, Paulina Conrad, Paula Andrea Perez-Toro
Computer Vision
CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysis
Haroon Wahab, Irfan Mehmood and Hassan Ugail
Wireless capsule endoscopy (WCE) enables non-invasive visual assessment of the small bowel, but its clinical utility is constrained by the large volume of frames generated per examination and the difficulty of recognising subtle abnormalities under highly variable imaging conditions. Existing learning-based approaches for WCE are predominantly
vision-only, often confined to narrow pathology sets, and show limited transfer across
datasets and centres. To address these limitations, this study introduces CapCLIP, a domain-specific
vision-
language representation learning framework for WCE. CapCLIP aligns capsule endoscopy frames with clinically grounded textual descriptions derived from standardised nomenclature and pathology-aware caption templates, thereby learning
embeddings that are both semantically informed and transferable. The proposed framework is evaluated against relevant open-source
vision and
vision-
language foundation models under strict zero-shot conditions using unseen WCE
datasets. Evaluation covers three downstream tasks: K-nearest neighbour classification, CLIP-style image-text classification, and text-to-image
retrieval. Across these settings, CapCLIP consistently outperforms the compared baselines, with particularly strong gains in zero-shot image-text classification and cross-modal
retrieval on out-of-distribution
datasets. The results indicate that
language-guided representation learning can improve both generalisation and semantic interpretability in WCE analysis. These findings position CapCLIP as a step toward foundation models tailored to capsule endoscopy and support the use of
language-grounded WCE analysis.
Read the full abstract ▾
Computer Vision
NICE FACT: Diagnosing and Calibrating VLMs in Quantitative Reasoning for Kinematic Physics
Jian Lan, Zhicheng Liu, Xinpeng Wang et al. (+5 more)
The ability to derive precise spatial and physical insights is a cornerstone of
vision-
language models (VLMs), yet their poor performances in related spatial intelligence tasks such as physical
reasoning remain a fundamental barrier. The community critically lacks a scientific analysis revealing whether VLMs faithfully reach answers or plausibly make guesses. This work aims to provide a fundamental understanding of how VLMs perceive the physical world, and utilize physical laws, while assessing the reliability of model confidence. We propose NICE and FACT, a dual-diagnostic paradigm that explicitly decomposes quantitative
reasoning for kinematic physics: FACT diagnoses visual fidelity, physical law comprehension, and temporal grounding. NICE studies our novel neighborhood-informed calibration method and novel metrics to evaluate and calibrate confidence reliability. Evaluated across 6 latest state-of-the-art VLMs, we uncover that models fail to identify visual preconditions or utilize necessary physical laws to reach answers. This work highlights and establishes a standardized diagnostic paradigm to guide the development of faithful, physically-grounded VLMs.
Read the full abstract ▾
All authors Jian Lan, Zhicheng Liu, Xinpeng Wang, Yuhao Zhou, Haokun Chen, Jiancheng Lv, Barbara Plank, Thomas Seidl
Computer Vision · Cross-listed from cs.LG
TARO: Temporal Adversarial Rectification Optimization Using Diffusion Models as Purifiers
Daniel Wesego and Pedram Rooshenas
Adversarial purification with
diffusion models seeks to project
adversarial examples back toward the data
manifold, but balancing semantic preservation and robustness against adaptive attacks remains challenging. Recent work shows that standard
diffusion purification can fail under adaptive evaluation, while test-time score-based optimization is more resilient. Existing optimization defenses, however, typically rely on a single
diffusion noise regime or treat timesteps uniformly, overlooking the distinct roles of coarse and fine denoising scales. We propose Temporal
Adversarial Rectification Optimization (TARO), an inference-time purification method that builds a temporally guided score prior from multiple denoising views along the
diffusion trajectory. TARO forms a coarse-to-fine residual target: high-noise experts provide globally smoothed structure with reduced
adversarial sensitivity, while low-noise experts restore image-specific, class-relevant details. A guidance strength controls this temporal correction, allowing TARO to balance robust global rectification with semantic preservation. Empirically, TARO improves robust accuracy across
datasets and adaptive threat models in a zero-shot setting, while remaining compatible with complementary
adversarial-likelihood objectives for further robustness gains.
Read the full abstract ▾
Computer Vision
Beyond Bag-of-Patches: Learning Global Layout via Textual Supervision for Late-Interaction Visual Document Retrieval
Pascal Tilli and Mohsen Mesgar
Visual Document
Retrieval (VDR) models mostly rely on late interaction architectures, in which documents are represented by a set of local patch
embeddings and then matched against query
tokens. While efficient, this architecture prioritizes local similarity over global layout structure of documents to estimate relevancy between documents and query. In practice, this leads to errors as relevance originates from layout structure of documents with heterogeneous layouts combining figures, tables, and text. We make document layout learnable without changing inference. We propose a
multimodal encoder that augments local patch representations with a global layout
embedding, trained via textual descriptions encoding document layout information. Across four ViDoRe-v2
datasets, our model improves over the strongest architecturally comparable ColPali/ColQwen baseline by +2.4 nDCG@5 and +2.3 MAP@5, with statistically significant per-
dataset gains over ColQwen.
Read the full abstract ▾
Computer Vision
Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy et al. (+1 more)
Multimodal Large
Language Models (MLLMs) have made rapid progress in single-
video understanding, yet their ability to reason across multiple independent
video streams remains poorly understood. Existing multi-
video benchmarks rely largely on human-annotated real-world footage, limiting the precision of spatial, temporal, and physical ground truth and making it difficult to diagnose model failures. We introduce SYNCR, a controlled synthetic
benchmark for cross-
video reasoning with programmatically verified grounding. Built using Habitat, Kubric, and CLEVRER simulator engines, SYNCR contains 4,000 multi-
video question-answer pairs grounded in 4,827 unique
videos. It evaluates MLLMs across eight tasks spanning four diagnostic pillars: Temporal
Alignment, Spatial Tracking, Comparative
Reasoning, and Holistic Synthesis. Our zero-shot evaluation of leading open- and closed-weight MLLMs reveals a substantial gap between current models and humans: the best model achieves only 64.5% average accuracy, compared to an 89.5% human baseline. Models perform relatively well on temporal ordering but struggle with precise physical and spatial
reasoning, with the best model reaching only 29.8% accuracy on Kinematic Comparison. We further find that parameter scaling and
reasoning-specialized post-training improve temporal
alignment capabilities, but do not reliably address fine-grained physical tracking or global spatial synthesis. Finally, a sim-to-real correlation analysis suggests that SYNCR tracks model-level trends on a real-world multi-
video benchmark.
Read the full abstract ▾
All authors Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami
Computer Vision · Cross-listed from cs.CL
Built Environment Reasoning from Remote Sensing Imagery Using Large Vision–Language Models
Dongdong Wang, Deepak Balakrishnan, Ravi Srinivasan et al. (+1 more)
This work investigates the use of large
language models (LLMs) for tasks in smart cities. The core idea is to leverage remote sensing imagery to characterize the built environment, including design suggestions, constructability assessment, landuse patterns, and risk identification. We examine remote sensing imagery at multiple spatial scales as inputs for
multimodal language modeling and evaluate their effects on built-environment-related
reasoning. In addition, we compare state-of-the-art LLMs, including InternVL and Qwen, in terms of accuracy and reliability when generating built environment recommendations. The results demonstrate the potential of integrating remote sensing imagery with large
language models to assist smart cities and decision-making.
Read the full abstract ▾
All authors Dongdong Wang, Deepak Balakrishnan, Ravi Srinivasan, Shenhao Wang
Computer Vision · Cross-listed from cs.LG
Exploring and Exploiting Stability in Latent Flow Matching
Rania Briq, Michael Kamp, Ohad Fried et al. (+2 more)
In this work, we show that Latent Flow-Matching (LFM) models are robust to different types of perturbations, including data reduction and model capacity shrinkage. We characterize this stability by these models' tendency to generate similar outputs under identical noise seeds. We provide a perspective relating this phenomenon to flow matching theory, which indicates that this stability is inherent to the FM objective. We further exploit this stability to derive practical
algorithms for more efficient training and inference. Concretely, first, we show that by training LFM models on significantly reduced
datasets, performance is preserved, and in compute-constrained regimes, the model converges faster while maintaining quality. This yields multiple advantages, including savings in the training time due to faster convergence, and alleviating annotation effort when training conditional models. Second, LFM stability under architectural shrinkage gives rise to a two-model coarse-to-fine approach, one using a light-weight architecture for the first phase of the FM trajectory, and one with higher capacity for the second, thereby reducing the inference cost substantially. To determine which samples are informative, we introduce three sample-scoring criteria and evaluate them under standard metrics for generative models. Our results are thoroughly evaluated on multiple
datasets, demonstrating the practical advantage of this stability, including data savings and a more than two-fold inference speedup while generating comparable outputs.
Read the full abstract ▾
All authors Rania Briq, Michael Kamp, Ohad Fried, Sarel Cohen, Stefan Kesselheim
Computer Vision
Delivering Science as a Service: Sci-Orchestra's Cloud-Native Approach to HPC
Harinarayan Krishnan, Shubhabrata Mukerjee, Jeffrey Donatelli et al. (+1 more)
The increasing complexity of modern computational environments often burdens researchers with infrastructure management, authentication protocols, and container deployments. We present Sci-Orchestra, a layered orchestration framework designed to fully automate experimental workflows, allowing scientists to prioritize scientific discovery over backend operations. By abstracting execution through an API-driven interface, the system assumes responsibility for secure authentication, resource management, and scalable deployment across diverse high-performance computing environments using Kubernetes architectures. A key innovation of Sci-Orchestra is its autonomous marketplace, which serves as a catalyst for cross-institutional collaboration. Through an intuitive user interface, researchers can rapidly deploy and share specialized services via simple selections, eliminating the need for complex installations and technical setups. This modular infrastructure is specifically designed to facilitate industry partnerships as it provides a secure execution environment and allows external collaborators to test and validate proprietary tools without the need for source-code exchange. This “black-box” interoperability protects intellectual property while enabling seamless integration into broader scientific pipelines, ultimately accelerating the transition from laboratory prototypes to industrial-scale applications.
Read the full abstract ▾
All authors Harinarayan Krishnan, Shubhabrata Mukerjee, Jeffrey Donatelli, Daniela Ushizima
Computer Vision
Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval
Mingyu Liu, Sihan Huang, Yijia Fan et al. (+4 more)
Zero-shot composed image
retrieval (ZS-CIR) retrieves a target image from a reference image and a text modification without human-annotated CIR triplets. Projection-based ZS-CIR methods are attractive because they do not rely on LLMs at inference and remain lightweight, but they often underperform LLM-based approaches on complex semantic modifications. This gap reflects a semantic transition bottleneck in projection-based ZS-CIR: endpoint-level matching can let the edit text act as a target-side attribute cue rather than grounding it as a source-conditioned semantic transition. We further show that adding semantic transition supervision to the same text adapter creates an endpoint–transition conflict between endpoint
alignment and semantic transition
alignment. To address this conflict, DeCIR decouples endpoint and transition learning. It constructs paired forward/reverse edit tuples from image-caption pairs, trains separate low-rank text adapter branches for endpoint
alignment and semantic transition
alignment, and merges them with Low-Rank Directional Merge (LRDM) into one deployable adapter. Extensive experiments on CIRR, CIRCO, FashionIQ, and GeneCIS demonstrate that DeCIR consistently improves projection-based ZS-CIR without increasing inference complexity.
Read the full abstract ▾
All authors Mingyu Liu, Sihan Huang, Yijia Fan, Yinlin Yan, Quan Zhang, Jian-Fang Hu, Jianhuang Lai
Computer Vision
UIESNN: A Scale-Aware Spiking Network for Underwater Image Enhancement
Shuang Chen, Ruochen Li, Zihan Zhu et al. (+3 more)
Underwater image enhancement (UIE) is a practically important yet underexplored application of spiking neural networks (SNNs), where the dominant degradations are large-scale and low-frequency, such as wavelength-dependent colour casts and scattering-induced veiling. Existing SNN restoration designs rely on locally bounded spiking perception, which can limit global correction and lead to saturated or inconsistent representations. To address these challenges, we propose a scale-aware SNN framework for UIE named UIESNN. At its core is a Multi-scale Pooling LIF Block (MPLB) that injects hierarchical multi-scale pooling responses into membrane dynamics, thereby enlarging the effective receptive field while preserving fine-grained details and inducing heterogeneous scale-dependent activations. Building on MPLB, we design a spiking residual architecture that integrates frequency decomposition and
attention-based refinement in a fully spike-driven pipeline. Extensive experiments on the EUVP and LSUI
benchmarks demonstrate that UIESNN achieves state-of-the-art performance among SNN-based methods, delivering improved colour fidelity and spatial coherence with competitive energy cost.
Read the full abstract ▾
All authors Shuang Chen, Ruochen Li, Zihan Zhu, Ronald Thenius, Farshad Arvin, Amir Atapour-Abarghouei
Computer Vision
NeuroGAN-3D: Enhancing Intrinsic Functional Brain Networks via High-Fidelity 3D Generative Super-Resolution
M. Moein Esfahani, Sepehr Salem Ghahfarokhi, Mohammed Alser et al. (+2 more)
Recent advances in neuroimaging have deepened our understanding of the
brain's complex functional and structural organization. Among these, functional
Magnetic Resonance Imaging (fMRI) - particularly resting-state fMRI (rs-fMRI) - has emerged as a tool for identifying biomarkers of intrinsic
brain connectivity and delineating large-scale neural networks. These networks are typically represented as volumetric spatial maps that capture functionally coherent
brain regions and reflect individual differences in
brain activity and structure. The spatial resolution of these maps plays an important role, as it determines the ability to localize functional units with precision, perform reliable
brain parcellation, and detect subtle, spatially specific neurobiological alterations associated with development, aging, or disease. Therefore, improving the effective resolution of neuroimaging-derived maps holds significant promise for enabling more detailed insights into
brain architecture and its relationship to behavior and pathology. To address this need, we propose NeuroGAN-3D, a novel 3D generative super-resolution model tailored to the computational demands of volumetric neuroimaging. Our model leverages a generative
adversarial network architecture to enhance the spatial resolution of rs-fMRI spatial maps, significantly outperforming a conventional baseline.
Read the full abstract ▾
All authors M. Moein Esfahani, Sepehr Salem Ghahfarokhi, Mohammed Alser, Jingyu Liu, Vince Calhoun
Computer Vision
PrePARE: Pre-AA Token Pruning for Frozen Multi-View Geometry Transformers
Haotang Li, Zhenyu Qi, Shaohan Henry Wang et al. (+5 more)
Multi-view geometry
transformers are feed-forward 3D foundation models that jointly predict depth maps, point maps, and camera poses for N images in a single forward pass. Most of them build on an alternating-
attention (AA) stack whose cross-view
attention runs over the patch
tokens of all N views in every block, so its cost grows quadratically in N: on 300-frame ScanNetv2 clips, VGGT peaks at 72.3 GiB, and MapAnything does not fit on a 93 GiB card. Existing methods reduce
tokens or
attention inside the AA stack, so the full patch grid still enters it, and the pre-AA interface, where the encoder hands its
tokens to the first AA block, has been overlooked. We propose PrePARE, which prunes patch
tokens once at the pre-AA interface and restores the dense grid once after the last AA block. Every block of the frozen AA stack then runs on the reduced
tokens, while the prediction heads receive the full grid. A
Token Scorer, supervised by how the unpruned AA stack uses each
token, and a Feature-guided Restoration module are the only trained parts. The
Token Routing module is rule-based. On ScanNetv2 at N = 300, PrePARE reduces the peak
memory of VGGT by 80%, from 72.3 to 14.4 GiB, below every in-AA method we ran, and runs 5.4 times faster, while its per-scene Chamfer distance is lower than the unpruned model's on 34 of 50 scenes (sign test p = 0.015). On pi^3 and Depth Anything 3 it saves 15% and 20% of peak
memory, where the in-AA methods save at most 9%, and at 1000 frames it is the only method we ran that fits Depth Anything 3 and MapAnything on one GPU.
Read the full abstract ▾
All authors Haotang Li, Zhenyu Qi, Shaohan Henry Wang, Kebin Peng, Zi Wang, Qing Guo, Sen He, Huanrui Yang
Computer Vision
An Efficient Token Compression Framework for Visual Object Tracking
Weijing Wu, Qihua Liang, Bineng Zhong et al. (+3 more)
Refining visual representations by eliminating their internal feature-level redundancy is crucial for simultaneously optimizing the performance and computational cost of models in visual tracking. To enhance their performance, many contemporary
Transformer-based trackers leverage a larger number of historical template frames to capture richer spatio-temporal cues. However, this strategy leads to a massive number of input visual
tokens. This creates two critical issues: it imposes a quadratic computational burden and can also degrade the tracker's overall performance. To bridge this gap, we propose a compress-then-interact tracking framework, ETCTrack, that learns to efficiently compress template
tokens from historical template frames into a robust target representation, moving beyond handcrafted rules. Our method first employs the Adaptive
Token Compressor to dynamically construct compact yet highly discriminative template
tokens by filtering out redundant visual
tokens. These refined template
tokens are then processed by our Hierarchical Interaction Encoder to achieve a deep, adaptive interaction with the search features. Refined search features ensure subsequent precise target localization. Experiments on seven
benchmarks demonstrate that our method outperforms current state-of-the-art trackers. ETCTrack-B224 reduces the number of template
tokens by 60%, leading to a 21.4% reduction in MACs with only a 0.4% drop in accuracy. The source code are available at https://github.com/PJD-WJ/ETCTrack.
Read the full abstract ▾
All authors Weijing Wu, Qihua Liang, Bineng Zhong, Haiying Xia, Zhiyi Mo, Shuxiang Song