Computer Vision
Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents
XiuYu Zhang, Junfeng Fang and Zhenkai Liang
Latent visual
reasoning (LVR) inserts supervised latent
tokens between perception and answer generation in
vision-
language models (VLMs). The field uses
alignment between these latents and their visual targets, i.e., cosine similarity or mean squared error (MSE), as both the training loss and the quality metric, assuming that better
alignment yields a better answer. We test this with a designed
matrix of five LVR variants and find the assumption inverted: cosine
alignment is negatively correlated with accuracy across all five (r=-0.94). To explain this, we introduce PRISM, a pair of inference-time diagnostics: a linear probe that asks where the answer is decodable, and a corruption test that asks whether the latent is load-bearing. The supervised latents are largely bypassed. Corrupting them shifts accuracy by at most four points. The answer is decodable downstream of the latent but not at it, and the size of this decodability gap predicts how much each variant relies on its latent under perturbation. Consistent with an Information Bottleneck reading of the loss, the auxiliary objective reshapes the
language model via shared parameters rather than via the latent variable it nominally optimizes.
Read the full abstract ▾
Computer Vision
Let It Be Simple: One-Step Action Generation for Vision-Language-Action Models
Yitong Chen, Shiduo Zhang, Jingjing Gong et al. (+1 more)
One-step generation from standard
diffusion or Flow-Matching training is rarely effective for text-to-image or
video generation without
distillation or specialized one-step objectives. We find that VLA action generation behaves strikingly differently. Biasing only the training timestep distribution toward the noise endpoint yields strong one-step policies across LIBERO, LIBERO-Plus, LIBERO-Pro, and real-world
robot tasks. Moreover, for open-source VLA models with different architectures, simply replacing iterative decoding with a single forward pass often performs comparably to the original multi-step decoding on
benchmarks such as LIBERO, CALVIN, and SimplerEnv. We attribute this difference to the condition-target structure of VLA generation, which is closer to image-to-text than text-to-image: rich observations strongly constrain a comparatively compact target. We formalize this intuition through the irreducible velocity loss $R_v(t,c)$ of standard Flow Matching, which becomes much smaller toward the noise endpoint for VLA, whereas image-generation models usually exhibit a U-shaped profile. The geometry induced by the learned generative map shows the same contrast: the Jacobian of this map with respect to the initial noise has a substantially smaller largest singular value for action generation than for image generation. Together, these results suggest that VLA's condition-target structure makes one-step generation unusually accessible to standard Flow Matching. Code and training configurations for this paper are available at https://github.com/bondtesty/let-it-be-simple.
Read the full abstract ▾
All authors Yitong Chen, Shiduo Zhang, Jingjing Gong, Xipeng Qiu
Computer Vision
VTI-CoT: Visual-Textual Interleaved Chain of Thought for Video Reasoning
Shufan Zhang, Ziyue Lin, Bairun Wang et al. (+4 more)
Video reasoning aims to understand complex temporal events and causal relationships within
videos. Recently, Chain-of-Thought (CoT) has been introduced to this field to enhance
reasoning accuracy. However, existing CoT-based
video reasoning methods primarily rely on text-only information for logical deduction, overlooking critical visual information during the inference process. Inspired by the human cognitive mechanism of reviewing visual segments during inference, we propose VTI-CoT, a Visual-Textual Interleaved CoT framework. VTI-CoT integrates textual
reasoning steps with corresponding visual frames. Given the scarcity of visual-textual interleaved CoT in existing
datasets, we develop an automated annotation pipeline to construct high-quality
multimodal CoT data. Further,
reasoning over long-form
videos entails increasingly long CoT
token sequences, which severely hinders training convergence and efficiency. To address this, we employ Optical Character Recognition (OCR)-based compression techniques to compress CoT supervision signals into a single canvas. Experimental results demonstrate that VTI-CoT achieves state-of-the-art performance among models of the same parameter scale while significantly improving training efficiency.
Read the full abstract ▾
All authors Shufan Zhang, Ziyue Lin, Bairun Wang, Lei Jin, Xuanding Ding, Xinzhu Ma, Kunlin Yang
Computer Vision
TextWand: A Unified Framework for Scene Text Editing
Shuyu Wang, Zhile Guan, Hongxiu Chen et al. (+5 more)
We propose TextWand, a general-purpose framework that unifies scene text removal, generation, and replacement into a single model. By decomposing complex editing tasks into the atomic primitives of rendering and erasure, TextWand achieves precise control over both text appearance and background integrity. Specifically, we introduce a novel design, Overlay-Reference Positional Encoding (ORPE), to enforce pixel-level layout fidelity and exemplar-driven style control, alongside a new strategy, Region-Adaptive Suppression (RAS), to ensure clean text erasure. To address the absence of a comprehensive
benchmark for general-purpose scene text editing among existing single-task
datasets, we construct TextWand-Bench. Extensive experiments demonstrate that TextWand outperforms existing leading open-source and closed-source models by delivering superior text content accuracy, layout and style consistency, and overall image quality across scene text removal, generation and replacement tasks.
Read the full abstract ▾
All authors Shuyu Wang, Zhile Guan, Hongxiu Chen, Yule Duan, Weiqi Li, Xin Shan, Ronggang Wang, Jian Zhang
Computer Vision
ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation
Kanghui Tian, Siyuan Liu, Ziang Yan et al. (+3 more)
On-policy
distillation (OPD) improves
reasoning by training a student on trajectories sampled from its own policy under supervision from a teacher. In
multimodal reasoning, a common extension is to use a privileged teacher that observes training-time-only signals such as reference answers or rationales. However, such answer-side privilege creates a train-test mismatch: the teacher's supervision may depend on signals unavailable to the student, encouraging shortcut imitation rather than visually grounded
reasoning. We propose ViCuR, a visually grounded privileged-teacher
distillation framework that replaces answer-side privilege with visual cues (query-related evidence in the input). Because these cues are derived from the same visual input available at inference, their evidence is recoverable by the student. To support this, ViCuR introduces a lightweight cue recovery module that uses dedicated sink-
token cross-
attention during prefill to aggregate task-relevant visual evidence into an internal representation, without changing the inference interface or requiring auxiliary cue-generation losses. Across seven
benchmarks with Qwen3-VL-2B and 8B students, ViCuR consistently improves over answer-based on-policy self-
distillation by +1.19 and +1.24 on overall average performance. It also extends naturally to stronger-teacher OPD, surpassing OPD baselines by +0.64 and +1.08, with consistent out-of-domain gains at the 8B scale. These results show that, in
multimodal on-policy
distillation, the design of teacher privilege is as important as teacher strength.
Read the full abstract ▾
All authors Kanghui Tian, Siyuan Liu, Ziang Yan, Sheng Xia, Shuai Dong, Yi Wang
Computer Vision
Real-Time Threat Detection from Surveillance Cameras using Machine Learning
Gajendra Mandal, J. P. Patra and Priyansh Mahant
Ensuring public safety in densely populated urban environments remains a critical challenge, necessitating the deployment of intelligent and automated
video surveillance systems. Traditional surveillance approaches rely heavily on manual monitoring, which is inefficient and susceptible to human fatigue, delayed response, and observational errors. To overcome these limitations, this work presents a real-time object detection-based surveillance framework. The proposed system focuses on detecting guns, knives, and region-specific blunt objects commonly involved in violent activities in Indian surveillance scenarios. A key contribution of this work is the use of a custom-created
dataset collected using a mobile camera, consisting of 336 labeled images of blunt objects such as iron rods, wooden sticks, and plastic rods. This
dataset is combined with a publicly available
dataset of 7,623 images of guns and knives, forming a consolidated
dataset of 7,959 images across three classes: gun, knife, and blunt object. The combined
dataset is used to train a YOLOv8-based object detection model for real-time performance. Experimental evaluation shows that increasing the training duration significantly improves recall and average precision for the blunt object class without signs of overfitting. Overall, the proposed framework achieves an effective balance between accuracy and efficiency, making it suitable for deployment in real-world surveillance environments such as campuses, public spaces, and transportation areas.
Read the full abstract ▾
Computer Vision
Parallel Jacobi Decoding for Fast Autoregressive Image Generation
Boya Liao, Ying Li, Siyong Jian et al. (+1 more)
Autoregressive (AR) models have demonstrated remarkable performance in generating high-fidelity images. However, their inherently sequential next-
token prediction leads to significantly slower inference. Recent studies have introduced Jacobi-style decoding to accelerate autoregressive image generation. Extending the draft sequence initially improves efficiency, yet the acceleration quickly saturates as error propagation in the one-dimensional sequence hinders convergence. Observing that images exhibit strong local spatial correlations, we propose Parallel Jacobi Decoding (PJD), a training-free decoding approach that expands draft
tokens in the two-dimensional spatial domain to enable efficient spatially parallel refinement. PJD adjusts the
attention mask to mitigate error accumulation and improve convergence stability. Extensive experiments on diverse
datasets show that PJD achieves 4.8x-6.4x acceleration across multiple autoregressive image generation models while maintaining competitive generation quality.
Read the full abstract ▾
All authors Boya Liao, Ying Li, Siyong Jian, Huan Wang
Computer Vision · Cross-listed from cs.AI
Seeing Time: Benchmarking Chronological Reasoning and Shortcut Biases in Vision-Language Models
Haoyu Zhou, Qing Qing, Caichong Li et al. (+6 more)
Recent advancements in
Vision-
Language Models (VLMs) have significantly enhanced their ability to interpret complex visual semantics, yet their capacity for chronological
reasoning remains under-explored. In this paper, we introduce a novel
benchmark specifically designed to evaluate how VLMs perceive and reason about chronological information within and across images. Unlike existing
video-based
benchmarks that focus on frame sequencing, our work delves into the underlying logic of chronological judgment and the expansion toward
multimodal integration. To facilitate this, we construct three specialized
datasets: one containing visually similar objects spanning long historical durations, another categorized by diverse event and object types, and a third pairing images with time-sensitive news text for cross-modal
alignment. Through extensive experiments, we analyze whether models exhibit performance disparities across categories and, crucially, explore whether they rely on “incorrect shortcuts”, such as image color rather than genuine chronological features. Our results reveal that while VLMs show promise, they frequently exploit superficial cues like grayscale versus color filters to bypass authentic chronological
reasoning. By providing these high-quality
datasets and a rigorous evaluation framework, we offer a diagnostic tool to identify current limitations and guide the development of more robust, logically grounded
multimodal models. The source code is shown in https://github.com/LuoRenqiang/ChronoVision.
Read the full abstract ▾
All authors Haoyu Zhou, Qing Qing, Caichong Li, Qixin Zhang, Yongcheng Jing, Ziqi Xu, Juncheng Hu, Xikun Zhang, Renqiang Luo
Computer Vision
T-SAR-JEPA: Self-Supervised Temporal Anomaly Detection in SAR Amplitude Stacks via Latent Prediction
Kerod Woldesenbet and Abem Woldesenbet
We present T-SAR-JEPA, a self-supervised framework for temporal anomaly detection in SAR amplitude stacks via latent prediction. A ViT-Base/16 encoder from SAR-JEPA is domain-adapted on 39,300 Capella patches using local masked reconstruction with gradient feature prediction. A temporal
transformer with sinusoidal time encoding forecasts future latent states from K=7 acquisitions, with progressive unfreezing substantially reducing validation loss. The model operates on amplitude alone; InSAR coherence serves exclusively as independent pseudo-ground-truth. On the DFC 2026
dataset (300 time-series, three AOIs), T-SAR-JEPA achieves ROC-AUC of 77.0% on the Hawaii eruption window, outperforming RX, PaDiM, Linear AR, and LSTM baselines (~50%). Spatial coherence of 99.9% (p < 0.001, permutation test) confirms structured detections. Code: https://github.com/TerraLatent/t-sar-jepa
Read the full abstract ▾
Computer Vision
LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video
Shiqiang Lang, Jing Liu, Haoyang He et al. (+6 more)
Multimodal Large
Language Models (MLLMs) have advanced image and
video understanding and can increasingly handle longer visual inputs. Long-horizon tasks such as autonomous driving and robotic
navigation require more than recognizing the current view, as models must remember and retrieve previously observed spatial layouts, routes, viewpoint changes, and object states. To evaluate this capability, we introduce LongSpace-Bench, a room-tour
video benchmark for long-horizon spatial
memory, covering scene perception, spatial relations, and spatial
memory. In this work, we further propose LongSpace, a
memory framework for long-
video spatial
reasoning. LongSpace models long
videos as sequential chunks, incorporates 3D structural cues into early decoder layers, and constructs layer-aware
memory for question-guided
retrieval. Experiments on multiple spatial
reasoning benchmarks show that LongSpace improves long-
video spatial understanding, further demonstrating explicit spatial
memory as a key capability for long-horizon
video MLLMs.
Read the full abstract ▾
All authors Shiqiang Lang, Jing Liu, Haoyang He, Peiwen Sun, Yuanteng Chen, Tao Liu, Lan Yang, Longteng Guo, Honggang Zhang
Computer Vision · Cross-listed from cs.LG
Two-Way Is Better Than One: Bidirectional Alignment with Cycle Consistency for Exemplar-Free Class-Incremental Learning
Hongye Xu and Bartosz Krawczyk
Continual learning (CL) seeks models that acquire new skills without erasing prior knowledge. In exemplar-free class-incremental learning (EFCIL), this challenge is amplified because past data cannot be stored, making representation drift for old classes particularly harmful. Prototype-based EFCIL is attractive for its efficiency, yet prototypes drift as the
embedding space evolves; therefore, projection-based drift compensation has become a popular remedy. We show, however, that existing one-directional projections introduce systematic bias: they either retroactively distort the current feature geometry or align past classes only locally, leaving cycle inconsistencies that accumulate across tasks. We introduce BiCyc, a bidirectional projector
alignment approach with a cycle-consistency objective. BiCyc jointly optimizes two maps, old-to-new and new-to-old, with stop-gradient gating so that transport and representation co-evolve. Analytically, we show that the cycle loss contracts the singular spectrum toward unity in whitened space, and that improved transport of class means and covariances yields smaller perturbations of classification log-odds, preserving old-class decisions and mitigating catastrophic forgetting. Empirically, across standard EFCIL
benchmarks, BiCyc substantially reduces forgetting and improves accuracy in from-scratch settings, while remaining competitive in the pretrained fine-grained regime.
Read the full abstract ▾
Computer Vision
V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation
Tao Liu, Leela Krishna, Gouti Pavan Kumar et al. (+2 more)
Video-to-
video (V2V) generation is difficult to evaluate because outputs must both follow editing instructions and preserve frame-level correspondence with the source
video, which existing T2V and I2V metrics do not capture. We introduce V2V-Bench, a 11-dimension
benchmark organized into five categories: temporal
alignment, structural fidelity, transformation quality,
video quality, and semantic
alignment. V2V-Bench pairs diverse source
videos with challenging editing tasks and evaluates two commercial models, Grok Imagine and Gemini Veo3, and one open-source model, Open Sora 2. Results show complementary model strengths: Grok performs better on editing fidelity, while Veo3 achieves stronger visual quality. On six V2V-specific dimensions, V2V-Bench reaches a Spearman correlation of 0.905 with human judgments.
Read the full abstract ▾
All authors Tao Liu, Leela Krishna, Gouti Pavan Kumar, Sreeja K, Vishav Garg
Computer Vision
CoFi-UCGen: Coarse-to-Fine Unsupervised Conditional Generation without Label Priors
Shengxi Li, Zhaokun Hu, Ce Zheng et al. (+3 more)
Unsupervised conditional image generation (UCGen) aims to control generation without relying on manually annotated labels, yet remains challenging due to unstructured semantic representations across granularities. To address this, we propose a novel coarse-to-fine UCGen framework (CoFi-UCGen) that explicitly disentangles global semantics from fine-grained variations, which to the best of our knowledge, sets out the first successful attempt for both coarse- and fine-grained conditional generation without any labels. More specifically, we first propose the
adversarial semantic reciprocal learning theory to ensure the semantic consistency and completeness between images and latent spaces. Based on the consistency, we propose the bit-codes to learn a structured coarse-grained latent space, and further prove distinct global semantics inherent from our bit-codes while preserving independent noise sampling for generation. Building upon these bit-codes, we establish a fine-grained semantic basis and introduce a hierarchical modulation mechanism in
diffusion models, by enabling layer-wise injection from coarse conditions to progressively control fine-grained attributes during generation. Extensive experiments demonstrate that without any label priors or pre-trained feature extractors, our CoFi-UCGen consistently outperforms existing UCGen methods in terms of image quality, semantic consistency, and control accuracy, verifying the effectiveness of explicit coarse-to-fine semantic decomposition for the challenging UCGen task.
Read the full abstract ▾
All authors Shengxi Li, Zhaokun Hu, Ce Zheng, Mai Xu, Jingyuan Xia, Si Liu
Computer Vision · Cross-listed from cs.MM
GS-NFS: Bandwidth-adaptive Streaming of Dynamic Gaussian Splats and Point Clouds
Rajrup Ghosh, Haodong Wang, Haoran Hong et al. (+6 more)
Dynamic 3D Gaussian Splatting (3DGS) holds great promise as a 3D
video streaming technology since it can represent complex 3D scenes with high fidelity. In this approach, every frame in a 3D
video represents the environment as a collection of Gaussians with position and other attributes such as scale, rotation, opacity, and color. Frames capture fine details, permit views from any arbitrary perspective, but are an order of magnitude, or more, larger than 2D
video frames. A line of recent work has explored how to compress dynamic 3DGS frames, but these approaches are often slow, in part because their compression techniques are not amenable to efficient acceleration. GS-NFS accelerates dynamic 3DGS compression and decompression on a GPU, to the point where it can encode and decode at full frame rate. It achieves this by developing novel GPU-based parallelizations of existing
algorithms for encoding both positions and attributes of Gaussians. As a result, it is 1-2 orders of magnitude faster than the state-of-the-art in encoding and decoding a frame, while offering competitive compression performance and rendering quality.
Read the full abstract ▾
All authors Rajrup Ghosh, Haodong Wang, Haoran Hong, Eduardo Pavez, Amartya Chaudhuri, Weiwu Pang, Harsha V. Madhyastha, Antonio Ortega, Ramesh Govindan
Computer Vision
Multi-Task Crack Foundation Model for Engineering-Reliable Crack Representation and Topology Preservation in Civil Infrastructure
Blessing Agyei Kyem, Joshua Kofi Asamoah, Eugene Denteh et al. (+1 more)
Reliable crack assessment requires not only accurate pixel-level masks but also connected crack geometry and confidence estimates that remain stable under domain shift. However, existing
segmentation models can achieve high overlap scores while fragmenting cracks, missing fine branches, and providing no calibrated uncertainty. To address this gap, this paper proposes CrackGeoFM, a multi-task framework that combines a frozen visual foundation backbone with crack-specific adaptation for mask prediction, skeleton reconstruction, and uncertainty estimation. The framework integrates a Frequency-Guided Crack Enhancement Module (FCEM) to enhance high-frequency crack cues, a Crack-Domain Feature Adaptation Module (CFAM) to adapt frozen backbone features to crack-domain patterns, and a Structure-Aware Multi-Task Decoder (SMTD) to jointly decode masks, skeletons, and uncertainty. Across 20 crack
datasets, CrackGeoFM achieves state-of-the-art
segmentation, improved
topology preservation, calibrated uncertainty, and effective few-shot adaptation with only five labeled images. These results support reliable, generalizable, and engineering-oriented crack analysis for infrastructure assessment.
Read the full abstract ▾
All authors Blessing Agyei Kyem, Joshua Kofi Asamoah, Eugene Denteh, Armstrong Aboah
Computer Vision
ShotCrop$^3$: Cropping Human-Centric Images into Cinematic Triple-Shot Compositions
Dehong Kong, Lina Lei, Lingtao Zheng et al. (+10 more)
Prior work on aesthetic composition typically produces a single aesthetically pleasing crop, overlooking the narrative value of composing multiple shots from one scene. In practice, multi-shot composition is critical for downstream creative workflows: commercial posters often require multiple crops with different emphases (e.g., context, subject, and
emotion/product details) to present key story beats. Therefore, we propose Triple-Shot Compositions (TSC), a composition task that generates a three-shot set – establishing, medium, and close-up – from a single human-centric image, each paired with a brief shot description to support visual narration. To learn TSC with limited expert annotations, we introduce ShotCrop which undergoes a three-stage training process: it first applies Chain-of-Thought supervised fine-tuning to establish basic
reasoning and aesthetic shot-cropping skills, then performs semi-supervised fine-tuning with high-confidence pseudo labels to further enhance aesthetic capability, and is finally optimized with Group Relative Policy Optimization for ShotCrop (GRPO-S) using a composite reward tailored for it. Specifically, our pseudo-labeling strategy combines MLLM-based scoring, aesthetic assessment, and CLIP similarity to retain high-confidence training signals. In addition, we present TSC-Bench, a
benchmark of 1.2k expert-annotated test cases. Notably, ShotCrop achieves an average improvement of 2.82 times over GPT-5 in shot localization accuracy.
Read the full abstract ▾
All authors Dehong Kong, Lina Lei, Lingtao Zheng, Chenyang Wu, Ailing Zhang, Xinran Qin, Teng Ma, Jiaqi Xu, Zhixin Wang, Zhikai Chen, Xuecheng Qi, Renjing Pei, Fan Li
Computer Vision
KV-Control: Parameter-Efficient K/V Injection for Trajectory-Controlled Text-to-Motion
Tengjiao Sun, Pengcheng Fang, Xiaoyu Zhan et al. (+4 more)
Text-conditioned 3D human motion models now synthesize plausible motions from
prompts, but practical animation and
embodied-
agent workflows rarely stop at text: a character may need to follow a sketched root path, hit an end-effector target, or satisfy a multi-joint trajectory while still preserving the gait, style, and intent described by
language. This exposes a control trade-off. A trajectory controller should be precise without overwriting the pretrained text-conditioned motion prior, yet existing solutions either duplicate large portions of the generator to regain per-layer control access or move much of the cost to test-time optimization. We introduce KV-Control, a compact
attention-side control interface for frozen masked text-to-motion
transformers. The key idea is to make
geometric constraints available as
memory inside self-
attention rather than injecting them through a global pose
token or enforcing them only at the output side. To support this interface, we co-design a part-tokenized motion substrate and controller: PartVQ learns anatomy-aligned part codebooks, T-Concat exposes each frame–part
token as an
attention-addressable site, and KV-Control injects control-conditioned key/value
memories at every self-
attention layer while preserving the pretrained query stream, text cross-
attention, FFN, and all backbone weights. The resulting adapter adds only trainable injection parameters atop a shared trajectory encoder, yet tracks root and multi-joint constraints with sub-centimeter accuracy under the inherited refinement protocol while retaining text-conditioned motion quality. KV-Control reframes trajectory conditioning as lightweight
memory retrieval, providing a small, precise, and transparent control interface for text-to-motion generation.
Read the full abstract ▾
All authors Tengjiao Sun, Pengcheng Fang, Xiaoyu Zhan, Yanwen Guo, Dongjie Fu, Xiaohao Cai, Hansung Kim
Computer Vision
What's Under the Skin? Estimating Swine Body Condition
Mk Bashar, Kuljit Bhatti, Gary Rohrer et al. (+3 more)
Sow body condition is an important indicator for growers as it has a large impact on lactation performance and piglet survival. However, body condition measures used during production, such as visual scoring and calipers, correlate poorly with underlying tissue composition. Ultrasound scans can provide direct measurements of subcutaneous backfat thickness and loin muscle depth, but their operation is labor intensive and not scalable for production. We present PigFormer, an end-to-end two-stage system that takes raw depth frames from a ceiling-mounted RGB-D camera and predicts subcutaneous backfat thickness, loin muscle depth, and total tissue thickness at the last rib. Stage 1 is a
geometric front-end that converts raw depth into a standardized height map via SAM3-to-MaskDINO
segmentation distillation, ground-plane removal, and orientation normalization. Stage 2 is a Slice
Attention Encoder that treats each height map as a sequence of cross-sectional slices and captures spatial relationships along the full dorsal surface. On a multi-site
dataset of 319 sow and gilt instances from two facilities, PigFormer achieves 2.43 mm backfat MAE and 3.87 mm overall MAE. It outperforms strong single-stage ResNet-18 and ViT-small baselines. PigFormer offers a practical path toward continuous, automated, non-contact body condition monitoring in commercial swine production. Code is available at https://github.com/iambashar/Pigformer.
Read the full abstract ▾
All authors Mk Bashar, Kuljit Bhatti, Gary Rohrer, Madonna Benjamin, Tami Brown-Brandl, Daniel Morris
Computer Vision
HDST-GNN: Heterogeneous Dynamic Spatiotemporal Graph Neural Networks for Multi-Object Tracking in UAV Aerial Imagery
Phillip Jiang
Multi-object tracking (MOT) from
UAV imagery presents unique challenges: altitude varies across sequences, objects are small and densely packed, and frequent occlusion causes identity switches. Existing graph-based trackers assume fixed spatial context and treat all objects uniformly, ignoring the heterogeneous lifecycle states of detections, active tracklets, and lost targets. We propose HDST-GNN, a Heterogeneous Dynamic Spatiotemporal Graph Neural Network with three novel contributions. First, Altitude-Adaptive Edge Construction estimates a camera-altitude proxy from mean object area and adjusts the graph connectivity radius accordingly. Second, Heterogeneous Node Representation models detections (Type-D), confirmed tracklets (Type-T), and lost tracklets (Type-L) as distinct node types with dedicated projections and typed edge relations. Third, Occlusion-Gated Temporal Aggregation gates each node's
attention contribution by its occlusion confidence, preventing occluded nodes from corrupting neighbour
embeddings. HDST-GNN is trained end-to-end with a differentiable Sinkhorn head using joint cross-
entropy and triplet loss. On VisDrone2019-MOT with oracle detections, HDST-GNN achieves 94.51% MOTA and 97.24% IDF1, outperforming SORT by +5.0 MOTA points and reducing identity switches by 81%. With real YOLOv8n detections, HDST-GNN reduces identity switches by 49% vs. SORT. Ablation studies confirm the independent contribution of each component.
Read the full abstract ▾
Computer Vision
BMCR: Adaptive Backbone Module Composition via Reinforcement Learning for Remote Sensing Object Detection
Wenlin Liu, Xikun Hu and Ping Zhong
In remote sensing object detection, Convolutional Neural Networks (CNNs) excel at capturing local details while
Vision Transformers (ViTs) are better at global context modeling. However, existing detectors typically rely on a single fixed backbone or a manually designed hybrid architecture, and thus fail to adaptively exploit these complementary strengths across inputs of diverse complexity. To address this limitation, we propose Backbone Module Composition via Reinforcement Learning (BMCR). BMCR dynamically assembles input-adaptive inference paths from reusable modules decomposed from off-the-shelf CNN and ViT backbones. To enable such cross-family composition, we first construct an extensible module toolbox. Specifically, we decompose representative CNN and ViT backbones into reusable functional modules and encapsulate each module with explicit structural, semantic, and computational metadata for compatibility-aware assembly. To bridge the gap between grid-based CNN features and
token-based ViT representations, we design a lightweight Optimal Transport (OT) based transition interface that ensures distribution-aware
alignment while respecting spatial consistency. The backbone composition process is then formulated as a sequential decision problem, in which a policy network progressively selects task-relevant modules according to intermediate multi-scale observations. To stabilize the joint optimization of reusable modules and the routing policy, we further develop an Adaptive Module Cooperative Optimization (AMCO) strategy that coordinates module updating, routing exploration, and reward assignment during training. On DOTA-v1.0, DOTA-v1.5 and DIOR-R, BMCR achieves 79.31%, 73.41% and 71.86% mAP, respectively, surpassing strong static and dynamic baselines by up to 2.5 points while maintaining competitive efficiency.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.GR
Monte Carlo Steklov Operators for Large-Scale Geometry Processing in the Wild
Arman Maesumi, Tanish Makadia, Aruna Anderson et al. (+3 more)
Intrinsic methods fill the default toolbox for geometry processing on meshes. Intrinsic operators, in particular the
Laplacian, underlie methods that require invariance to isometry and have hence been employed in many
algorithms for shape analysis, learning, and editing. However, intrinsic methods are predicated on assumptions that quickly become brittle when working with in-the-wild geometry, where (i) mesh quality is not guaranteed, and (ii) many meshes are modeled with multiple connected components. In such settings, volumetric constructions are better-defined, since restrictions on surface
topology can be relaxed. This paper presents a Monte Carlo method for estimating the Dirichlet-to-Neumann (DtN) operator – a boundary-to-boundary volumetric operator – and its associated Steklov eigenmodes. We build on recent developments in Monte Carlo geometry processing by casting this boundary operator itself as the subject of estimation. The DtN operator, defined through a volumetric
stochastic process, is then generalized to the exterior domain, where it couples disconnected components through the surrounding ambient space. We show that our method is orders of magnitude faster than existing boundary-element approaches for computing Steklov spectra while remaining robust to poor triangulations, high-resolution meshes, and multi-component geometry. To demonstrate this scalability, we compute interior and exterior Steklov eigenspectra for approximately 450,000 shapes from the uncurated Objaverse
dataset. We incorporate these operators into Steklov-CLIP, a mesh-based neural network that uses volumetric spectral operators for large-scale contrastive 3D representation learning. The resulting network learns semantically meaningful global and dense shape representations, illustrating that geometrically-principled volumetric operators can be made practical at the scale of modern 3D
datasets.
Read the full abstract ▾
All authors Arman Maesumi, Tanish Makadia, Aruna Anderson, Oras Phongpanangam, Justin Solomon, Daniel Ritchie
Computer Vision
UltraVR: A Diagnostic Ultra-Resolution Image-VQA Benchmark for Evidence-Grounded Reasoning
Gexin Huang, Yanting Yang, Myeongkyun Kang et al. (+6 more)
Vision-
language models (VLMs) excel on visual question answering and
multimodal reasoning benchmarks. Yet their capability on ultra-resolution images - where critical evidence is tiny, subtle, spatially distant, or distributed - remains unclear. Existing evaluations largely report final-answer accuracy, offering limited insight into whether models acquire and integrate the necessary visual evidence. We introduce UltraVR, a diagnostic
benchmark for evidence-grounded visual
reasoning over ultra-resolution images. UltraVR spans four high-value scenarios: CCTV surveillance, remote sensing (RS), whole-slide image (WSI) pathology, and industrial anomaly detection (AD). These domains pose complementary challenges: fine-grained object grounding in crowded CCTV scenes, long-range spatial comparison in RS, multi-scale evidence
navigation in WSI, and subtle irregularity detection in repetitive industrial layouts. Beyond standard QA triples, each instance includes a structured ground-truth chain of thought with step-level questions, intermediate answers, and
reasoning labels. These labels decompose
reasoning into evidence grounding, local perception, quantification, evidence integration, and decision inference, enabling process-level diagnosis over black-box scoring. Using UltraVR, we evaluate frontier VLMs and show that current models remain far from reliable on ultra-resolution
reasoning. Importantly, the structured annotations allow us to localize failures across the visual-to-decision pipeline: errors concentrate in evidence grounding and local perception, while downstream inference often recovers when intermediate visual facts are supplied. These findings demonstrate UltraVR as a diagnostic testbed for measuring not only whether VLMs answer correctly, but where their ultra-resolution
reasoning process breaks.
Read the full abstract ▾
All authors Gexin Huang, Yanting Yang, Myeongkyun Kang, Beidi Zhao, Jun Zhou, Chen Zhou, Gang Wang, Zu-hua Gao, Xiaoxiao Li
Computer Vision
Dual Feature Decoupling for Fine-Grained OOD Detection
Xiaokun Li, Yaping Huang and Qingji Guan
Out-of-distribution detection (OOD) is an indispensable technique when applying machine learning models to real-world scenarios. Most existing OOD detection methods have been developed under the idealized assumption of large inter-class distributional differences, while largely overlooking fine-grained tasks characterized by subtle variations, such as medical image classification and vehicle recognition. The high visual similarity among fine-grained subcategories, together with the interference of background factors, makes OOD detection extremely challenging. To tackle this problem, we propose a novel Dual Feature Decoupling Network (DFDNet), which addresses fine-grained OOD detection from the perspective of feature disentanglement. The proposed DFDNet comprises two key components: a spatial-frequency decoupling module and a reconstruction-guided decoupling module. The spatial-frequency decoupling module is designed to preserve content features that are discriminative for classification while suppressing task-irrelevant style information. On the other hand, the reconstruction-guided decoupling module introduces a novel pixel-level
adversarial reconstruction task to further remove low-level, non-discriminative information and enhance category-specific high-level semantic representations. Extensive experiments demonstrate that our method achieves competitive performance improvements on multiple
datasets.
Read the full abstract ▾
Computer Vision
Noise-Aware Visual Representation Learning for Medical Visual Question Answering
I Putu Adi Pratama, Bahadorreza Ofoghi, Atul Sajjanhar et al. (+1 more)
Medical visual question answering (Med-VQA) has strong potential for clinical decision support by enabling AI models to interpret medical images and answer clinically relevant queries. Recent approaches typically connect off-the-shelf
vision encoders with large
language models (LLMs) through lightweight mapping networks to reduce computational cost. However, these methods often overlook the importance of handling noise and small irrelevant changes in visual representations. To address these challenges, we propose a noise-aware Med-VQA framework that incorporates a denoising autoencoder before visual
embeddings are mapped into the input space of an LLM. The denoising autoencoder is pretrained to reconstruct clean visual
embeddings from corrupted inputs, encouraging the model to learn robust visual representations that are less sensitive to noise. The resulting
embeddings are then projected into the
language model
embedding space using a multi-layer perceptron (MLP), forming visual prefix
tokens that provide image information to the LLM. To enable efficient adaptation without full retraining, we employ parameter-efficient fine-tuning using low-rank adaptation (LoRA). The proposed method is evaluated on the SLAKE and PathVQA
benchmarks. Experimental results show improved robustness to noisy input
embeddings while maintaining competitive clean performance across multiple evaluation criteria. These findings suggest that learning more robust visual representations can enhance Med-VQA performance and robustness.
Read the full abstract ▾
All authors I Putu Adi Pratama, Bahadorreza Ofoghi, Atul Sajjanhar, Shang Gao
Computer Vision · Cross-listed from cs.LG
What Objects Enable, Not What They Are: Functional Latent Spaces for Affordance Reasoning
Rohan Siva, Neel P. Bhatt, Yunhao Yang et al. (+6 more)
Existing
robot planning systems rely on appearance-based
reasoning, where visual observations are encoded into latent spaces organized around object appearances (e.g., recognizing a "cart" based on how it looks). However, planning requires
reasoning about task-relevant functionalities of objects (e.g., whether an object is "movable"), which appearance-based latent spaces do not capture. As a result, existing approaches struggle to generalize to novel
robot-object interactions. We address this limited generalizability through affordance
reasoning, enabling planning based on task-relevant object functionalities instead of appearance alone. We introduce A4D, which maps visual observations into a shared latent space structured around affordances (e.g., "movable"). By projecting visual observations into this functional latent space and measuring their proximity to affordances, A4D infers functionalities relevant to the observed object. Furthermore, we introduce an affordance discovery mechanism that expands the latent space to handle unseen scenarios where existing affordances are insufficient. A4D uses proximity in the functional latent space to quantify uncertainty in affordance inference and selectively triggers affordance discovery. We evaluate A4D across several planning tasks involving diverse and unseen affordances. A4D achieves 94% inference accuracy on existing affordances outperforming state-of-the-art approaches by over 15% points, improves new-affordance inference accuracy from 70% to over 90% with fewer than 10% of the original training data, and enables 100x faster inference. Code,
videos, and data available at: https://A4Dance-
reasoning.github.io.
Read the full abstract ▾
All authors Rohan Siva, Neel P. Bhatt, Yunhao Yang, Seoyoung Lee, Nishant Gadde, Christian Ellis, Alvaro Velasquez, Zhangyang Wang, Ufuk Topcu
Computer Vision
Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
Mohammad Mahdi Abootorabi, Omid Ghahroodi, Anas Madkoor et al. (+4 more)
Despite the rapid progress of
Vision-
Language Models (VLMs), the field lacks
benchmarks that rigorously diagnose their true
reasoning abilities and chart meaningful progress toward human-like
multimodal intelligence. Most existing evaluations focus on piecemeal or disconnected tasks, obscuring critical cognitive weaknesses and providing little insight for targeted improvement. To address this gap, we introduce BloomBench, part of the Almieyar benchmarking series, the first cognitively human-grounded, bilingual (English-Arabic)
multimodal benchmark for VLMs. Grounded in Bloom's Taxonomy, BloomBench systematically evaluates six levels of cognition (Remember, Understand, Apply, Analyze, Evaluate, Create) through carefully designed image-question-answer tasks. Built with a semi-automated pipeline and validated through a stratified hybrid quality assurance protocol, it ensures scalability, cultural inclusivity, and linguistic fidelity. Leveraging this framework, we conduct a comprehensive study of state-of-the-art VLMs to diagnose their cognitive profiles. Our analysis reveals a sharp cognitive asymmetry: while state-of-the-art models achieve strong performance ceilings in semantic understanding, they struggle substantially with factual recall and creative synthesis. This demonstrates that current general
multimodal proficiency masks deeper limitations in specific cognitive layers. Furthermore, our study highlights a critical performance gap between Arabic and English, exposing limitations in current cross-lingual
multimodal reasoning. These findings establish a foundation for developing more cognitively aligned and inclusive VLMs. The
benchmark framework and
dataset is available at: https://github.com/qcri/Almieyar-Oryx-BloomBench.
Read the full abstract ▾
All authors Mohammad Mahdi Abootorabi, Omid Ghahroodi, Anas Madkoor, Marzia Nouri, Doratossadat Dastgheib, Mohamed Hefeeda, Ehsaneddin Asgari