Computer Vision
Latent-Frequency Validity: Fast Spectral Editing with Screened Video-VAE Transfer Operators
Bowen Xue, Jiafeng Xiong and Xin Quan
Direct spectral editing in
video-VAE latents can control noise, flicker, smoothness, and frequency content without a decode–filter–reencode pass. However,
video VAEs may redistribute pixel-space frequency bands across latent channels, and latent edits can disrupt VAE round-trip dynamics. We introduce latent-frequency validity (LFV), which learns a compact VAE-specific spectral response and deploys it only when it improves decoded-target fidelity without worsening round-trip drift. LFV follows a validation-selected path from a diagonal per-frequency calibrator (C1) to full channel mixing (CM), making cross-channel capacity a controllable per-edit resource. Across 544 VAE–edit
cells spanning six spectral families, LFV emits 423 cheap operators: 277 are handled by C1, while 146 (34.5% of emitted operators) require channel mixing. On the primary 120-
cell radial sweep, 99/100 emitted operators pass source-
video-grouped held-out evaluation. Across five additional filter families, all 323 emitted operators pass held-out evaluation. Fully frozen OpenVid-fitted operators, including the validation-selected path coefficient, pass all 20 tested CogVideoX and HunyuanVideo generated-domain
cells without adaptation. The selected response matches direct latent-filter latency and is about $3\times$ faster than pixel filter–reencode. The resulting maps reveal distinct VAE regimes, including strongly channel-coupled CogVideoX responses and a sharp Open-Sora high-band stability frontier.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning
Yuyao Sun, Tao Deng, Shuang Li et al. (+3 more)
Multimodal large
language models (MLLMs) achieve strong performance across diverse
vision-
language tasks, but their efficiency is limited by the cost of processing numerous visual
tokens. Visual
token pruning can reduce this cost, but requires accurate
token importance estimates. Recent studies have demonstrated that text-to-
vision attention from middle
language model layers can effectively guide visual
token pruning, typically using
attention from a predefined middle layer to select the visual
tokens to retain. Two problems therefore remain. First, our analysis shows that the layer whose
attention is most responsive to the question varies substantially across samples, making a fixed layer suboptimal. Second, obtaining
attention from the appropriate middle layer requires processing numerous visual
tokens through several
language model layers, by which point considerable computation has already been spent. To address both problems, we propose Middle-layer
Attention Prediction (MAP), which uses Question Contrastive Teacher Selection to identify a sample-specific teacher layer by contrasting
attention under the original and reference questions, and distills
attention from the selected layer into a lightweight predictor that estimates visual
token importance from multi-modal input features. During inference, MAP combines the predicted importance scores with a diversity criterion to prune visual
tokens before the first
language model layer. Thus, MAP requires no
attention maps for
pruning and remains compatible with existing inference acceleration techniques. Across ten
benchmarks on LLaVA-NeXT-7B, MAP retains 97.5% of the unpruned model performance with only 5.56% of the visual
tokens, yielding a 3.09x end-to-end speedup.
Read the full abstract ▾
All authors Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, Minjun Yu
Computer Vision
Binding Biometrics with AI Agent Identifiers for Delegation of Authority
Joseph Geo Benjamin, Anil K Jain and Karthik Nandakumar
The proliferation of agentic artificial intelligence (AI) systems has raised serious questions about the accountability for tasks performed by AI
agents. Ideally, an AI
agent must not be allowed to perform critical tasks without explicit authorization by a human operator. Since biometric recognition is one of the most reliable approaches for authenticating individuals, it has the potential to enable authenticated delegation of authority to AI
agents. In this work, we present a framework called BIND, which leverages ideas from the field of biometric cryptosystems, to securely bind biometric data of the human user to the AI
agent identity (ID) and authority scope (task-specific constraints) at the time of
agent authorization. This
token/identifier can be presented by the AI
agent to an Identity Auditor, who simultaneously performs biometric authentication and recovers the
agent ID and scope, thereby enabling real-time user authentication and establishing a non-repudiable proof of human control and delegation of authority. We also provide a practical implementation of the proposed BIND framework based on face features extracted using standard deep neural network models. To facilitate this implementation, we propose a feature adaptation module that transforms real-valued feature
embeddings into fixed-length binary representations suitable for a fuzzy commitment construct based on turbo error correcting codes. Experiments demonstrate the practical feasibility of the proposed face cryptosystem, achieving a True Match Rate of $96\%$ at zero False Match Rate and supporting $1024$-bit
agent tokens.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibration
Harshitha Rajaprakash, Aditeya Prajapati, Rong Xue et al. (+2 more)
Vision-
language-action policies often fail under deployment-time distribution shifts such as clutter, distractor objects, lighting changes, novel objects, altered initial states, and reworded instructions. Hidden-state-based risk probes combined with functional conformal prediction can detect rollout failures, but their reliability depends on calibration data matching deployment conditions. We introduce SAFECAST, which leverages contrast set perturbations to improve hidden-state probe training and calibration for deployment time shift. SAFECAST statistically significantly improves failure detection ROC-AUC scores over a state of the art baseline in both real-world DROID and LIBERO simulation experiments across multiple VLM backbones. We further find that SAFECAST benefits most when both visual and
language contrast set perturbations are used to augment data, and that with contrast set perturbations, sim-to-real calibration leads to better probes than using real rollout data only.
Read the full abstract ▾
All authors Harshitha Rajaprakash, Aditeya Prajapati, Rong Xue, Abrar Anwar, Jesse Thomason
Computer Vision
SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models
Sirun Li, Minghao Liu, Ling Dai et al. (+4 more)
Multimodal large
language models (MLLMs) make grounded predictions in real-world scenes by combining visual and textual cues, yet existing
benchmarks rarely reveal how they arbitrate between these evidence sources when they conflict. We introduce SIGNPOST-Bench, a controlled counterfactual
benchmark for evaluating text-
vision conflict resolution. Each source image is transformed into a counterfactual quintuplet of Original, Blank, Similar, Random, and
Adversarial variants. Synthetic, localized scene-text interventions are designed to preserve non-textual content, enabling paired measurements of changes in localization performance and directed shifts toward geographic targets introduced by conflicting text. SIGNPOST-Bench contains 5,111 counterfactual groups and 25,555 image variants from four
datasets. We evaluate 20 MLLMs from seven providers. Compared with Original images,
Adversarial variants raise median localization error from 282 km to 1,347 km, a 4.8-fold increase. Among geocodable
adversarial samples, 6.5-20.1% of predictions lie less than 50 km from the injected target across models, and every evaluated model exhibits a positive mean paired reduction in target distance from Blank to
Adversarial. Compatible, unrelated, and conflicting text replacements produce distinct effects on model predictions, while clean-input localization performance does not fully predict robustness to conflicting text. These results establish visual geolocation as a continuous diagnostic of scene-text arbitration and provide a controlled framework for evaluating how MLLMs resolve conflicting
multimodal evidence.
Read the full abstract ▾
All authors Sirun Li, Minghao Liu, Ling Dai, Yong Li, Haoxin Lyu, Junting Zhou, Fan Zhang
Computer Vision · Cross-listed from cs.LG
Transferable Dual-Stream Representations for Mesoscale-Preserving Sea Surface Temperature Downscaling
Parth Doshi, Priyanka Aravindan, Vaishnav Vaidheeswaran et al. (+2 more)
Deep learning models for scientific spatio-temporal downscaling often minimize reconstruction error while failing to preserve physically meaningful multi-scale structure. For sea surface temperature prediction, this can yield outputs that are numerically plausible yet overly smooth, missing mesoscale variability critical to regional ocean dynamics. Existing methods often focus on pixel-wise objectives or single-context conditioning, which limits their ability to preserve spectral fidelity and generalize across regions. To address this, we propose EddyFlow, a representation learning framework for kilometer-scale sea surface temperature downscaling that balances predictive accuracy, scale-dependent structure, and regional generalization. EddyFlow is trained on the Gulf of St.~Lawrence and evaluated in zero-shot and few-shot settings on the Bay of Fundy and the Gulf of Mexico. EddyFlow demonstrates that physics-informed representation learning reduces zero-shot RMSE by 21%, achieves up to 85.6% skill relative to persistence on unseen domains, and maintains near-ideal spectral fidelity with a PSD ratio of $\approx 1.00$.
Read the full abstract ▾
All authors Parth Doshi, Priyanka Aravindan, Vaishnav Vaidheeswaran, Md Mahbub Alam, Gabriel Spadon
Computer Vision
OmniVR: Audio-Video Conditional Generation for Archival Footage Restoration
Xin Lu, Zihao Fan, Jie Huang et al. (+4 more)
Archival footage often suffers from coupled visual and acoustic degradations, yet most restoration systems process the two modalities separately. To address this problem, we present OmniVR, the first systematic framework for joint audio-
video restoration, covering data construction, model adaptation, efficient inference, and evaluation. We construct a high-quality audio-
video corpus with detailed captions and use a joint degradation pipeline to produce aligned clean and degraded pairs. Using these pairs, we adapt a pretrained text-to-audio-
video model (T2AV) by introducing degraded audio-
video conditions (TAV2AV), then progressively replace sample captions with a fixed restoration
prompt while retaining caption/null rehearsal. The resulting AV2AV model requires no user-provided text. Under a compatible residual-learning model, we prove that this condition-annealing schedule reduces gradient variance and expected restoration risk relative to direct fixed-
prompt adaptation at the same training budget. For efficient deployment, OmniVR-Flash combines reduced-resolution
video conditioning, MeanFlow-based one-step
distillation, and Turbo VAE, achieving approximately 38 fps at 1K and 18 fps at 2K on a single B200 GPU. We further introduce OmniVRBench to evaluate four complementary dimensions: visual quality, audio quality, temporal consistency, and audio-visual synchrony. OmniVR achieves state-of-the-art results on public
benchmarks and OmniVRBench. Data, code, and model weights will be released. Project Page: https://xin1u.github.io/OminiVR_PAGE/
Read the full abstract ▾
All authors Xin Lu, Zihao Fan, Jie Huang, Mingchen Zhong, Hexin Zhang, Xueyang Fu, Zheng-Jun Zha
Computer Vision
PADFormer: Pose-agnostic Anomaly Detection from Sparse View Images
Ruiqi Wang, Yiming Qian, Fenggen Yu et al. (+4 more)
Pose-agnostic Anomaly Detection (PAD) remains challenging as anomalies can appear under arbitrary viewpoints, requiring methods to handle significant pose variations. Existing approaches rely on complex 3D reconstruction, which are computationally expensive and require extensive multi-view data. We propose PADFormer, a novel image-space approach that leverages
Vision Transformer (ViT) to directly reconstruct anomaly-free versions of query images while preserving pose information. Our key insight is to adapt cross-view masked reconstruction for anomaly detection through training exclusively on normal data, combined with dynamic patch selection and spatial
alignment mechanisms that enable effective learning from sparse reference views under significant pose variations. During inference, we perform multiple forward passes with different masking patterns to generate an ensemble of anomaly-free reconstructions, ensuring comprehensive coverage of the query image. Anomalies are detected by comparing these reconstructions with the query image. PADFormer achieves state-of-the-art results on the PAD
benchmark while maintaining comparable performance on classic few-shot anomaly detection (FSAD) tasks, demonstrating superior efficiency and generalization without requiring 3D reconstruction.
Read the full abstract ▾
All authors Ruiqi Wang, Yiming Qian, Fenggen Yu, Yuxuan Lu, Dakuo Wang, Hao Zhang, Jing Huang
Computer Vision · Cross-listed from cs.RO
SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulation
Nie Lin, Takehiko Ohkawa, Sijin Chen et al. (+10 more)
Recent years have witnessed an explosive trend of scaling ego-centric human
videos for
robot manipulation, yet it remains unclear which data actually benefits dexterous manipulation. We present SiMDex, a similarity-based data mining framework that casts human data selection for VLA post-training in dexterous manipulation as a recommendation problem. For each
robot demonstration, SiMDex runs a three-stage recall-ranking-re-ranking cascade over ~32M
egocentric human samples: recall and ranking select a task-relevant subset in a morphology-agnostic action space, and re-ranking verifies it on pixels, with no change to VLA architecture or training. Against a strong baseline trained with an equal amount of randomly sampled human data, SiMDex uses only ~1.49M mined samples (<5% of the pool) yet improves the overall success rate from 47.7% to 61.7%, showing that selective curation outperforms indiscriminate data mixing.
Read the full abstract ▾
All authors Nie Lin, Takehiko Ohkawa, Sijin Chen, Ruoshi Wen, Zhuohang Li, Liqun Huang, Zhengming Zhu, Yiming Bao, Yunfei Li, Minjie Cai, Xiao Ma, Wei Xu, Yoichi Sato
Computer Vision · Cross-listed from cs.AI
Adversarially Robust Abductive Fusion of Pre-trained Transformer-based Perception Models
Mario Leiva, Yue Ma, Qinru Qiu et al. (+2 more)
Deploying pre-trained perception models in novel environments degrades their accuracy under distributional shift, and assembling them alone does not recover it: combiners such as majority voting trade recall for precision and are brittle to coordinated failures. Prior metacognitive methods learn logical rules that flag a model's errors, but rely on hand-authored domain-knowledge cues (object-size priors,
segmentation masks) that do not transfer to genuinely novel scenes. We show that this metacognitive layer can be learned without any domain knowledge by exploiting vector-space geometry: per-model Label Vector Pools (LVP), built from each model's own training
embeddings, yield error-detection rules from the geometry of detections relative to training-determined prototypes, reaching parity with domain-knowledge rules to within $0.002$ every F1 on test set. Because the approach remains neurosymbolic, these
geometric rules share a single logical framework and can still be complemented by domain knowledge when available. We frame the fusion of multiple imperfect ViT-based detectors as a consistency-based abduction problem solved at test time by an exact Integer Program (IP) and a
polynomial-time heuristic. On an aerial-imagery
benchmark of 15 weather-shifted test sets and six ViT detectors, our domain-knowledge-free layer matches the strongest majority-vote variant on clean data (within $0.005$ F1) and, unlike every majority-vote baseline, retains its performance under a coordinated label-flipping attack: at a $90\%$ flip rate it averages $0.42$ F1 versus $0.35$ for MV-Plurality (a $22\%$ relative gain) and attains the highest F1 on every test set once the flip rate exceeds $0.4$
Read the full abstract ▾
All authors Mario Leiva, Yue Ma, Qinru Qiu, Gerardo Simari, Paulo Shakarian
Computer Vision
TriCLE: Tri-Modal Vision-Language Reasoning for Edge-Deployed Fine-Grained Clustering
Kishor Datta Gupta, Md. Mahfuzur Rahman, Fahad Rahman et al. (+4 more)
Edge platforms used for aerial observation must interpret aircraft imagery under limited
memory, limited compute, and intermittent connectivity. This setting is difficult for standard RGB-only recognition models and general-purpose
vision-
language models, especially when calibrated thermal and
LiDAR aircraft data are unavailable. We present TriCLE, an application-oriented tri-modal
vision-
language system for aircraft taxonomic grouping under edge constraints. From a single RGB aircraft image, TriCLE generates a structure-preserving FLIR-style thermal view and a pseudo-
LiDAR depth projection, then fuses the aligned views with task instructions in a compact Qwen3-VL backbone. The model is aligned to an expert aircraft taxonomy based on propulsion, airframe family, size, design era, and configuration, so its outputs reflect engineering-relevant similarity rather than only surface appearance. We evaluate supervised fine-tuning, rotation-preserving SFT, and three policy-
alignment strategies: GRPO, GSPO, and DAPO. Sequence-level GSPO gives the strongest validation performance, reaching 88.33% validation accuracy and 0.91 weighted F1 on valid aircraft outputs. On a held-out aircraft test partition, GSPO achieves 78.00% accuracy and 0.793 weighted F1 while preserving 94.00% parseable output formatting. After 4-bit quantization and
attention-
memory optimization, the aligned 4B model fits an 8GB deployment target and processes each tri-modal triplet in 1.48 seconds. These results support TriCLE as a practical prototype for interpretable, edge-feasible aircraft grouping, while emphasizing the need for further validation on real aligned thermal and
LiDAR sensor streams.
Read the full abstract ▾
All authors Kishor Datta Gupta, Md. Mahfuzur Rahman, Fahad Rahman, Ahmed Rafi Hasan, Faysal Mehrab Chowdhury, Mohd Ariful Haque, Roy George
Computer Vision
TRNet: Learning with Topographic Priors for VHR Paddy Rice Mapping
Kaiwen Xiao, Chunlong Fu, Liping Zheng et al. (+2 more)
Mapping paddy rice from very high resolution (VHR) imagery in mountainous and hilly regions remains challenging because terrain variations alter optical appearance and increase confusion with visually similar vegetation. To address this issue, we propose TRNet for
multimodal paddy rice
segmentation using 0.5 m GaoJing 1 red green blue (RGB) imagery, a 5 m TanDEM X digital elevation model (DEM), and derived slope information. TRNet employs separate visual and terrain encoders to preserve modality specific representations. At an early encoder stage, the proposed Topographic Energy Spectral Rectification (TESR) performs terrain conditioned low frequency modulation and asymmetric high frequency regulation to suppress steep slope clutter while selectively enhancing rice related cues on compatible low slope regions. The Topography Guided Paddy Structure Decoder (TPSD) further integrates semantic, rice background boundary, and interior cues with coarse topographic context to refine structural predictions. Experiments are conducted on an Area A internal test set and a geographically held out Area B with steeper terrain and lower rice prevalence. TRNet achieves Rice IoU scores of 85.10% and 80.68% on Areas A and B, outperforming the original Dual Encoder U Net by 9.15 and 18.83 percentage points, respectively. Without any adaptation, evaluation on matched August 2024 imagery retains Rice IoU scores of 82.04% and 76.12%. Extensive ablation, slope stratified, and cross year seasonal analyses demonstrate that the improvements arise from effective frequency rectification and structure learning, which reduce steep terrain false positives and low slope rice omissions. These results demonstrate that coarse topography can serve as a stable contextual prior for robust VHR paddy rice mapping.
Read the full abstract ▾
All authors Kaiwen Xiao, Chunlong Fu, Liping Zheng, Yanfeng Su, Yuanhao Xi
Computer Vision · Cross-listed from cs.LG
LiNC: Lightweight Noise Correction via Per-Sample Trust and Gaussian Mixture Modeling
Abhishek Moturu, Babak Taati and Anna Goldenberg
Label noise is common in medical imaging
datasets due to factors such as inter-rater variability, annotation errors, and ambiguous cases. This can severely undermine the reliability and clinical effectiveness of machine learning models trained using those
datasets. To address this challenge, we introduce Lightweight Noise Correction (LiNC), which adds a single trainable trust parameter per training sample and learns when to use the observed label and when to defer to the model during a standard training loop. The key idea is to train using a convex combination of the observed label and the model's own predictive distribution, controlled by a per-sample trust parameter. We show that the gradient of this objective drives trust values in opposite directions for clean versus noisy samples in the early training phase, yielding separable trust distributions. We use a 3-component Gaussian Mixture Model over the trust values to separate them into clean, ambiguous, and noisy cases and then execute a short soft-correction phase on the noisy cases and a final hard correction phase. Experiments on ten 2D
datasets from MedMNISTv2 under label noise of up to 50% show consistent gains in accuracy and strong mislabel detection. LiNC adds negligible asymptotic overhead: the training-time complexity remains dominated by the base network, with additional
memory growing linearly with the size of the training set.
Read the full abstract ▾
Computer Vision
RUTA: Principled Visual Token Allocation via Rate-Utility Optimization
Jian Zou, Xiaoyu Xu, Zhihua Wang et al. (+3 more)
High-resolution images and long
videos provide
vision-
language models with rich context for
multimodal reasoning and fine-grained perception, but the resulting long visual
token sequences make large
language model-side computation and
memory costly. Existing visual
token reducers often operate at prescribed rates, while recent methods adapt
token counts across inputs using method-specific learned thresholds or importance predictors. We introduce RUTA, a principled Rate-Utility
Token Allocation method that performs pre-LLM reduction by jointly learning which
tokens to retain and how many to allocate to each image-query pair. RUTA constructs query-conditioned candidate
tokens and predicts a retention probability for each candidate. During training, these probabilities parameterize independent Bernoulli gates, while their sum provides a differentiable training-time estimate of the
token count for each pair. Retained
tokens serve as anchors that aggregate information from non-retained
tokens according to semantic affinity and spatial proximity. RUTA is optimized with a penalized rate-utility objective that balances downstream task loss against expected
token usage. Averaged across five
benchmarks and measured relative to each backbone's full-
token baseline, RUTA uses only $2.0\%$ and $4.2\%$ of visual
tokens while preserving $88.2\%$ and $94.4\%$ of task performance on LLaVA-NeXT-7B and Qwen3-VL-8B, respectively.
Read the full abstract ▾
All authors Jian Zou, Xiaoyu Xu, Zhihua Wang, Yilin Wang, Balu Adsumilli, Kede Ma
Computer Vision
Radar4D-VLM: Proposal-Grounded Temporal 4D Radar Reasoning Across Frozen Language Models
Jiaju Han, Xuemeng Sun, Qike Zhang et al. (+6 more)
Vision-
language models for autonomous driving primarily rely on cameras and
LiDAR, leaving 4D radar largely unexplored as a standalone perceptual modality despite its robustness to adverse visibility and direct measurement of radial velocity. We introduce Radar4D-VLM, a radar-only temporal
vision-
language model that reasons from ten consecutive 4D-radar point-cloud sweeps without camera or
LiDAR input. Radar4D-VLM extracts geometrically grounded object proposals and organizes radar evidence into a compact hierarchy of object, scene, and kinematic
tokens. A parameter-efficient projector maps these
tokens into frozen
language backbones, while auditable prediction heads jointly model object count, spatial distribution, motion state, collision risk, semantic category, and radial velocity. Radar4D-VLM combines proposal-grounded temporal object tokenization, global scene context, and explicit kinematic
tokens within a unified frozen-backbone interface. On sequence-isolated K-Radar development validation, its Top-64 proposal recall reaches 98.13% at 4 m, exceeding fixed-
lattice and uniform-random controls by 6.40 and 22.83 percentage points, respectively. We further evaluate 24 matched runs spanning eight frozen Qwen, Phi, Mistral, Llama, and Gemma backbones under an identical adaptation budget. The radar-
token interface remains compatible across all five
language-model families, while matched aligned, permuted, and no-
language controls show sensor dependence but no stable direct-head gain from aligned
language supervision. These results establish a reproducible foundation for radar-only
multimodal scene and motion
reasoning while separating interface compatibility from the benefit of
language supervision.
Read the full abstract ▾
All authors Jiaju Han, Xuemeng Sun, Qike Zhang, Xiang Chen, Luwei Yang, Jiahuan Long, Yiwei Wei, Jiujiang Guo, Chengyin Hu
Computer Vision
Teaching Foundation Models to Read mmWave: Pose-Guided Kinematic Representation for Human Behavior Understanding
Duo Zhang, Zhehui Yin, Zhiyun Yao et al. (+8 more)
Large
language model
agents need to perceive human behavior in physical environments. Millimeter-wave (mmWave) radar provides a
privacy-friendly and contactless sensing modality, but radar observations are difficult to align with
language. Existing radar-
language methods often rely on synthetic data or lack explicit supervision for human body structure and motion. We present mmMind, a radar-
language model that uses synchronized 3D pose as training-only supervision. A spatio-temporal radar encoder is pretrained to capture body configuration and motion dynamics, after which the pose head is removed so that inference requires radar alone. The learned radar representations are then aligned with an LLM for behavior captioning and spatio-temporal question answering. We also introduce mmMind-Bench, a real-world mmWave-
language benchmark containing 17.9 hours of recordings from 23 participants across seven indoor environments. Experiments on captioning, question answering, and unseen-action generalization show that mmMind consistently outperforms existing radar-
language baselines, while ablations confirm the importance of pose-guided pretraining.
Read the full abstract ▾
All authors Duo Zhang, Zhehui Yin, Zhiyun Yao, Haotong Qin, Xusheng Zhang, Hongliu Yang, Jianyu Sun, Junzhe Wang, Zizhou Fan, Michele Magno, Daqing Zhang
Computer Vision
Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering
Haotian Xia, Zilin Xiao, Junbo Zou et al. (+2 more)
Video question answering requires models to ground
language queries in visual evidence and, when necessary, reason over that evidence across time. Existing methods typically rely on long textual chain-of-thought rationales, even though many questions can be answered as soon as the relevant object, action, or frame is localized. We propose Dynamic Latent
Reasoning (DyLaR), which first grounds a question in a short block of perception latents (continuous hidden states that encode query-relevant visual evidence), and then adaptively decides whether to append
reasoning latents (continuous thoughts that reason over this evidence in latent space) before answering. DyLaR learns this behavior by grounding perception latents in verified visual evidence and distilling verified rationales into
reasoning latents, followed by reinforcement learning that further refines when to reason. Across nine
video benchmarks and four
multimodal language model backbones, DyLaR improves average accuracy over same-backbone baselines while generating fewer than 20
tokens per query. On Qwen3-VL-4B, for example, DyLaR improves average accuracy over Qwen3-VL-4B-Thinking from 54.0 to 58.2 while reducing response length from 1,220.7 to 18.5
tokens per query. Ablations further show that grounded perception latents, rationale-supervised
reasoning latents, and adaptive routing each improve accuracy.
Read the full abstract ▾
All authors Haotian Xia, Zilin Xiao, Junbo Zou, Vicente Ordonez, Hanjie Chen
Computer Vision · Cross-listed from cs.RO
Interpretable Fuzzy Inference for UAV Target Tracking Using Bounding-Box Geometry
Reza Ahmari, Ahmad Mohammadi, Vahid Hemmati et al. (+5 more)
Vision-based guidance of unmanned aerial vehicles (
UAVs) toward unmanned ground vehicles (UGVs) supports cooperative aerial–ground robotics, but reliable continuous yaw estimation from onboard
vision remains challenging because of sensing uncertainty, limited computation, and the need for interpretable control. Existing deep-learning and
geometric-reconstruction approaches often require large
datasets, external localization, or complex modeling assumptions, reducing transparency and deployment suitability on resource-constrained platforms. We present an interpretable fuzzy-inference framework that generates continuous yaw commands from low-dimensional features extracted from YOLO boxes: target centroid location, area, and aspect ratio. No explicit
geometric modeling is required. A Mamdani fuzzy system serves as an interpretable baseline using a shoulder–triangle–shoulder input partition. It is followed by a first-order Takagi–Sugeno model with three antecedent membership terms per input, whose parameters are derived from training-set
quantiles, yielding a compact 27-rule structure. Evaluation uses 6{,}169 labeled samples from a VICON motion-capture environment. Across five randomized train–test splits, the Takagi–Sugeno model achieves a test-set mean absolute error of $0.140^\circ \pm 0.003^\circ$, a root mean squared error of $0.200^\circ \pm 0.008^\circ$, and a maximum absolute error of $1.254^\circ \pm 0.121^\circ$. Within-threshold accuracies are $99.676% \pm 0.270%$ for $\pm1^\circ$ and $100.000% \pm 0.000%$ for both $\pm3^\circ$ and $\pm5^\circ$. Directional consistency between image-plane horizontal displacement and predicted yaw sign reaches $90.254% \pm 0.612%$. These results show that the framework is transparent, data-efficient, computationally lightweight, and suitable for real-time
vision-based
UAV guidance toward mobile ground targets.
Read the full abstract ▾
All authors Reza Ahmari, Ahmad Mohammadi, Vahid Hemmati, Nicholas Edmond, Hossein Z. Saghazadeh, Olusola Odeyomi, Parham Kebria, Abdollah Homaifar
Computer Vision
GEB-Bench: Abstract Structures Told in Many Voices
Tong Zhang, Zhiyuan Shi, Yun Peng et al. (+1 more)
Analogy allows models to apply relational knowledge from one situation to another.
Benchmarks of equivalent representations test different ways of expressing the same underlying problem, whereas analogy often links situations involving different objects and content. To evaluate the analogy ability of models, we introduce GEB-Bench, a
benchmark centered on cross-voice motif matching. A voice presents a motif as a scene, story, mathematical statement, or programmatic skeleton. The first task is identification across these voices. The second is matching scenes to stories or mathematical statements, and stories to mathematical statements, by their shared motif. A complete construction pipeline generates items in these voices from the relation specified for each motif, checks the order and repetition of story units, and assembles paired questions. We evaluate current models on our
benchmark. Results show that matching performance depends on the target voice. The paired scene-to-story study reveals that (1) evaluated models still make matching errors when they separately identify the source and every candidate correctly, and (2) identification improves more consistently than scene-to-story matching across the evaluated Qwen model sizes. These findings show that successful motif identification does not ensure reliable matching across voices.
Read the full abstract ▾
All authors Tong Zhang, Zhiyuan Shi, Yun Peng, Tao Xie
Computer Vision
LoRetta: A Foundation Model and Extensive Dataset for Global-Scale Remote Sensing Dense Image Matching
Siwei Yu, Han Guo, Zhenwei Shi et al. (+1 more)
Dense image matching establishes pixel-wise correspondences and underpins broad applications in computer
vision and photogrammetry. However, extending dense matching to global-scale remote sensing remains challenging because image pairs may differ in acquisition time, season, viewpoint, spatial resolution, and land-cover state. The resulting large
geometric offsets, partial overlap, and intrinsically unmatchable regions make direct dense correspondence prediction unreliable and inefficient. We thus reformulate dense matching as localization-and-registration: first localizing the matchable overlap and affine geometry, then refining dense residuals within the aligned frame. Based on this formulation, we propose LoRetta, a foundation model coupling matchability-aware affine localization with guided dense registration. We also introduce LEVIR-GM, a global-scale multi-temporal optical matching
benchmark with
dataset-native matchability labels (103K aligned, 827K augmented pairs, six continents, five years, 0.5-1024 m resolution). We further establish a unified evaluation protocol for sparse, semi-dense, and dense matchers. On LEVIR-GM, LoRetta achieves an area under the curve (AUC) of 83.3%, outperforming the strongest baseline RoMa v2 by 1.6 points, with larger percentage of correct keypoints (PCK) gains of 6.5 and 8.2 points at 1 and 2 pixels, while reducing inference latency by 47.8%. Astronaut-to-satellite and unmanned aerial vehicle (
UAV)-to-satellite geolocalization experiments further demonstrate its transferability as a reusable
geometric aligner.
Read the full abstract ▾
All authors Siwei Yu, Han Guo, Zhenwei Shi, Zhengxia Zou
Computer Vision · Cross-listed from cs.LG
SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization
Boyao Wang and Zhihan Lei
Mixture-of-experts (MoE) networks pursue specialization through learned routers, gates, and load-balancing losses, yet at matched total-parameter budgets learned routers can underperform equal-weight No-Routing baselines. Is the bottleneck the routing
algorithm, or the
alignment between training-signal granularity and the target categories? We probe the question with SpecDrop, a fixed parameter-free routing scheme: each of $K$ branches receives weight $p_a$ for its assigned category and a small leakage $p_i > 0$ otherwise, merged through a category-independent fixed denominator, with no learned routing parameters and no auxiliary losses; the category label is required at inference. On
vision tasks where each image has one superclass label (CIFAR-100 on ResNet-110; ImageNet-1K on ViT-S/16), SpecDrop reaches 79.23% on CIFAR-100 and 79.89% on ImageNet-1K, exceeding parameter-matched baselines that do not use the label (+4.75 over dense on CIFAR-100; +6.53 over the No-Routing+SE control on ImageNet-1K). These gains quantify what category supervision buys when deployed through routing – not an advantage over label-aware deployments of the baselines: given the same label, masking a dense model's outputs is stronger for accuracy alone (85.2 / 83.7). SpecDrop's contribution is converting the label into trained-in modular structure: 68%/100% branch-category
alignment, and masking gains of 0.00 (CIFAR) / +1.06 (ImageNet) – the output-space restriction is largely internalized during training. On fuzzy partitions, where training units span multiple categories (SlimPajama-6B
language modeling with a 30M
Transformer; SuperNI instruction tuning over Llama-3.2-1B with LoRA), the routing mechanism reduces to the matched No-Routing controls within seed noise, the null our thesis predicts. Granularity
alignment, not
algorithm choice, localizes when routing helps. Code: https://github.com/Beryex/SpecDrop
Read the full abstract ▾
Computer Vision
Advancing Utility Pole and Sign Detection Through Deep Learning
Carl Dickinson and Gaetano Di Caterina
Utility poles are an essential part of the infrastructure used to support power distribution systems and other critical public services. Their regular inspection is crucial to ensure the stability and safety of the electrical grid. A deep learning framework is presented for the automated detection,
segmentation and lean angle estimation of wooden utility poles, and classification of attached electrical warning signs, using ground-level imagery. The system is trained on a custom
dataset of 4,570 annotated images extracted from Google Street View, featuring challenging real-world scenes with visually ambiguous wooden poles lacking distinctive features. The proposed model is based on the Detection
Transformer (DETR), suitably modified and trained on the custom
dataset. The model outperforms standard object detectors (RetinaNet, Faster R-CNN, YOLOv3-Tiny), achieving a mean average precision of 90.43% for pole detection and 88.26% for sign detection. Extending this model with a
segmentation head enables per-instance mask generation, which is then used to estimate pole lean angle. The model accurately estimates lean for 1,367 out of 1,433 test-set poles, with a mean absolute error of 1.01 degrees. Moreover, the custom
dataset created in this work is also made publicly available to be used as a
benchmark.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CR
When Modalities Fail to Tango: Conformal Backdoor Detection in Multimodal Contrastive Learning
Yiming Chen, Kemou Li, Haiwei Wu et al. (+1 more)
Backdoor attacks in
multimodal contrastive learning (MCL) have garnered growing
attention in recent years, as many downstream tasks critically depend on pre-trained MCL models. Existing detection-based defenses predominantly rely on the CLIPScore metric, under the assumption that poisoned pairs exhibit lower semantic similarity between the image and the caption. However, we identify two critical flaws remaining in existing methods: (1) the substantial overlap between CLIPScore distributions of benign and poisoned pairs undermines the reliability of this metric, and (2) fixed-threshold detection cannot provide statistical guarantees for ambiguous samples within overlapping regions. To overcome these limitations, we propose integrating conformal prediction (CP), a statistical framework that quantifies uncertainty through nonconformity scores (NCSs), to establish provable confidence bounds for detecting poisoned image-caption pairs. Building on CP, we introduce CASCADE, a novel two-stage Coarse-to-Fine Conformal
Backdoor Detection framework. The coarse-grained stage uses cross-modality consistency to identify high-confidence benign and poisoned pairs. In the fine-grained stage, a reference set is constructed from high-confidence poisoned pairs, and instance-level NCSs based on text-space similarity are computed for each sample in the unidentified subset. These NCSs measure conformity to the poisoning distribution and enable precise identification of latent poisoned pairs within the unidentified subset. Extensive experiments on the large-scale CC3M
dataset demonstrate that CASCADE achieves an average FPR of 5.79% at 100% TPR and an average AUROC of 0.9867 across diverse attacks, while remaining effective against adaptive attacks.
Read the full abstract ▾
All authors Yiming Chen, Kemou Li, Haiwei Wu, Jiantao Zhou
Computer Vision
ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs
Yang Yang, Qinyu Zhao, Mouxiang Chen et al. (+5 more)
Existing scaling strategies for
Multimodal Large
Language Models (MLLMs) typically expand either model parameters or sequential inference computation, incurring substantial
memory or latency overhead. More importantly, most existing methods fail to alter the rigid, fixed computation allocation between the
Vision Transformer and the Large
Language Model components, limiting task-specific optimization. To address this, we introduce the Parallel
Vision-
Language (ParVL) scaling framework for MLLMs, which scales parallel computation by reusing the existing ViT and LLM backbone parameters across multiple
vision and
language branches. This framework raises a central question: given a fixed backbone parameter budget, how should additional shared-backbone computation be allocated between the
vision and
language modalities? We instantiate each parallel computational stream with branch-specific prefix parameters over a shared backbone, and train the entire model end-to-end via full-parameter supervised fine-tuning on roughly 13B
tokens. We systematically study the computation-allocation trade-off between the ViT encoder and LLM decoder. ParVL improves overall
multimodal performance over same-recipe single-branch baselines, and the best evaluated
vision–
language allocation varies across tasks. Code is available at https://github.com/YangYangGirl/ParVL.
Read the full abstract ▾
All authors Yang Yang, Qinyu Zhao, Mouxiang Chen, Xiaohui Li, Lixin Gu, Wenhai Wang, Hongjie Zhang, Wenwei Zhang
Computer Vision
Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation
Wanli Ma, Jiangwen Lu, Qinmu Peng et al. (+1 more)
Training-free open-vocabulary semantic
segmentation (OVSS) partitions an image into semantically distinct regions based on arbitrary text descriptions, without learning any additional parameters. However, existing methods typically focus on improving visual representations while treating text
embeddings that encode only generic category concepts as fixed classification references. The resulting semantic gap between these generic concepts and the visual representations that capture the specific appearances of target instances often causes incomplete masks and erroneous predictions in non-target regions. Inspired by the symbol-percept correspondence underlying perceptual anchoring, we propose Prototype-Guided Text Calibration (PTC) for training-free OVSS. In the Perceiving stage, PTC selects reliable visual evidence based on initial matching scores to construct category-specific visual prototypes. In the Anchoring stage, PTC uses these prototypes to calibrate their corresponding text
embeddings, with the calibration strength adaptively adjusted based on the amount of visual evidence. Consequently, the calibrated text
embeddings align more accurately with instance-specific visual representations while preserving generic category semantics and open-vocabulary generalization. Moreover, PTC requires neither additional training nor external models and can serve as a plug-and-play module for existing methods. Extensive experiments across eight
benchmarks show that PTC significantly enhances the performance of six representative methods and yields more complete and accurate
segmentation results. These results validate PTC as a simple and effective approach to improving visual-text
alignment.
Read the full abstract ▾
All authors Wanli Ma, Jiangwen Lu, Qinmu Peng, Xinge You
Computer Vision
Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent
Zhen Fang, Yu Zeng, Wenxuan Huang et al. (+17 more)
We introduce
Video-DeepResearch (
Video-DR), extending
multimodal agents from static images to continuous
video streams, a setting that demands dense spatiotemporal grounding coupled with open-web exploration. Preliminary evaluations reveal two critical bottlenecks in current models: (1) modality bias, where
agents bypass visual tools in favor of textual search, and (2) parametric knowledge leakage, where models rely on internal
memory rather than genuine tool-augmented execution. To address these challenges, we propose
Video-DR, featuring a decoupled perception-exploration pipeline with stage-wise tool unlocking that compels exhaustive cross-frame visual grounding prior to web
retrieval. Our framework adopts a two-stage training recipe: supervised fine-tuning followed by Group Relative Policy Optimization (GRPO), enabling autonomous exploration that breaks the imitation-learning ceiling. Furthermore, we curate
Video-DR-Bench, a human-AI collaborative
benchmark comprising 200 complex, multi-hop VQA instances. Empirical results demonstrate that our
Video-DeepResearch-35B-A3B establishes a new state-of-the-art of 64.0% average accuracy, surpassing proprietary Claude-4.5-Sonnet (59.0%) by 5.0 points and significantly outperforming GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%). The 30B-A3B variant achieves 59.3%, competitive with Claude-4.5-Sonnet and demonstrating the effectiveness of our training paradigm even at compact scale. Code: https://github.com/Osilly/
Vision-DeepResearch.
Read the full abstract ▾
All authors Zhen Fang, Yu Zeng, Wenxuan Huang, Yiming Zhao, Shiting Huang, Tianfei Ren, Qi Lu, Qingnan Ren, Qisheng Su, Lionel Z. Wang, Qingyu Yin, Shuang Chen, Zehui Chen, Lin Chen, Zhenfei Yin, Yao Hu, Shaohui Lin, Wanli Ouyang, Shaosheng Cao, Feng Zhao
Computer Vision
JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion
Yicheng Xiao, Wenxun Dai, Xinran Qin et al. (+22 more)
Real-time
video editing requires low-latency causal generation with bounded computational resources while preserving source fidelity and long-term temporal consistency. We present JoyAI-
Video-Edit, a 16B-parameter autoregressive
diffusion framework for real-time, open-ended
video editing without access to future frames or a predefined
video duration. Our method combines chunk-wise autoregressive adaptation, Source-Anchored Distribution Matching
Distillation (SA-DMD), and Long-Horizon Autoregressive
Distillation to reduce train–inference mismatch, preserve source fidelity during two-step generation, and mitigate accumulated temporal drift. Extensive automatic and human evaluations show that JoyAI-
Video-Edit substantially outperforms existing streaming editors and remains competitive with strong offline systems on both short and long
videos. The complete system achieves end-to-end 720p
video editing at approximately 30 FPS on a single Nvidia B200 GPU. Code is available at https://github.com/jd-opensource/JoyAI-
Video-Edit.
Read the full abstract ▾
All authors Yicheng Xiao, Wenxun Dai, Xinran Qin, Lin Song, Maoquan Zhang, Hang Xu, Yukang Chen, Yitong Li, Guohui Zhang, Yuan Zhang, Xuying Zhang, Tommy Zhang, Jianlong Yuan, Peihao Li, Shuai Lu, Siming Fu, Chuyang Zhao, Xin Han, Jie Huang, Wenbo Li, Guoqing Ma, Wei Huang, Xiaojuan Qi, Haoyang Huang, Nan Duan
Computer Vision
UniWorld-Design: From Pixel Generation to Layer-Native Design
Zongjian Li, Zhiyuan Yan, Chenxu Bai et al. (+9 more)
We introduce UniWorld-Design, a framework that redefines image generation from flat pixel synthesis to structured visual composition, with semantic RGBA layers as the atomic units of generation, understanding, and editing. Our key insight is that pixels define how an image is rendered, whereas layers define how an image is created, understood, and edited. Just as human designers create and manipulate visual content through layers rather than raw pixels, UniWorld-Design equips
multimodal generative models with a layer-native design space. UniWorld-Design comprises two models. The Text-to-RGBA (T2RGBA) model generates standalone RGBA assets directly from text. The Image-to-Layer (I2L) model conditions on a finished image, a global instruction and per-layer
prompts, and jointly produces ordered, complete semantic RGBA layers. Its instruction interface supports top-level decomposition, recursive decomposition and targeted extraction, making layering an instruction-addressable operation for agentic editing. Because I2L learns complete semantic objects rather than visible-pixel partitions, its layers stay usable when moved or removed. On the Crello
benchmark, I2L reduces per-layer RGB L1 error by 37% and achieves a 34% relative improvement in Alpha Soft IoU over Qwen-Image-Layered. Separately, T2RGBA achieves the highest CLIP Score, outperforming LayerDiffuse and OmniAlpha.
Read the full abstract ▾
All authors Zongjian Li, Zhiyuan Yan, Chenxu Bai, Chen Chen, Haoxiang Sun, Shaodong Wang, Feize Wu, Shenghai Yuan, Bin Lin, Zheyuan Liu, Yuwei Niu, Li Yuan
Computer Vision
Progressive Learning of a Diffusion-based Inpainting Model for Separating Overlapped Fingerprints
Noor Hussein, Anil K. Jain and Karthik Nandakumar
Overlapped friction ridge patterns are a recurring problem in latent fingerprints recovered from crime scenes and in live-scan scenarios where residual fingerprints on the sensor may corrupt subsequent acquisitions. Existing approaches for separating overlapped fingerprints either rely on rule-based orientation field completion that requires strong domain knowledge or train end-to-end deep neural networks that do not account for domain-specific considerations. This work introduces a
diffusion-based pipeline for separating component fingerprints from an image containing overlapping friction ridge patterns. We formulate the separation problem as an inpainting task and progressively learn a
diffusion model for this task in multiple stages. Starting from a pre-trained Stable
Diffusion model, we progressively incorporate a fingerprint prior, add the ability to complete partial fingerprints, and finally propose overlap-aware inpainting that reconstructs each component print using a
diffusion inpainting model based on multi-channel conditioning. Experiments on two public
datasets demonstrate that component fingerprints reconstructed using the proposed
diffusion-based inpainting method can match with their mated counterparts with very high probability.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Latent Reward Registers for Diffusion Preference Alignment
Yuanshen Guan, Zipeng Feng, Chengru Song et al. (+2 more)
Aligning
diffusion models with human preferences usually relies on a sparse terminal reward evaluated on the final generated samples, which creates a severe temporal credit-assignment problem across the denoising process. We propose Latent Reward Registers, a mechanism that estimates terminal preference directly from intermediate noisy latents. Learnable, position-free register
tokens are appended as an auxiliary read path to a frozen
Diffusion Transformer (DiT), extracting preference signals without altering the generator's hidden states or velocity field. The resulting dense, differentiable reward field spans the full denoising trajectory and supports two
alignment strategies. For training, Reward-Gradient On-Policy
Distillation (RG-OPD) converts this dense reward field into per-step targets at states visited by the current generator, replacing rollout-intensive policy gradients with direct on-policy
distillation. For inference, Reward-Guided Sampling (RGS) steers trajectories with magnitude-matched reward-gradient corrections and no parameter updates. Empirically, at high noise levels (t=0.8) the registers reach the highest pairwise accuracy among the evaluated latent reward models. RG-OPD outperforms online reinforcement learning baselines while reducing GPU hours by up to 33x. RGS achieves significant reward improvement with a favorable reward-quality balance against training-free baselines. Code and weights are to be available at https://github.com/Guanys-dar/latent-reward-register
Read the full abstract ▾
All authors Yuanshen Guan, Zipeng Feng, Chengru Song, Zhiwei Xiong, Peiqin Sun