Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Computer Science Computer Vision

cs.CV Watch Computer Vision

Friday 9 October

Computer Vision

Learning Features, Not Layouts: Feature-Axis Attention as a Transfer Prior for Medical Imaging

Aon Safdar and Mohamed Saadeldin

Natural-image-pretrained foundation models are increasingly adapted to medical imaging, yet transfer remains unreliable under domain shift, modality heterogeneity, and label scarcity. We trace part of this fragility to an overlooked design choice. Standard vision transformers place all global modeling on self-attention over spatial tokens, an object-layout prior well suited to natural scenes but not to medical images, where the discriminative signal lies in feature correlations and texture that recur across scanners while layout does not. We ask whether the axis of global attention, spatial versus feature, is itself a lever for transferable medical representations. Existing channel-aware hybrids confound this question, adding channel attention to a spatially-mixed backbone and leaving the axes coupled and the feature axis unmeasurable in isolation. We design FAD-ViT, a backbone that strictly decouples the two axes, making feature-axis attention the primary global operator and the attention axis a controlled variable. Evaluating comprehensively across 23 classification, segmentation, and low-data tasks under matched tokenization, capacity, and protocol, we find that shifting global attention from the spatial to the feature axis leaves from-scratch performance essentially unchanged, yet markedly strengthens transfer. Identical ImageNet pretraining yields up to roughly three times the transfer gain, with consistent improvements on every held-out segmentation task at up to a quarter the parameter count and far lower compute than standard and channel-attention baselines. The advantage is largest where texture rather than geometry is discriminative, tracking greater reuse of pretrained features. The axis of global attention is thus a design lever for medical transfer, and making it channel-primary is a simple, cheap, and principled choice.
Read the full abstract

Computer Vision

The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs

Qiji Zhou, Junyi Hu, Peiyu Wang et al. (+1 more)

Visual interventions provide a training-free way to reduce hallucinations in multimodal large language models. However, an intervention may have little leverage at some layers and may damage a correct answer even when it changes the output. We investigate where visual interventions retain leverage and when they improve predictions. First, we apply the same visual-residual edit at successive decoder layers, matching its RMS strength across depth. Across six checkpoints from three model families, we observe a persistent drop in intervention effect and define its operational onset for this edit as the Evidence Anchoring Boundary (EAB). With boundaries fixed on calibration queries, held-out effects on disjoint evaluation queries before EAB are $13.78\times$–$26.10\times$ larger than those after it. Comparison with intermediate readouts shows that the edit can lose leverage before answer preferences stabilize. Second, we compare counterfactual subtraction across presence, attribute, and relation questions. Subtraction can correct a misleading preference, leave a decision unchanged, or remove a preference supporting the correct answer. Based on these observations, we develop calibrated selective routing, which preserves the original prediction unless calibration supports an intervention for the question type. It improves full-set attribute accuracy by $1.56$–$1.81$ percentage points across three checkpoints, while universal subtraction is neutral or harmful.
Read the full abstract

All authors Qiji Zhou, Junyi Hu, Peiyu Wang, Yue Zhang

Computer Vision

Learning Representations to Guide Pixel Diffusion Transformers

Zihao Pan, Mengting Chen, Cheng Shi et al. (+6 more)

Visual representations are largely developed for understanding, yet what makes them useful for generation remains unclear. We study this question in pixel diffusion transformers and uncover two key insights. First, better representation quality does not necessarily translate into greater generative utility, as externally aligned semantics can be progressively lost during pixel synthesis. Second, architectural separation with specialized pixel decoders mainly redistributes semantic and spatial modeling, without necessarily yielding representations better suited to generation. These insights suggest that a useful generative representation must be both properly formed and actively exploited. Motivated by these insights, we introduce Representation Guidance (RG), which directly forms and exploits internal representations while retaining a plain Transformer architecture. A low-dimensional bottleneck forces generation-relevant information into a compact representation, which is then explicitly used to guide decoder generation by modifying the flow-matching target according to its induced prediction offset. RG learns clearer semantic and spatial structure while scaling efficiently in both parameters and computation. On ImageNet 256x256, RG-L/16 with only 459M parameters outperforms substantially larger pixel diffusion transformers, while RG-XL/16 achieves an FID of 1.55. RG further reaches an FID of 1.74 at 512x512 and extends to text-to-image generation, achieving 0.89 on GenEval and 87.1 on DPG-Bench.
Read the full abstract

All authors Zihao Pan, Mengting Chen, Cheng Shi, Taihang Hu, Zhengrong Yue, Zikang Wang, Yuan Wang, Wei Li, Sibei Yang

Computer Vision

MotionCoach: Constructing Multimodal Guidance for Video Professional Motion Editing

Siyi Qian, Hanqing Yang, Weihuang Lin et al. (+4 more)

Video editing models achieve strong performance in modifying appearance, backgrounds, styles, and motions, empowered by large-scale training data. Beyond general scenarios, professional motions are significant in performance, sports, and martial arts. This paper focuses on professional motion editing, which involves fine-grained motion technique and temporal structure. Given professional motion names but lacking detailed descriptions, existing models often fail to generate professional motions, such as incorrectly lifting the front wheel under a Stoppie instruction. To address this challenge, we introduce MotionCoach, a trained multimodal agent that uses multi-turn tools and performs self-verification, producing a grounded prompt and a reference clip as reliable and sufficient multimodal guidance to inject external professional motion knowledge into video editing models. To supply training data, we build a data pipeline to construct trajectories. We cold-start the agent on MotionCoach-SFT-10k and apply Group Relative Policy Optimization (GRPO) on MotionCoach-RL-2k with rewards on the motion guidance prompt, the motion outcome video, and the format. For evaluation, we introduce VPMEBench, comprising 314 professional motion editing tasks with fine-grained references and annotations. Extensive experiments show that MotionCoach improves the overall score from 55.1 to 65.6 on Seedance 2.0 and from 51.2 to 64.5 on MiniMax-H3. Learning to construct multimodal guidance enables precise professional motion editing without retraining video editing models.
Read the full abstract

All authors Siyi Qian, Hanqing Yang, Weihuang Lin, Zihao Yang, Yichen Lu, Yuanhong Zheng, Sheng Li

Computer Vision

MVGraph: Benchmarking LVLMs on Evidence Composition across Multi-View Visual Graph

Yingjie Zhu, Han Cheng, Xuefeng Bai et al. (+3 more)

We introduce MVGraph, a benchmark for evaluating large vision-language models (LVLMs) on multi-graph reasoning where the evidence required by a query is distributed across multiple interrelated visual graph views. Although such evidence commonly resides in source-specific graphs, temporal snapshots, or local observations, existing graph reasoning benchmarks remain confined to a single self-contained structure, assessing only graph perception and reasoning while leaving the alignment and composition of evidence across views underexplored. MVGraph comprises 8k instances spanning 8 graph-theoretic tasks, abstracted from two multi-graph scenarios (partial and temporal graph views). Each problem is programmatically constructed and verified to be unsolvable from a single view. Beyond end-to-end evaluation, we decompose multi-graph reasoning into four complementary capabilities—Perception, Alignment, Composition, and Reasoning—and design 29 diagnostic tasks with 64k instances to probe each one, along with golden intermediate representations for evaluating the intermediate process. Across 15 LVLMs, we reveal the difficulty of multi-graph reasoning and identify cross-view edge alignment and relation-level composition as prominent bottlenecks through stage-oriented diagnostics. Meanwhile, temporal constraints and increasing view counts expose further capability boundaries. Code and data are available at https://github.com/AAAndy-Zhu/MVGraph.
Read the full abstract

All authors Yingjie Zhu, Han Cheng, Xuefeng Bai, Youcheng Pan, Yang Xiang, Kehai Chen

Computer Vision

Video Codec-Guided Training-Free Token Compression for Video Large Language Models

Xihua Sheng, Ruijie Liu and Chang Wen Chen

Video large language models (VLLMs) rely on long visual token sequences to understand complex temporal content, which makes visual token compression important for efficient inference. Existing training-free compression methods mainly reduce the number of tokens sent to the language model, but the compression process itself can still require many visual feature comparisons, especially for intra-frame token protection and inter-frame temporal token merging. This paper proposes a video codec-guided training-free token compression method for VLLMs that uses coding information already available in compressed video as a low-cost search prior. The method aligns reference-frame dependencies, motion vectors, and residuals with the visual token grid. Residual information forms compact candidate pools for token protection, while motion information is propagated along reference-frame chains to construct localized candidates for temporal token merging between sparsely sampled frames. Local visual feature similarity comparison is then used within these candidates to make the final compression decisions, without modifying the base VLLM. Experiments on three representative VLLMs and four video-understanding benchmarks show that the proposed method achieves performance comparable to the state-of-the-art training-free baseline FlashVID, and in some settings surpasses it, while substantially reducing the feature comparisons required by token protection and temporal matching. These results demonstrate that codec information can effectively improve the accuracy-complexity tradeoff of video token compression.
Read the full abstract

Computer Vision · Cross-listed from cs.CR

WaveMark: Robust Wavelet-Domain Image Watermarking for Diffusion Models

Jie Cao, Qi Li, Chaklam Cheong et al. (+4 more)

Image watermarking enables provenance tracing for photorealistic images generated by diffusion models. However, state-of-the-art (SOTA) removal attacks can suppress embedded signals with little perceptual degradation. This vulnerability limits the effectiveness of watermarks for intellectual property protection and misuse attribution. We propose WaveMark, a novel wavelet-domain watermarking method that embeds messages in a diffusion model's initial noise. To support robust recovery from perturbed carriers, WaveMark distributes keyed replicas of message bits across spatial positions, wavelet orientations, and scales. Each carrier groups coefficients from four latent channels and encodes a randomized bit through magnitude projection, providing an explicit decoding margin. We evaluate WaveMark with SD2.1 on SDP and MS-COCO Captions under 14 attacks. WaveMark achieves 100.0% bit accuracy (BA) without attacks, while maintaining image quality comparable to the evaluated baselines. Under SOTA MarkNull and MarkNull-A, it retains over 88% BA, outperforming all baselines.
Read the full abstract

All authors Jie Cao, Qi Li, Chaklam Cheong, Yutong Jin, Enze Yu, Zelin Zhang, Jianbing Ni

Computer Vision

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

Kun Wang, Xindi Wu, Sanghyuk Chun et al. (+2 more)

Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to properly respond to visual evidence when input images violate those regularities. It is challenging to disambiguate these prior-induced errors from errors caused by the model missing the underlying visual capabilities. In this work, we use controlled visual in-context learning to encourage the model to focus on the visual evidence, and demonstrate that this alone is frequently sufficient to correct the errors – suggesting that the models in fact do possess the relevant visual capabilities. Specifically, we show that across five vision-centric tasks, demonstrations yield large gains on prior-conflicting counterfactual examples ($+27.2\%$ for Qwen3-VL 235B and $+15.1\%$ for Qwen3-VL 32B) while leaving accuracy on prior-aligned examples nearly unchanged. These findings generalize across five VLM families with gains from $+9.7\%$ to $+30.3\%$. We analyze visual representations and find that demonstrations indeed redirect attention toward grounding cues while reducing reliance on cues that support the learned prior. Finally, we find that in-context learning is limited when the underlying visual capability is weak or absent. Together, these results show that current VLMs possess visual grounding capabilities that standard evaluations may overlook, underscoring the need for controlled evaluation design in diagnosing and improving model capability.
Read the full abstract

All authors Kun Wang, Xindi Wu, Sanghyuk Chun, Olga Russakovsky, Esin Tureci

Computer Vision

UniDeblur: Unified Motion Deblurring via Fourier Convolution Learning

Xintian Mao, Jiansheng Wang, Huaiyuan Wang et al. (+2 more)

Image deblurring is a fundamental task in computer vision, yet existing methods mainly specialize in isolated scenarios. Consequently, the deblurring performance of these models drops significantly across scenarios with different cameras and image contents. This motivates the development of a unified motion deblurring framework that can handle diverse blur patterns. To this end, we construct a multi-source dataset and develop a data pruning and quality control pipeline. Then, to restore blur images with various motion blur kernels across different scenarios, we propose a unified motion deblurring framework via Fourier Convolution Learning. Existing deblurring networks usually rely on pixel-level mappings and overlook the convolutional nature of blur. Thus, the network mostly fits the image content itself rather than learning the convolution process, making it difficult to handle multi-source blur images. Instead of relying on spatial mapping, we perform blur-adaptive global filter learning in Fourier space, which avoids the limited spatial support of kernel estimation. Extensive experiments demonstrate that our unified model effectively handles motion blurs across diverse sources and scenes, achieving competitive performance across both in-domain and OOD benchmarks. Code has been released at github.com/DeepMed-Lab-ECNU/Single-Image-Deblur.
Read the full abstract

All authors Xintian Mao, Jiansheng Wang, Huaiyuan Wang, Qingli Li, Yan Wang

Computer Vision

What Egocentric Video Embeddings Encode and How to Combine Them with Inertial Sensors for Human Activity Recognition

Kansuke Habano

Egocentric video and body-worn inertial sensors are increasingly recorded together, and the video is usually encoded by pretrained models. What these embeddings represent when the camera sits on the head, and how they should be combined with inertial sensing, has received little attention. Across VideoMAEv2, CLIP, DINOv2 and the egocentric VC-1, the leading principal directions of the embedding are dominated by where the camera looks. Measured head orientation explains them on Ego-Exo4D for egocentric (R^2 0.49-0.70) but not exocentric cameras, and the scene layout explains them on WEAR. The part that the limbs' inertial signals predict separates activities across people. In a set-level probe, the rest separates exercise sets within a participant but not activities across people. This is consistent with separate roles: the video finds the clips of the same set, and the inertial classifier labels them. On WEAR, with temporal order removed and one random limb per one-second clip, we propagate inertial predictions over a within-participant video graph whose leading components are removed. This raises subject-wise cross-validated macro-F1 from 0.648 to 0.852. In our comparisons, propagation helps only over the video graph, and fusing the modalities in one classifier, or adding inertial edges to the graph, performs worse.
Read the full abstract

Computer Vision

BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Models

Payam Abdisarabshali, Fardis Nadimi, Kasra Borazjani et al. (+3 more)

Federated vision-language models (FedVLMs) represent one of the emerging frontiers of machine learning, aiming to integrate federated learning within the fine-tuning pipelines of vision-language models (VLMs). In this work, we introduce BLEND, a new framework designed to jointly enhance the personalization and generalization capabilities of FedVLMs. BLEND proposes a selective VLM parameter fine-tuning and aggregation strategy that consists of (i) global vision and text adapters shared across clients, and (ii) a personalized vision projection adapter tailored to each client. BLEND further introduces a novel loss function that blends the outputs of the global and personalized adapters to achieve personalization, while simultaneously promoting generalization through a principled anchor design implemented via the projection heads of the zero-shot VLM. Extensive experiments demonstrate that BLEND consistently outperforms state-of-the-art baselines, highlighting its ability to effectively balance personalization and generalization across diverse system settings. (Code is available at: https://github.com/payamsiabd/BLEND.git)
Read the full abstract

All authors Payam Abdisarabshali, Fardis Nadimi, Kasra Borazjani, Naji Khosravan, Richeng Jin, Seyyedali Hosseinalipour

Computer Vision

JMT-YOLO: Multi-task Domain Adaptation via Segmentation-Guided Consistency Learning

Zhenhua Yu, Jiahao Zhang, Wenchao Zhang et al. (+2 more)

Domain adaptive object detection (DAOD) aims to reduce performance degradation due to distribution shifts between training (source) and testing (target) datasets. However, owing to insufficient modeling of fine-grained features and limited instance-level alignment, existing approaches become less effective under complex environmental conditions, such as extreme weather conditions and object occlusion. We overcome these challenges by proposing JMT-YOLO, a novel joint multi-task learning framework for DAOD. JMT-YOLO is based on the Mean Teacher architecture. It introduces an auxiliary semantic segmentation task and a masking mechanism to enhance the modeling of foreground-background semantic boundaries in the target domain. Specifically, fine-grained feature degradation is reduced by introducing a multiscale context-aware segmentation head equipped with a dense connection structure to effectively capture local and global contextual features. A mask-consistency loss is formulated by applying random masking operations to target images, thereby guiding the network to learn structural context and improving instance-level feature alignment. Extensive experiments on three benchmark domain adaptation tasks (Pascal VOC to Clipart1k, Cityscapes to Foggy Cityscapes, and KITTI to Cityscapes) demonstrate that JMT-YOLO achieves detection performances of 46.9%, 57.3%, and 61.0%, respectively. These results demonstrate superior performance over existing state-of-the-art and baseline methods, thereby validating the effectiveness and generalization of JMT-YOLO.
Read the full abstract

All authors Zhenhua Yu, Jiahao Zhang, Wenchao Zhang, Ou Ye, Hengxiang Zhao

Computer Vision

A Lightweight Velocity Correction Framework for Improving Instruction Following in Few-Step Text-to-Image Generation

Yating Gao, Zhenlong Zhu, Wen Zihao et al. (+6 more)

Few-step text-to-image generators produce high-quality images with few network forward passes, yet struggle with complex prompts involving multiple subjects, interactions, spatial relations, counting, and attribute binding. Can these errors be reduced without changing the generator's parameters or increasing its forward-pass count? We propose a lightweight correction framework for a frozen 6B-parameter generator. The generator predicts velocities that determine how its intermediate image representation is updated; a corrector with fewer than 20M parameters learns residual adjustments at selected sampling steps. Because earlier corrections change the states encountered at later steps, our On-Trajectory Flow-Bridge Supervision derives residual targets at the states actually reached during corrected generation. A single corrector is shared across selected steps and runs in parallel with the generator. Only the corrector is trained, requiring neither reward optimization nor re-distillation. To our knowledge, this is the first framework for lightweight velocity-residual adaptation of fully frozen few-step text-to-image generators that preserves their original sampling schedule and forward-pass count. On our target complex-scene distribution, the framework achieves a +17.60-percentage-point NetWin over frozen Z-Image-Turbo in pairwise instruction-following evaluation, with only 26.9 ms (0.84%) additional end-to-end latency per image and no increase in backbone evaluations.
Read the full abstract

All authors Yating Gao, Zhenlong Zhu, Wen Zihao, Lenan Zhang, Jiexiong Wang, Yang Yang, Guanjun Jiang, Yixiong Zou, Takuya Maekawa

Computer Vision

AD-SRF: Attack-Defense Semantic Reinforcement for Multi-Modal Image Fusion

Hao Zhang, Zhiqian Zheng, Linfeng Tang et al. (+2 more)

Existing image fusion methods struggle to balance visual quality and semantic attributes due to the inherent conflict between their optimization objectives. To address this challenge, we propose an attack-defense semantic reinforcement framework for multi-modal image fusion, termed AD-SRF. By decoupling the optimization of these two aspects and then losslessly reintegrating them, this framework injects rich semantic attributes into the fused image without compromising its visual quality. First, we develop a transformer-based restorative visual fusion network as a strong visual baseline. Equipped with the cross-modal retrieval module and the information restoration-fusion collaborative loss, it produces highly visible fused images even under degraded conditions. Second, we introduce an attack-defense game mechanism to strengthen the semantic attributes of the fused images. The attacker learns perturbation patterns that impair downstream segmentation, whereas the defender restores missing semantics from the original inputs and reintegrates them into the fused image through an imperceptible semantic map. Through iterative adversarial interaction, this mechanism drives the defender to continually strengthen the fused image's semantic attributes, boosting downstream segmentation performance without compromising visual quality. Extensive experiments demonstrate that AD-SRF achieves a win-win performance in both visual quality and semantic attributes. The code is publicly available at https://github.com/HaoZhang1018/AD-SRF.
Read the full abstract

All authors Hao Zhang, Zhiqian Zheng, Linfeng Tang, Xinyu Xiang, Jiayi Ma

Computer Vision

MeshBind: Point-to-Mesh Binding for Non-Rigid Scene Flow Automatic Annotation

Yinqi Chen, E Li, Shuaijun Wang et al. (+3 more)

Scene flow estimation is essential for many autonomous driving (AD) and robotic applications, but obtaining ground-truth scene flow for non-rigid objects such as pedestrians is highly difficult. We present MeshBind, an automatic annotation method that uses mesh models to generate per-point scene flow for non-rigid objects. The novelty of this work is threefold: 1) developing a multi-view module that reconstructs an articulated mesh model from six surround cameras and registers it to the LiDAR frame; 2) developing a local-frame binding technique that attaches each point to the mesh via a per-triangle tangent-normal frame, invariant under rigid motion yet adaptive to articulated deformation; 3) developing a three-layer flow registration pipeline that combines ego-vehicle motion compensation, box binding, and mesh binding to produce full-scene per-point flow. We implement MeshBind in SF-Studio, an interactive annotation tool that visualises flow and allows human mesh corrections with automatic recomputation. We also release SFB (SUScape-Flow Benchmark), a large-scale benchmark built on the public SUScape dataset, covering 41k frames with per-point non-rigid pedestrian annotations and a Rigid/Semi-Rigid/Non-Rigid evaluation protocol. Experiments show that fine-tuning on SFB reduces three-way endpoint error (EPE) by 2-3x, and brings non-rigid EPE to the same level as rigid EPE.
Read the full abstract

All authors Yinqi Chen, E Li, Shuaijun Wang, Weiyuan Peng, Meiying Zhang, Qi Hao

Computer Vision

Win-Win Fusion: Multi-Modal Image Fusion with Visual-Semantic Collaboration

Hao Zhang, Xuhui Zuo and Jiayi Ma

Existing multi-modal image fusion (MMIF) methods generally face an optimization gap between the visual and semantic domains, limiting the enhancement of perception capabilities. To overcome this long-standing barrier, this work proposes W2F, a new visual-semantic mutual reinforcing paradigm for MMIF. First, we reinforce the foundation for visual-semantic collaboration by amplifying feature complementarity through two targeted designs. Specifically, the segmentation-guided fusion loss introduces object-aware structural guidance into visual fusion, while the embedded information restoration prior strengthens the ability of visual fusion to provide reliable perceptual evidence for the semantic branch under degraded conditions. Then, we introduce a strong multi-modal information interaction mechanism composed of interactive gated mixed attention (IGM-Att) and progressive coupling attention (PC-Att). These modules ensure comprehensive scene representation from both local and global perspectives, providing informational support for visual-semantic collaboration. Finally, we construct a collaborative mechanism to couple visual fusion and semantic fusion. It employs a feature search module to extract complementary key features from both tasks, enriching the representational potential of visual and semantic information. Meanwhile, a gradient correction strategy ensures collaboration from the optimization perspective, enhancing performance for both visual and semantic fusion through consistency learning. Extensive experiments demonstrate that W2F surpasses state-of-the-art methods in both visual appeal and decision accuracy. The code is publicly available at https://github.com/HaoZhang1018/W2F.
Read the full abstract

Computer Vision

ROSETTA: Rotation Subspace Estimation for Training-free Transmodal Alignment

Alex Junho Lee

When a query must be matched against a database built with a different sensor or processed by a different encoder, the independently trained encoders impose distinct coordinate systems on the feature space. No canonical reference frame exists to which all encoders can be aligned, and no single training procedure has fully closed this gap. We show that the relative transformation between any two such systems is a spatially preserving rotation in whitened feature space, whose angular velocity exhibits white-noise characteristics: large in norm, temporally uncorrelated, and unpredictable from the features themselves. However, the cross-covariance between encoder outputs has low effective rank, constraining only a small fraction of the rotation parameters, therefore local alignment is tractable. ROSETTA is a training-free calibration layer that estimates the local rotation via an expectation-maximization procedure, requiring no coobservation at query time. We validate ROSETTA on three public driving datasets spanning LiDAR, camera, thermal, and event modalities in cross-modal, cross-device, cross-representation, and cross-backbone settings, with milliseconds overhead per query.
Read the full abstract

Computer Vision

Selective Spatio-Temporal Fruit Inspector (S$^2$FI): A Mamba-based Framework for Rotational Fruit Quality Assessment

Hao Tian, Jie Hu, Guoqi Shan et al. (+3 more)

The global fruit and vegetable industry confronts severe post-harvest sorting challenges for small and medium-scale growers. Large intelligent sorting systems are too costly, while manual sorting suffers from low efficiency, inconsistent standards, and high labor demand. To bridge this gap between expensive equipment and inefficient manual work, this study develops a low-cost, high-efficiency, lightweight fruit sorting system driven by deep learning, suitable for small and medium growers. We propose the Selective Spatio-Temporal Fruit Inspector (S$^2$FI) architecture, which combines a 3D convolutional encoder with a Mamba-based backbone.It efficiently extracts spatiotemporal features, adaptively weighs informative frames during fruit rotation, and models long-range dependencies with linear complexity, reducing motion blur and occlusion. The hardware uses a dual-conveyor and curved rail design, supporting household power, fast deployment, and lightweight embedded deployment to improve affordability. We also built a conveyor-based fruit quality video dataset with 6,460 samples. Experiments on the held-out test set show that our method achieves the best trade-off among accuracy, efficiency, and practicality, outperforming the compared baselines on the test set, achieving an excellent balance of accuracy, speed, and practicality for real-world sorting.
Read the full abstract

All authors Hao Tian, Jie Hu, Guoqi Shan, Fan Wang, Wenzhuo Zhang, Youshan Zhang

Computer Vision

CaptchaAgent: Verifiable Visual Reasoning for CAPTCHA Solving

Shangquan Sun, Ruoyu Chen, Zhirui Liu et al. (+4 more)

Modern reasoning CAPTCHAs require fine-grained perception, compositional relation reasoning, and grounded browser interaction. End-to-end vision-language model agents remain brittle in this setting because a missed object, incorrect count, or confused relation can propagate directly to the final action. To address this problem, we present CaptchaAgent, a structured visual reasoning framework for interactive CAPTCHA solving. Instead of predicting an action directly from a screenshot, CaptchaAgent first constructs a visual intermediate representation (V-IR) that explicitly records entities, attributes, and spatial or topological relations. A symbol-only ASCII protocol further requires the parser to commit to a concrete object set and count. Building on this global representation, localized evidence refinement revisits ambiguous regions at higher effective resolution, while counterfactual consistency verification tests whether the resulting action remains stable under task-preserving transformations. Across three proprietary backbones on Open CaptchaWorld, CaptchaAgent improves Pass@1 by 21.3-25.8 percentage points over a commercially available browser agent baseline evaluated with the same backbones. On the separate offline Oedipus-data benchmark, it further improves average Pass@1 by 9.2-12.3 percentage points over the Oedipus solver across three matched backbones. The intermediate traces show that exact counting, continuous geometry, and ordered execution remain major sources of automation difficulty for the evaluated agents, providing practical guidance for designing more resilient compositional CAPTCHAs.
Read the full abstract

All authors Shangquan Sun, Ruoyu Chen, Zhirui Liu, Xiang Liu, Chenghao Qian, Wenqi Ren, Xiaochun Cao

Computer Vision

APEX: Asynchronous Prefix Denoising for Extreme Compression

David Dinkevich, Nisan Chiprut, Yoav HaCohen et al. (+1 more)

Fewer latent tokens promise cheaper diffusion. Yet aggressive latent compression creates a reconstruction-generation dilemma: adding channels improves reconstruction but makes generation harder. We introduce APEX, short for Asynchronous Prefix denoising for EXtreme compression, which pairs a hierarchical latent space with asynchronous denoising over channel groups. Our autoencoder orders channel prefixes from structure to detail, while one shared denoiser generates them at staggered local times without adding tokens. On ImageNet-512, APEX outperforms existing methods at every evaluated channel count under 64x and 128x compression. At matched training and guided sampling compute, it also outperforms a 256-token model at every evaluated budget, with larger gains at lower compute. Code is available at https://daviddinkevich.github.io/APEX.
Read the full abstract

All authors David Dinkevich, Nisan Chiprut, Yoav HaCohen, Dani Lischinski

Computer Vision

Structure-Aware Matching for Robust UAV Thermal-to-Satellite Geo-Localization

Chun Liu, Yuxin Wang, Zhenxiang Huo et al. (+3 more)

UAV thermal-to-satellite geo-localization localizes UAVs by matching real-time thermal images against georeferenced satellite images, enabling autonomous navigation in GNSS-denied and low-visibility conditions. However, substantial cross-modal disparities in color, brightness and viewpoint hinder conventional appearance-based methods from establishing reliable correspondences. While existing approaches predominantly address global modality shifts, they overlook the invariant scene structures inherently shared between thermal and satellite views. To bridge this gap, we propose SAMF, a Structure-Aware Matching Framework that explicitly exploits stable cross-modal structural cues to improve UAV thermal geolocalization. Specifically, the Directional Structure Enhancement Module (DSEM) strengthens robust structural representations by capturing directional and low-frequency spatial patterns, and the Modality-Aware Affine Remapping Module (MARM) subsequently applies affine correction to these enhanced features, further promoting cross-modal alignment. Additionally, we introduce the Thermal-UAV-Paired benchmark derived from the Thermal-UAV dataset. Extensive experiments on both the Boson-nighttime and Thermal-UAV-Paired datasets show that SAMF achieves state-of-the-art performance. Ablation studies further validate the effectiveness and synergistic complementarity of DSEM and MARM. The code and dataset are released at: https://github.com/YuxinWang879/SAMF.git.
Read the full abstract

All authors Chun Liu, Yuxin Wang, Zhenxiang Huo, Linlin Zhu, Tao Xu, Henggang Han

Thursday 8 October

Computer Vision · Cross-listed from eess.IV

Temporal Diffusion Bridge Model for Vessel Extraction Using Physics-informed Synthetic Data

Jinkui Hao, Donald R. Cantrell, Ramez Abdalla et al. (+2 more)

Accurate artery extraction from X-ray coronary angiography (XCA) is essential for coronary artery disease diagnosis and intervention. However, low vessel visibility, respiratory and cardiac motion, and limited annotations challenge existing methods. Moreover, most deep learning approaches produce binary vessel masks without preserving clinically relevant intra-arterial contrast information. We propose a Temporal Diffusion Bridge Model (TDBM) that formulates vessel extraction as image translation and learns from physics-informed synthetic data without manual vessel annotations. A physics-based XCA synthesis engine generates diverse, perfectly aligned pairs of contrast-enhanced and contrast-free images. Using these pairs, TDBM learns cross-domain mappings to recover high-fidelity vessel information. Temporal conditioning incorporates adjacent frames to improve consistency, while a vessel-aware contrastive loss exploits background-invariant synthetic pairs to enhance vascular discrimination. We evaluate TDBM on two XCA vessel segmentation datasets and conduct a reader study using two real-patient XCA sequence datasets. Zero-shot segmentation is further assessed on retinal imaging and fluorescence microscopy datasets. Results demonstrate that TDBM consistently outperforms existing methods and generalizes effectively across imaging modalities. Code and trained weights will be available at: https://github.com/Advanced-AI-in-Medicine-and-Physics-Lab/TDBM
Read the full abstract

All authors Jinkui Hao, Donald R. Cantrell, Ramez Abdalla, Sameer A. Ansari, Bo Zhou

Computer Vision · Cross-listed from eess.IV

RePrior: Measurement-Reconciled Prior Correction for Sparse-View CBCT Reconstruction

Mengrui Yin, Tianxingjian Ding, Wufan Zhao et al. (+2 more)

Sparse-view cone-beam computed tomography (CBCT) is severely under-constrained by limited X-ray projections. Learned reconstruction can provide complementary anatomical information, but using it as a prior in patient-specific Gaussian reconstruction introduces two challenges. First, the discrepancy between the learned prior and current reconstruction is spatially non-uniform: some local differences contain useful missing 3D information, whereas others reflect prior bias. Second, even a volumetrically beneficial correction can conflict with the observed X-ray measurements. Since X-ray measurements are line integrals, spatially separated corrections may affect overlapping rays, with their signed projection responses reinforcing or partially canceling in the measurement residual, so their feasibility cannot always be assessed independently. In this paper, we introduce RePrior, a multi-stage progressive prior-correction framework that addresses correction actionability, measurement feasibility, and representation expressivity. Selective Correction Proposal (SCP) first identifies actionable signed local corrections from prior–reconstruction discrepancies. Measurement-Constrained Reconciliation (MCR) then jointly enforces their consistency with patient-specific X-ray measurements, while Hybrid Residual Enhancement (HRE) recovers additional correction capacity through complementary voxel- and Gaussian-domain degrees of freedom around the reconciled anchor. RePrior requires neither test-time ground truth nor case-specific network fine-tuning. RePrior gains approximately 2.6~dB in 3D PSNR over R$^2$-Gaussian across LUNA16, PANORAMA, and PENGWIN. Beyond prior-assisted reconstruction, it delivers additional gains in volumetric quality and observed-view projection fidelity, with consistent gains in held-out-view fidelity in the backbone-matched evaluation.
Read the full abstract

All authors Mengrui Yin, Tianxingjian Ding, Wufan Zhao, Ziyang Yan, Yu Tian

Computer Vision

Scenes as an Attack Surface: Uncovering Scene-Grounded Safety Failures in MLLMs

Tongxi Wu, Yuyang Cheng, Jiacheng Xian et al. (+5 more)

Existing jailbreak attacks have exposed serious safety weaknesses in multimodal large language models (MLLMs). Most prior attacks rely on specially constructed visual inputs to carry or support harmful intent. In this paper, we uncover a different vulnerability: task-relevant scene semantics alone can shift an MLLM from refusal to compliance, even when the visual input introduces no additional attack instructions and the original task intent remains unchanged. We formalize this failure as a scene-grounded visual safety vulnerability and introduce SceneSafetyBench to investigate it across eight risk categories and nine white- and black-box MLLMs. Our analysis reveals that this failure is associated with how task-relevant visual evidence influences downstream safety representations, suggesting that visual grounding alone cannot fully explain the observed safety shift. Building on this finding, we propose Scene-Grounded Attacks (SGA), which preserve the original task while constructing validated scene-grounded queries and combining their independent responses. Across nine target MLLMs, SGA achieves an average attack success rate (ASR) of 86.17%, outperforming the strongest baseline by 8.55 percentage points and achieving the highest ASR on eight models. Our results identify scene semantics as an overlooked attack surface in MLLMs and highlight a growing tension between stronger visual grounding and robust multimodal safety alignment.
Read the full abstract

All authors Tongxi Wu, Yuyang Cheng, Jiacheng Xian, Hu Huang, Zhihao Wu, Menglin Yang, Jian Zhang, Yang Gao

Computer Vision

Human Pose Estimation by Probabilistic Mixtures of Gaussian and Uniform Distributions

Santiago Reina Alguacil and Ezequiel López Rubio

Despite advancements in sub-pixel decoding, modern Human Pose Estimation (HPE) architectures remain constrained by deterministic coordinate extraction from discrete heatmaps, which systematically fails under dense occlusions and Out-of-Distribution (OoD) ambiguities. To address these limitations, we propose a post-hoc continuous probabilistic framework that intercepts raw heatmaps and models them as a Gaussian Mixture Model (GMM) augmented with a spatially agnostic uniform distribution, requiring no architectural modifications or retraining. This continuous representation allows us to comprehensively explore the distinct contributions of density modeling to HPE. We show that the spatial expectation of our fitted GMM achieves precision parity with leading sub-pixel decoders without heuristic post-processing. Leveraging this continuous space, our test-time augmentation aggregates probability distributions rather than discrete activations, mitigating multi-scale heatmap poisoning. Furthermore, because our multimodal GMM parameterization isolates competing spatial hypotheses, it naturally interfaces with geometric regularizers such as Markov Random Fields. However, our analysis exposes the inherent vulnerabilities of 2D kinematic priors under perspective foreshortening and visual degradation. Crucially, this density formulation yields significant improvements in uncertainty quantification: our calibrated geometric covariance volume outperforms standard peak-intensity heuristics in OoD anomaly detection under severe occlusion. For in-distribution calibration, dynamically fusing our geometric metric via Adaptive Softmax reduces or matches baseline Area Under the Sparsification Error (AUSE) across benchmarks and degradation levels. Finally, we establish that this continuous parameterization is extensible to multi-view 3D architectures, bypassing the $\mathcal{O}(N^3)$ voxel memory bottleneck.
Read the full abstract

Computer Vision

Pristine Models

Anirudh Sundara Rajan, Krishna Kumar Singh, Richard Zhang et al. (+1 more)

Training on broad, unfiltered data enables high-quality image generation, but can also teach models concepts that should not be generated; restricting the training data ensures safety, but reduces quality because examples containing unsafe concepts also provide useful knowledge that is not itself unsafe. Existing approaches largely address this trade-off post hoc through concept unlearning, which can lead to catastrophic forgetting as the number of concepts to remove grows. We instead ask whether unsafe concept knowledge can be compartmentalized during pretraining so that access to it can be disabled at inference. We introduce Pristine Models, a novel text-to-image generation architecture with an attention mechanism that encourages the separation of safe and unsafe knowledge into independently parameterized branches. The branch-specific parameters are learned during training, while the unsafe branch can be detached at inference. On ImageNet, Pristine Models enable safer generation while retaining much of the utility of full-data pretraining, achieving higher image quality and better text alignment than training only on safe data or applying post-hoc concept removal. We further scale our approach to CC12M, demonstrating that it remains effective in a substantially larger and more diverse open-world pretraining setting.
Read the full abstract

All authors Anirudh Sundara Rajan, Krishna Kumar Singh, Richard Zhang, Yong Jae Lee

Computer Vision · Cross-listed from cs.RO

SenseWeather: Multimodal Sensing for Robust Robotic Perception under Extreme Conditions

Fucai Ke, Sukai Huang, Junyi Hao et al. (+4 more)

Robotic perception must remain reliable when extreme environmental conditions severely degrade the primary RGB observation. Although robotic platforms commonly carry complementary sensors such as depth and infrared, much of the modern perception stack remains RGB-centric, creating a mismatch between multimodal onboard sensing and pretrained downstream models. We study whether complementary sensing can be used upstream to recover a task-compatible RGB observation that preserves existing perception capabilities without modifying or retraining the downstream stack. To this end, we introduce SenseWeather, a controlled multimodal benchmark for robotic perception under severe rain, snow, dust, fog, and darkness, with registered RGB, depth, and infrared observations, clean references, and grounded high-level perception tasks. We further develop Fusion-to-RGB, with two parameter-efficient sensor-fusion realizations based on token-space fusion for a diffusion transformer and residual latent conditioning for an inpainting UNet. Across frozen open-vocabulary grounding and vision-language models, RGB+D+IR fusion improves all paired downstream comparisons over the corresponding RGB-only restorers, including AP50 gains across all ten detector comparisons. Importantly, these perception gains do not consistently follow conventional image-fidelity metrics, and increasing weather severity does not induce monotonic downstream degradation. These results show that complementary sensing can preserve reusable robotic perception under extreme conditions, while highlighting task-level perception robustness as an objective distinct from visual restoration quality.
Read the full abstract

All authors Fucai Ke, Sukai Huang, Junyi Hao, Kal Backman, Naim Rastgoo, Gholamreza Haffari, Hamid Rezatofighi

Computer Vision

Spend Where It Matters: Particle Search with MH Refinement for Budget-Constrained Audio-Visual Alignment

Shivanshu Shekhar, Raghavendra Addanki, Tong Zhang et al. (+1 more)

Reward-guided inference-time alignment can improve joint audio-visual generation, but optimizing multiple alignment objectives requires many expensive model evaluations, making existing methods costly under limited compute. Particle-based search enables global exploration but incurs estimation error with a finite population, while Metropolis-Hastings (MH) can refine candidates locally but benefits from a strong initialization. We introduce Spend Where It Matters (\modelname), a two-stage, training-free, reward-agnostic framework that combines particle search with MH refinement under a fixed compute budget. In the first stage, we adaptively allocate particles across denoising checkpoints using online variance estimates, minimizing estimation variance while concentrating compute where it is most useful. This search provides a promising candidate for MH refinement, which uses local noise-and-denoise proposals on completed outputs to further improve reward. By starting from a strong candidate, MH can recover additional reward without requiring a larger particle population or costly global exploration. We show that, at matched compute, combining particle search with MH refinement achieves higher rewards than either approach alone. On JavisBench-Mini, at closely matched compute, \modelname achieves mean relative gains across seven metrics of $30.7\%$ over BoN and $14.1\%$ over EvoSearch with JavisDiT++, and $9.2\%$ over BoN and $17.0\%$ over EvoSearch with LTX-2. Beyond joint audio-visual generation, we also demonstrate the benefits of adaptive particle allocation on SD1.5 image generation, improving AestheticScore by $2.2\%$ and ImageReward by $6.0\%$ over uniform allocation.
Read the full abstract

All authors Shivanshu Shekhar, Raghavendra Addanki, Tong Zhang, Uttaran Bhattacharya

Computer Vision

Domain Adaptation of SAM 2.1 for Building-Component Segmentation in Image-Based 3D Reconstruction

Ehsan Agha Ebrahimi, Yong-Cheol Lee and Keith G. Mills

3D reconstruction of building components using image-based scanning requires appropriate segmentation of the objects in the 2D building images. In order to benefit from consistent grouping of the 2D masks in 3D environments, vision foundation models such as the Segment Anything Model have been utilized by Group Anything with Radiance Fields (GARField) to perform 3D segmentation. This study fine-tuned SAM 2.1 on the HBD dataset, consisting of 2,679 annotated images from 42 buildings, to enhance multi-class segmentation of 2D building images, covering 11 classes (Wall, Window, Ceiling, Door, Column, Beam, Floor, Opening, Curtain Wall, Roof, and Lift). The model was evaluated using per-class IoU, Dice, and mIoU metrics, and the results show an improvement in mIoU from 49.9% to 69.6%, indicating a meaningful improvement in class segmentation and providing a more accurate model for distilling the masks in 3D. SegFormer was then used as a baseline, generating per-class masks for comparison with SAM 2.1, which indicates improvement in some object classes with larger surface area. The masks in the building images were grouped in the neural radiance field, and the pretrained SAM 2.1 was compared with the fine-tuned SAM 2.1 in the newly rendered images. This comparison shows that fine-tuning improves the grouping of building components at a user-selected grouping scale. This advances the application of Group Anything with Radiance Fields (GARField) in 3D, enabling better segmentation of building structures for 3D auto-reconstruction.
Read the full abstract

Computer Vision

Shift-Aware Identity-Guided Latent Refinement for Referring Audio–Visual Segmentation

Kun Li, Sami Sebastian Brandt and Michael Ying Yang

Referring Audio–Visual Segmentation (Ref-AVS) is an emergent multimodal task requiring the precise identification and segmentation of a specific object based on a natural language expression grounded in both visual and auditory cues. Unlike traditional AVS, which segments generic sound sources, or referring video object segmentation (Ref-VOS), which relies solely on visual–textual grounding, Ref-AVS necessitates fine-grained tri-modal interaction across vision, audio, and language. However, existing methods often fail to maintain consistency when subjected to target shift, a phenomenon where the model erroneously transfers its focus between objects due to transient occlusions, acoustic fading, or identity ambiguities. To address this, we propose a novel Shift-Aware Identity-guided Latent (SAIL) refinement framework. It introduces a dual-token strategy that decouples frame-level dynamics from a global semantic anchor to explicitly model target-shift cues. We design a shift-aware refinement module that rectifies drift by aligning tokens with domain-specific evidence. Finally, we leverage a decoder that integrates the refined embeddings with a continuous memory update to maintain spatio-temporal coherence and identity consistency throughout the video sequence. Extensive experiments conducted on two Ref-AVS benchmarks demonstrate the effectiveness of our proposed method, significantly outperforming existing AVS, Ref-VOS, and Ref-AVS approaches. The code will be released after publication.
Read the full abstract