Computer Vision
US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound
Ashwath Radhachandran, Vedrana Ivezić, Shreeram Athreya et al. (+2 more)
Ultrasound (US) imaging poses unique challenges for representation learning due to its inherently noisy acquisition process. The low signal-to-noise ratio and
stochastic speckle patterns hinder standard self-supervised learning methods relying on a pixel-level reconstruction objective. Joint-
Embedding Predictive Architectures (JEPAs) address this drawback by predicting masked latent representations rather than raw pixels. However, standard approaches depend on hyperparameter-brittle and computationally expensive online teachers updated via exponential moving average. We propose US-JEPA, a self-supervised framework that adopts the Static-teacher Asymmetric Latent Training (SALT) objective. By using a frozen, domain-specific teacher to provide stable latent targets, US-JEPA decouples student-teacher optimization and pushes the student to expand upon the semantic priors of the teacher. In addition, we provide the first rigorous comparison of all publicly available state-of-the-art ultrasound foundation models on UltraBench, a public
dataset benchmark spanning multiple organs and pathological conditions. Under linear probing for diverse classification tasks, US-JEPA achieves performance competitive with or superior to domain-specific and universal
vision foundation model baselines. Our results demonstrate that masked latent prediction provides a stable and efficient path toward robust ultrasound representations.
Read the full abstract ▾
All authors Ashwath Radhachandran, Vedrana Ivezić, Shreeram Athreya, Corey W. Arnold, William Speier
Computer Vision
Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition
Alexandros Haliassos, Rodrigo Mira and Stavros Petridis
Unified
Speech Recognition (USR) has emerged as a semi-supervised framework for training a single model for audio, visual, and audiovisual
speech recognition, achieving state-of-the-art results on in-distribution
benchmarks. However, its reliance on autoregressive pseudo-labelling makes training expensive, while its decoupled supervision of CTC and
attention branches increases susceptibility to self-reinforcing errors, particularly under distribution shifts involving longer sequences, noise, or unseen domains. We propose CTC-driven teacher forcing, where greedily decoded CTC pseudo-labels are fed into the decoder to generate
attention targets in a single forward pass. Although these can be globally incoherent, in the pseudo-labelling setting they enable efficient and effective knowledge transfer. Because CTC and CTC-driven
attention pseudo-labels have the same length, the decoder can predict both simultaneously, benefiting from the robustness of CTC and the expressiveness of
attention without costly beam search. We further propose mixed sampling to mitigate the exposure bias of the decoder relying solely on CTC inputs. The resulting method, USR 2.0, halves training time, improves robustness to out-of-distribution inputs, and achieves state-of-the-art results on LRS3, LRS2, and WildVSR, surpassing USR and modality-specific self-supervised baselines.
Read the full abstract ▾
Computer Vision
IPv2: An Improved Image Purification Strategy for Real-World Ultra-Low-Dose Lung CT Denoising
Guoliang Gong and Man Yu
The image purification strategy constructs an intermediate distribution with aligned anatomical structures, which effectively corrects the spatial misalignment between real-world ultra-low-dose CT and normal-dose CT images and significantly enhances the structural preservation ability of denoising models. However, this strategy exhibits two inherent limitations. First, it suppresses noise only in the chest wall and bone regions while leaving the image background untreated. Second, it lacks a dedicated mechanism for denoising the lung parenchyma. To address these issues, we systematically redesign the original image purification strategy and propose an improved version termed IPv2. The proposed strategy introduces three core modules, namely Remove Background, Add noise, and Remove noise. These modules endow the model with denoising capability in both background and lung tissue regions during training data construction and provide a more reasonable evaluation protocol through refined label construction at the testing stage. Extensive experiments on our previously established real-world patient lung CT
dataset acquired at 2% radiation dose demonstrate that IPv2 consistently improves background suppression and lung parenchyma restoration across multiple mainstream denoising models. The code is publicly available at https://github.com/MonkeyDadLufy/Image-Purification-Strategy-v2.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
WildOS: Open-Vocabulary Object Search in the Wild
Hardik Shah, Manthan Patel, Erica Tevere et al. (+6 more)
Autonomous
navigation in complex, unstructured outdoor environments requires
robots to operate over long ranges without prior maps and limited depth sensing. In such settings, relying solely on
geometric frontiers for exploration is often insufficient. In such settings, the ability to reason semantically about where to go and what is safe to traverse is crucial for robust, efficient exploration. This work presents WildOS, a unified system for long-range, open-vocabulary object search that combines safe
geometric exploration with semantic visual
reasoning. WildOS builds a sparse
navigation graph to maintain spatial
memory, while utilizing a foundation-model-based
vision module, ExploRFM, to score frontier nodes of the graph. ExploRFM simultaneously predicts traversability, visual frontiers, and object similarity in image space, enabling real-time, onboard semantic
navigation tasks. The resulting
vision-scored graph enables the
robot to explore semantically meaningful directions while ensuring
geometric safety. Furthermore, we introduce a particle-filter-based method for coarse localization of the open-vocabulary target query, that estimates candidate goal positions beyond the
robot's immediate depth horizon, enabling effective planning toward distant goals. Extensive closed-loop field experiments across diverse off-road and urban terrains demonstrate that WildOS enables robust
navigation, significantly outperforming purely
geometric and purely
vision-based baselines in both efficiency and autonomy. Our results highlight the potential of
vision foundation models to drive open-world robotic behaviors that are both semantically informed and geometrically grounded. Project Page: https://leggedrobotics.github.io/wildos/
Read the full abstract ▾
All authors Hardik Shah, Manthan Patel, Erica Tevere, Deegan Atha, Marcel Kaufmann, Shehryar Khattak, Marco Hutter, Jonas Frey, Patrick Spieler
Computer Vision
MRI Contrast Enhancement Kinetics World Model
Jindi Kong, Yuting He, Cong Xia et al. (+2 more)
Clinical MRI contrast acquisition suffers from inefficient information yield, which presents as a mismatch between the risky and costly acquisition protocol and the fixed and sparse acquisition sequence. Applying world models to simulate the contrast enhancement kinetics in the human body enables continuous contrast-free dynamics. However, the low temporal resolution in MRI acquisition restricts the training of world models, leading to a sparsely sampled
dataset. Directly training a generative model to capture the kinetics leads to two limitations: (a) Due to the absence of data on missing time, the model tends to overfit to irrelevant features, leading to content distortion. (b) Due to the lack of continuous temporal supervision, the model fails to learn the continuous kinetics law over time, causing temporal discontinuities. For the first time, we propose MRI Contrast Enhancement Kinetics World model (MRI CEKWorld) with SpatioTemporal Consistency Learning (STCL). For (a), guided by the spatial law that patient-level structures remain consistent during enhancement, we propose Latent
Alignment Learning (LAL) that constructs a patient-specific template to constrain contents to align with this template. For (b), guided by the temporal law that the kinetics follow a consistent smooth trend, we propose Latent Difference Learning (LDL) which extends the unobserved intervals by interpolation and constrains smooth variations in the latent space among interpolated sequences. Extensive experiments on two
datasets show our MRI CEKWorld achieves better realistic contents and kinetics. Codes will be available at https://github.com/DD0922/MRI-Contrast-Enhancement-Kinetics-World-Model.
Read the full abstract ▾
All authors Jindi Kong, Yuting He, Cong Xia, Rongjun Ge, Shuo Li
Computer Vision
A Two-Stage Detection-Tracking Framework for Stable Apple Quality Inspection in Dense Conveyor-Belt Environments
Keonvin Park, Aditya Pal and Jin Hong Mok
Industrial fruit inspection systems must operate reliably under dense multi-object interactions and continuous motion, yet most existing works evaluate detection or classification at the image level without ensuring temporal stability in
video streams. We present a two-stage detection-tracking framework for stable multi-apple quality inspection in conveyor-belt environments. An orchard-trained YOLOv8 model performs apple localization, followed by ByteTrack multi-object tracking to maintain persistent identities. A ResNet18 defect classifier, fine-tuned on a healthy-defective fruit
dataset, is applied to cropped apple regions. Track-level aggregation is introduced to enforce temporal consistency and reduce prediction oscillation across frames. We define
video-level industrial metrics such as track-level defect ratio and temporal consistency to evaluate system robustness under realistic processing conditions. Results demonstrate improved stability compared to frame-wise inference, suggesting that integrating tracking is essential for practical automated fruit grading systems.
Read the full abstract ▾
Computer Vision
DD-CAM: Minimal Sufficient Explanations for Vision Models Using Delta Debugging
Krishna Khadka, Yu Lei, Raghu N. Kacker et al. (+1 more)
We introduce a gradient-free framework for identifying minimal, sufficient, and decision-preserving explanations in
vision models by isolating the smallest subset of representational units whose joint activation preserves predictions. Unlike existing approaches that aggregate all units, often leading to cluttered saliency maps, our approach, DD-CAM, identifies a 1-minimal subset whose joint activation suffices to preserve the prediction (i.e., removing any unit from the subset alters the prediction). To efficiently isolate minimal sufficient subsets, we adapt delta debugging, a systematic reduction strategy from software debugging, and configure its search strategy based on unit interactions in the classifier head: testing individual units for models with non-interacting units and testing unit combinations for models in which unit interactions exist. We then generate minimal, prediction-preserving saliency maps that highlight only the most essential features. Our experimental evaluation demonstrates that our approach can produce more faithful explanations and achieve higher localization accuracy than the state-of-the-art CAM-based approaches.
Read the full abstract ▾
All authors Krishna Khadka, Yu Lei, Raghu N. Kacker, D. Richard Kuhn
Computer Vision · Cross-listed from cs.AR
CORVET: A CORDIC-Powered, Resource-Frugal Mixed-Precision Vector Processing Engine for High-Throughput AIoT applications
Sonu Kumar, Mohd Faisal Khan, Mukul Lokhande et al. (+1 more)
This brief presents a runtime-adaptive, performance-enhanced vector engine featuring a low-resource, iterative CORDIC-based MAC unit for edge AI acceleration. The proposed design enables dynamic reconfiguration between approximate and accurate modes, exploiting the latency-accuracy trade-off for a wide range of workloads. Its resource-efficient approach further enables up to 4x throughput improvement within the same hardware resources by leveraging vectorised, time-multiplexed execution and flexible precision scaling. With a time-multiplexed multi-AF block and a lightweight pooling and normalisation unit, the proposed vector engine supports flexible precision (4/8/16-bit) and high MAC density. The ASIC implementation results show that each MAC stage can save up to 33% of time and 21% of power, with a 256-PE configuration that achieves higher compute density (4.83 TOPS/mm2 ) and energy efficiency (11.67 TOPS/W) than previous state-of-the-art work. A detailed hardware-software co-design methodology for object detection and classification tasks on Pynq-Z2 is discussed to assess the proposed architecture, demonstrating a scalable, energy-efficient solution for edge AI applications.
Read the full abstract ▾
All authors Sonu Kumar, Mohd Faisal Khan, Mukul Lokhande, Santosh Kumar Vishvakarma
Computer Vision
RegionRoute: Regional Style Transfer with Diffusion Model
Bowen Chen, Jake Zuena, Alan C. Bovik et al. (+1 more)
Precise spatial control in
diffusion-based style transfer remains challenging. This challenge arises because
diffusion models treat style as a global feature and lack explicit spatial grounding of style representations, making it difficult to restrict style application to specific objects or regions. To our knowledge, existing
diffusion models are unable to perform true localized style transfer, typically relying on handcrafted masks or multi-stage post-processing that introduce boundary artifacts and limit generalization. To address this, we propose an
attention-supervised
diffusion framework that explicitly teaches the model where to apply a given style by aligning the
attention scores of style
tokens with object masks during training. Two complementary objectives, a Focus loss based on KL divergence and a Cover loss using binary cross-
entropy, jointly encourage accurate localization and dense coverage. A modular LoRA-MoE design further enables efficient and scalable multi-style adaptation. To evaluate localized stylization, we introduce the Regional Style Editing Score, which measures Regional Style Matching through CLIP-based similarity within the target region and Identity Preservation via masked LPIPS and pixel-level consistency on unedited areas. Experiments show that our method achieves mask-free, single-object style transfer at inference, producing regionally accurate and visually coherent results that outperform existing
diffusion-based editing approaches.
Read the full abstract ▾
All authors Bowen Chen, Jake Zuena, Alan C. Bovik, Divya Kothandaraman
Computer Vision
No Need For Real Anomaly: MLLM Empowered Zero-Shot Video Anomaly Detection
Zunkai Dai, Ke Li, Jiajia Liu et al. (+2 more)
The collection and detection of
video anomaly data has long been a challenging problem due to its rare occurrence and spatio-temporal scarcity. Existing
video anomaly detection (VAD) methods under perform in open-world scenarios. Key contributing factors include limited
dataset diversity, and inadequate understanding of context-dependent anomalous semantics. To address these issues, i) we propose LAVIDA, an end-to-end zero-shot
video anomaly detection framework. ii) LAVIDA employs an Anomaly Exposure Sampler that transforms segmented objects into pseudo-anomalies to enhance model adaptability to unseen anomaly categories. It further integrates a
Multimodal Large
Language Model (MLLM) to bolster semantic comprehension capabilities. Additionally, iii) we design a
token compression approach based on reverse
attention to handle the spatio-temporal scarcity of anomalous patterns and decrease computational cost. The training process is conducted solely on pseudo anomalies without any VAD data. Evaluations across four
benchmark VAD
datasets demonstrate that LAVIDA achieves SOTA performance in both frame-level and pixel-level anomaly detection under the zero-shot setting. Our code is available in https://github.com/VitaminCreed/LAVIDA.
Read the full abstract ▾
All authors Zunkai Dai, Ke Li, Jiajia Liu, Jie Yang, Yuanyuan Qiao
Computer Vision
Knowledge-aware Visual Question Generation for Remote Sensing Images
Siran Li, Li Mi, Javiera Castillo-Navarro et al. (+1 more)
With the rapid development of remote sensing image archives, asking questions about images has become an effective way of gathering specific information or performing image
retrieval. However, automatically generated image-based questions tend to be simplistic and template-based, which hinders the real deployment of question answering or visual dialogue systems. To enrich and diversify the questions, we propose a knowledge-aware remote sensing visual question generation model, KRSVQG, that incorporates external knowledge related to the image content to improve the quality and contextual understanding of the generated questions. The model takes an image and a related knowledge triplet from external knowledge sources as inputs and leverages image captioning as an intermediary representation to enhance the image grounding of the generated questions. To assess the performance of KRSVQG, we utilized two
datasets that we manually annotated: NWPU-300 and TextRS-300. Results on these two
datasets demonstrate that KRSVQG outperforms existing methods and leads to knowledge-enriched questions, grounded in both image and domain knowledge.
Read the full abstract ▾
All authors Siran Li, Li Mi, Javiera Castillo-Navarro, Devis Tuia
Computer Vision
Controlled Face Manipulation and Synthesis for Data Augmentation
Joris Kirchner, Amogh Gudi, Marian Bittner et al. (+1 more)
Deep learning
vision models excel with abundant supervision, but many applications face label scarcity and class imbalance. Controllable image editing can augment scarce labeled data, yet edits often introduce artifacts and entangle non-target attributes. We study this in facial expression analysis, targeting Action Unit (AU) manipulation where annotation is costly and AU co-activation drives
entanglement. We present a facial manipulation method that operates in the semantic latent space of a pre-trained face generator (
Diffusion Autoencoder). Using lightweight linear models, we reduce
entanglement of semantic features via (i) dependency-aware conditioning that accounts for AU co-activation, and (ii) orthogonal projection that removes nuisance attribute directions (e.g., glasses), together with an expression neutralization step to enable absolute AU edit. We use these edits to balance AU occurrence by editing labeled faces and to diversify identities/demographics via controlled synthesis. Augmenting AU detector training with the generated data improves accuracy and yields more disentangled predictions with fewer co-activation shortcuts, outperforming alternative data-efficient training strategies and suggesting improvements similar to what would require substantially more labeled data in our learning-curve analysis. Compared to prior methods, our edits are stronger, produce fewer artifacts, and preserve identity better.
Read the full abstract ▾
All authors Joris Kirchner, Amogh Gudi, Marian Bittner, Chirag Raman
Computer Vision
Questions beyond Pixels: Integrating Commonsense Knowledge in Visual Question Generation for Remote Sensing
Siran Li, Li Mi, Javiera Castillo-Navarro et al. (+1 more)
With the rapid development of remote sensing image archives, asking questions about images has become an effective way of gathering specific information or performing semantic image
retrieval. However, current automatically generated questions tend to be simplistic and template-based, which hinders the deployment of question answering or visual dialogue systems for real-world applications. To enrich and diversify the questions with both image content and commonsense knowledge, we propose a Knowledge-aware Remote Sensing Visual Question Generation model (KRSVQG). The proposed model incorporates related knowledge triplets from external knowledge sources to broaden the question content, while employing image captioning as an intermediary representation to ground questions to the corresponding images. Moreover, KRSVQG utilizes a
vision-
language pre-training and fine-tuning strategy, enabling the model's adaptation to low data regimes. To evaluate the proposed KRSVQG model, we construct two knowledge-aware remote sensing visual question generation
datasets: the NWPU-300
dataset and the TextRS-300
dataset. Evaluations, including metrics and human assessment, demonstrate that KRSVQG outperforms existing methods and leads to rich questions, grounded in both image and domain knowledge. As a key practice in
vision-
language research, knowledge-aware visual question generation advances the understanding of image content beyond pixels, facilitating the development of knowledge-enriched
vision-
language systems with
vision-grounded human commonsense.
Read the full abstract ▾
All authors Siran Li, Li Mi, Javiera Castillo-Navarro, Devis Tuia
Computer Vision
SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation
Yujie Lu, Jingwen Li, Sibo Ju et al. (+5 more)
Medical image
segmentation is vital for clinical diagnosis and quantitative analysis, yet remains challenging due to the heterogeneity of imaging modalities and the high cost of pixel-level annotations. Although general interactive
segmentation models like SAM have achieved remarkable progress, their transfer to medical imaging still faces two key bottlenecks: (i) the lack of adaptive mechanisms for modality- and anatomy-specific tasks, which limits generalization in out-of-distribution medical scenarios; and (ii) current medical adaptation methods fine-tune on large, heterogeneous
datasets without selection, leading to noisy supervision, higher cost, and negative transfer. To address these issues, we propose SegMoTE, an efficient and adaptive framework for medical image
segmentation. SegMoTE preserves SAM's original
prompt interface, efficient inference, and zero-shot generalization while introducing only a small number of learnable parameters to dynamically adapt across modalities and tasks. In addition, we design a progressive
prompt tokenization mechanism that enables fully automatic
segmentation, significantly reducing annotation dependence. Trained on MedSeg-HQ, a curated
dataset less than 1% of existing large-scale
datasets, SegMoTE achieves SOTA performance across diverse imaging modalities and anatomical tasks. It represents the first efficient, robust, and scalable adaptation of general
segmentation models to the medical domain under extremely low annotation cost, advancing the practical deployment of foundation
vision models in clinical applications.
Read the full abstract ▾
All authors Yujie Lu, Jingwen Li, Sibo Ju, Yanzhou Su, he yao, Yisong Liu, Min Zhu, Junlong Cheng
Computer Vision
GS-CLIP: Zero-shot 3D Anomaly Detection by Geometry-Aware Prompt and Synergistic View Representation Learning
Zehao Deng, An Liu and Yan Wang
Zero-shot 3D Anomaly Detection is an emerging task that aims to detect anomalies in a target
dataset without any target training data, which is particularly important in scenarios constrained by sample scarcity and data
privacy concerns. While current methods adapt CLIP by projecting 3D point clouds into 2D representations, they face challenges. The projection inherently loses some
geometric details, and the reliance on a single 2D modality provides an incomplete visual understanding, limiting their ability to detect diverse anomaly types. To address these limitations, we propose the Geometry-Aware
Prompt and Synergistic View Representation Learning (GS-CLIP) framework, which enables the model to identify
geometric anomalies through a two-stage learning process. In stage 1, we dynamically generate text
prompts embedded with 3D
geometric priors. These
prompts contain global shape context and local defect information distilled by our
Geometric Defect
Distillation Module (GDDM). In stage 2, we introduce Synergistic View Representation Learning architecture that processes rendered and depth images in parallel. A Synergistic Refinement Module (SRM) subsequently fuses the features of both streams, capitalizing on their complementary strengths. Comprehensive experimental results on four large-scale public
datasets show that GS-CLIP achieves superior performance in detection. Code can be available at https://github.com/zhushengxinyue/GS-CLIP.
Read the full abstract ▾
Computer Vision
UniE2F: A Unified Diffusion Framework for Event-to-Frame Reconstruction with Video Foundation Models
Gang Xu, Zhiyu Zhu and Junhui Hou
Event cameras excel at high-speed, low-power, and high-dynamic-range scene perception. However, as they fundamentally record only relative intensity changes rather than absolute intensity, the resulting data streams suffer from a significant loss of spatial information and static texture details. In this paper, we address this limitation by leveraging the generative prior of a pre-trained
video diffusion model to reconstruct high-fidelity
video frames from sparse event data. Specifically, we first establish a baseline model by directly applying event data as a condition to synthesize
videos. Then, based on the physical correlation between the event stream and
video frames, we further introduce the event-based inter-frame residual guidance to enhance the accuracy of
video frame reconstruction. Furthermore, we extend our method to
video frame interpolation and prediction in a zero-shot manner by modulating the reverse
diffusion sampling process, thereby creating a unified event-to-frame reconstruction framework. Experimental results on real-world and synthetic
datasets demonstrate that our method outperforms previous reconstruction approaches on most quantitative metrics and achieves superior qualitative results, while achieving competitive zero-shot performance compared with dedicated methods explicitly trained for interpolation and prediction tasks. We also refer the reviewers to the
video demo contained in the supplementary material for
video results. The code will be publicly available at https://github.com/CS-GangXu/UniE2F.
Read the full abstract ▾
Computer Vision
Prompt Tuning for CLIP on the Pretrained Manifold
Xi Yang, Yuanrong Xu, Weigang Zhang et al. (+3 more)
Prompt tuning introduces learnable
prompt vectors that adapt pretrained
vision-
language models to downstream tasks in a parameter-efficient manner. However, under limited supervision,
prompt tuning alters pretrained representations and drives downstream features away from the pretrained
manifold toward directions that are unfavorable for transfer. This drift degrades generalization. To address this limitation, we propose ManiPT, a framework that performs
prompt tuning on the pretrained
manifold. ManiPT introduces cosine consistency constraints in both the text and image modalities to confine the learned representations within the pretrained
geometric neighborhood. Furthermore, we introduce a structural bias that enforces incremental corrections, guiding the adaptation along transferable directions to mitigate reliance on shortcut learning. From a theoretical perspective, ManiPT alleviates overfitting tendencies under limited data. Our experiments cover four downstream settings: unseen-class generalization, few-shot classification, cross-
dataset transfer, and domain generalization. Across these settings, ManiPT achieves higher average performance than baseline methods. Notably, ManiPT provides an explicit perspective on how
prompt tuning overfits under limited supervision.
Read the full abstract ▾
All authors Xi Yang, Yuanrong Xu, Weigang Zhang, Guangming Lu, David Zhang, Jie Wen
Computer Vision
FUSAR-GPT : A Spatiotemporal Feature-Embedded and Two-Stage Decoupled Visual Language Model for SAR Imagery
Xiaokun Zhang, Yi Yang, Ziqi Ye et al. (+6 more)
Research on the intelligent interpretation of all-weather, all-time Synthetic Aperture Radar (SAR) is crucial for advancing remote sensing applications. In recent years, although Visual
Language Models (VLMs) have demonstrated strong open-world understanding capabilities on RGB images, their performance is severely limited when directly applied to the SAR field due to the complexity of the imaging mechanism, sensitivity to scattering features, and the scarcity of high-quality text corpora. To systematically address this issue, we constructed the inaugural SAR Image-Text-AlphaEarth feature triplet
dataset and developed FUSAR-GPT, a VLM specifically for SAR. FUSAR-GPT innovatively introduces a geospatial baseline model as a 'world knowledge' prior and embeds multi-source remote-sensing temporal features into the model's visual backbone via 'spatiotemporal anchors', enabling dynamic compensation for the sparse representation of targets in SAR images. Furthermore, we designed a two-stage SFT strategy to decouple the knowledge injection and task execution of large models. The spatiotemporal feature
embedding and the two-stage decoupling paradigm enable FUSAR-GPT to achieve state-of-the-art performance across several typical remote sensing visual-
language benchmark tests, significantly outperforming mainstream baseline models by over 10%.
Read the full abstract ▾
All authors Xiaokun Zhang, Yi Yang, Ziqi Ye, Baiyun, Xiaorong Guo, Qingchen Fang, Ruyi Zhang, Xinpeng Zhou, Haipeng Wang
Computer Vision
PositionOCR: Augmenting Positional Awareness in Multi-Modal Models via Hybrid Specialist Integration
Chen Duan, Zhentao Guo, Pei Fu et al. (+3 more)
In recent years, Multi-modal Large
Language Models (MLLMs) have achieved strong performance in OCR-centric Visual Question Answering (VQA) tasks, illustrating their capability to process heterogeneous data and exhibit adaptability across varied contexts. However, these MLLMs rely on a Large
Language Model (LLM) as the decoder, which is primarily designed for linguistic processing, and thus inherently lacks the positional
reasoning required for precise visual tasks, such as text spotting and text grounding. Additionally, the extensive parameters of MLLMs necessitate substantial computational resources and large-scale data for effective training. Conversely, text spotting specialists achieve state-of-the-art coordinate predictions but lack semantic
reasoning capabilities. This dichotomy motivates our key research question: Can we synergize the efficiency of specialists with the contextual power of LLMs to create a positionally-accurate MLLM? To overcome these challenges, we introduce PositionOCR, a parameter-efficient hybrid architecture that seamlessly integrates a text spotting model's positional strengths with an LLM's contextual
reasoning. Comprising 131M trainable parameters, this framework demonstrates outstanding multi-modal processing capabilities, particularly excelling in tasks such as text grounding and text spotting, consistently surpassing traditional MLLMs.
Read the full abstract ▾
All authors Chen Duan, Zhentao Guo, Pei Fu, Zining Wang, Kai Zhou, Pengfei Yan
Computer Vision
VLM-Guided Group Preference Alignment for Diffusion-based Human Mesh Recovery
Wenhao Shen, Hao Wang, Wanqi Yin et al. (+5 more)
Human mesh recovery (HMR) from a single RGB image is inherently ambiguous, as multiple 3D poses can correspond to the same 2D observation. Recent
diffusion-based methods tackle this by generating various hypotheses, but often sacrifice accuracy. They yield predictions that are either physically implausible or drift from the input image, especially under occlusion or in cluttered, in-the-wild scenes. To address this, we introduce a dual-
memory augmented HMR critique
agent with self-reflection to produce context-aware quality scores for predicted meshes. These scores distill fine-grained cues about 3D human motion structure, physical feasibility, and
alignment with the input image. We use these scores to build a group-wise HMR preference
dataset. Leveraging this
dataset, we propose a group preference
alignment framework for finetuning
diffusion-based HMR models. This process injects the rich preference signals into the model, guiding it to generate more physically plausible and image-consistent human meshes. Extensive experiments demonstrate that our method achieves superior performance compared to state-of-the-art approaches.
Read the full abstract ▾
All authors Wenhao Shen, Hao Wang, Wanqi Yin, Fayao Liu, Xulei Yang, Chao Liang, Zhongang Cai, Guosheng Lin
Computer Vision
EMAD: Evidence-Centric Grounded Multimodal Diagnosis for Alzheimer's Disease
Qiuhui Chen, Xuancheng Yao, Zhenglei Zhou et al. (+2 more)
Deep learning models for medical image analysis often act as black boxes, seldom aligning with clinical guidelines or explicitly linking decisions to supporting evidence. This is especially critical in Alzheimer's disease (AD), where predictions should be grounded in both anatomical and clinical findings. We present EMAD, a
vision-
language framework that generates structured AD diagnostic reports in which each claim is explicitly grounded in
multimodal evidence. EMAD uses a hierarchical Sentence-Evidence-Anatomy (SEA) grounding mechanism: (i) sentence-to-evidence grounding links generated sentences to clinical evidence phrases, and (ii) evidence-to-anatomy grounding localizes corresponding structures on 3D
brain MRI. To reduce dense annotation requirements, we propose GTX-Distill, which transfers grounding behavior from a teacher trained with limited supervision to a student operating on model-generated reports. We further introduce Executable-Rule GRPO, a reinforcement fine-tuning scheme with verifiable rewards that enforces clinical consistency, protocol adherence, and
reasoning-diagnosis coherence. On the AD-MultiSense
dataset, EMAD achieves state-of-the-art diagnostic accuracy and produces more transparent, anatomically faithful reports than existing methods. We will release code and grounding annotations to support future research in trustworthy medical
vision-
language models.
Read the full abstract ▾
All authors Qiuhui Chen, Xuancheng Yao, Zhenglei Zhou, Xinyue Hu, Yi Hong
Computer Vision
BriMA: Bridged Modality Adaptation for Multi-Modal Continual Action Quality Assessment
Kanglei Zhou, Chang Li, Qingyi Pan et al. (+1 more)
Action Quality Assessment (AQA) aims to score how well an action is performed and is widely used in sports analysis, rehabilitation assessment, and human skill evaluation. Multi-modal AQA has recently achieved strong progress by leveraging complementary visual and kinematic cues, yet real-world deployments often suffer from non-stationary modality imbalance, where certain modalities become missing or intermittently available due to sensor failures or annotation gaps. Existing continual AQA methods overlook this issue and assume that all modalities remain complete and stable throughout training, which restricts their practicality. To address this challenge, we introduce Bridged Modality Adaptation (BriMA), an innovative approach to multi-modal continual AQA under modality-missing conditions. BriMA consists of a
memory-guided bridging imputation module that reconstructs missing modalities using both task-agnostic and task-specific representations, and a modality-aware replay mechanism that prioritizes informative samples based on modality distortion and distribution drift. Experiments on three representative multi-modal AQA
datasets (RG, Fis-V, and FS1000) show that BriMA consistently improves performance under different modality-missing conditions, achieving 6–8% higher correlation and 12–15% lower error on average. These results demonstrate a step toward robust multi-modal AQA systems under real-world deployment constraints.
Read the full abstract ▾
All authors Kanglei Zhou, Chang Li, Qingyi Pan, Liyuan Wang
Computer Vision
JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation
Kai Liu, Yanhao Zheng, Kai Wang et al. (+7 more)
AIGC has rapidly expanded from text-to-image generation toward high-quality
multimodal synthesis across
video and audio. Within this context, joint audio-
video generation (JAVG) has emerged as a fundamental task that produces synchronized and semantically aligned sound and
vision from textual descriptions. However, compared with advanced commercial models such as Veo3, existing open-source methods still suffer from limitations in generation quality, temporal synchrony, and
alignment with human preferences. To bridge the gap, this paper presents JavisDiT++, a concise yet powerful framework for unified modeling and optimization of JAVG. First, we introduce a modality-specific mixture-of-experts (MS-MoE) design that enables cross-modal interaction efficacy while enhancing single-modal generation quality. Then, we propose a temporal-aligned RoPE (TA-RoPE) strategy to achieve explicit, frame-level synchronization between audio and
video tokens. Besides, we develop an audio-
video direct preference optimization (AV-DPO) method to align model outputs with human preference across quality, consistency, and synchrony dimensions. Built upon Wan2.1-1.3B-T2V, our model achieves state-of-the-art performance merely with around 1M public training entries, significantly outperforming prior approaches in both qualitative and quantitative evaluations. Comprehensive ablation studies have been conducted to validate the effectiveness of our proposed modules. All the code, model, and
dataset are released at https://JavisVerse.github.io/JavisDiT2-page.
Read the full abstract ▾
All authors Kai Liu, Yanhao Zheng, Kai Wang, Shengqiong Wu, Rongjunchen Zhang, Jiebo Luo, Dimitrios Hatzinakos, Ziwei Liu, Hao Fei, Tat-Seng Chua
Computer Vision
Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation
Lunjie Zhu, Yushi Huang, Xingtong Ge et al. (+5 more)
Latent
diffusion models have enabled high-quality
video synthesis, yet their inference remains costly and time-consuming. As
diffusion transformers become increasingly efficient, the latency bottleneck inevitably shifts to VAE decoders. To reduce their latency while maintaining quality, we propose a universal acceleration framework for VAE decoders that preserves full
alignment with the original latent distribution. Specifically, we propose (1) an independence-aware channel
pruning method to effectively mitigate severe channel redundancy, and (2) a stage-wise dominant operator optimization strategy to address the high inference cost of the widely used causal 3D convolutions in VAE decoders. Based on these innovations, we construct a Flash-VAED family. Moreover, we design a three-phase dynamic
distillation framework that efficiently transfers the capabilities of the original VAE decoder to Flash-VAED. Extensive experiments on Wan and LTX-
Video VAE decoders demonstrate that our method outperforms baselines in both quality and speed, achieving approximately a 6$\times$ speedup while maintaining the reconstruction performance up to 96.9%. Notably, Flash-VAED accelerates the end-to-end generation pipeline by up to 36% with negligible quality drops on VBench-2.0. Our code is available at https://github.com/Aoko955/Flash-VAED.
Read the full abstract ▾
All authors Lunjie Zhu, Yushi Huang, Xingtong Ge, Yufei Xue, Zhening Liu, Yumeng Zhang, Zehong Lin, Jun Zhang
Computer Vision
Artefact-Aware Fungal Detection in Dermatophytosis: A Real-Time Transformer-Based Approach for KOH Microscopy
Rana Gursoy, Abdurrahim Yilmaz, Baris Kizilyaprak et al. (+5 more)
Dermatophytosis is commonly assessed using potassium hydroxide (KOH) microscopy, yet accurate recognition of fungal hyphae is hindered by artefacts, heterogeneous keratin clearance, and notable inter-observer variability. This study presents a
transformer-based detection framework using the RT-DETR model architecture to achieve precise, query-driven localization of fungal structures in high-resolution KOH images. A
dataset of 2,540 routinely acquired microscopy images was manually annotated using a multi-class strategy to explicitly distinguish fungal elements from confounding artefacts. The model was trained with morphology-preserving augmentations to maintain the structural integrity of thin hyphae. Evaluation on an independent test set demonstrated robust object-level performance, with a recall of 0.9737, precision of 0.8043, and an AP@0.50 of 93.56%. When aggregated for image-level diagnosis, the model achieved 100% sensitivity and 98.8% accuracy, correctly identifying all positive cases without missing a single diagnosis. Qualitative outputs confirmed the robust localization of low-contrast hyphae even in artefact-rich fields. These results highlight that an artificial intelligence (AI) system can serve as a highly reliable, automated screening tool, effectively bridging the gap between image-level analysis and clinical decision-making in dermatomycology.
Read the full abstract ▾
All authors Rana Gursoy, Abdurrahim Yilmaz, Baris Kizilyaprak, Esmahan Caglar, Burak Temelkuran, Huseyin Uvet, Ayse Esra Koku Aksu, Gulsum Gencoglan
Computer Vision
VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrieval
Diogo Glória-Silva, David Semedo and João Maglhães
We introduce VIGiA, a novel
multimodal dialogue model designed to understand and reason over complex, multi-step instructional
video action plans. Unlike prior work which focuses mainly on text-only guidance, or treats
vision and
language in isolation, VIGiA supports grounded, plan-aware dialogue that requires
reasoning over visual inputs, instructional plans, and interleaved user interactions. To this end, VIGiA incorporates two key capabilities: (1)
multimodal plan
reasoning, enabling the model to align uni- and
multimodal queries with the current task plan and respond accurately; and (2) plan-based
retrieval, allowing it to retrieve relevant plan steps in either textual or visual representations. Experiments were done on a novel
dataset with rich Instructional
Video Dialogues aligned with Cooking and DIY plans. Our evaluation shows that VIGiA outperforms existing state-of-the-art models on all tasks in a conversational plan guidance setting, reaching over 90% accuracy on plan-aware VQA.
Read the full abstract ▾
Computer Vision
CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusion
Sijie Mai and Shiqin Han
Modality gap significantly restricts the effectiveness of
multimodal fusion. Previous methods often use techniques such as
diffusion models and
adversarial learning to reduce the modality gap, but they typically focus on one-to-one
alignment without exposing the data points of the source modality to the global distribution information of the target modality. To this end, leveraging the characteristic of rectified flow that can map one distribution to another via a straight trajectory, we extend rectified flow for modality distribution mapping. Specifically, we leverage the `one-to-many mapping' strategy in rectified flow that allows each data point of the source modality to observe the overall target distribution. This also alleviates the issue of insufficient paired data within each sample, enabling a more robust distribution transformation. Moreover, to achieve more accurate distribution mapping and address the ambiguous flow directions in one-to-many mapping, we design `adaptive relaxed
alignment', enforcing stricter
alignment for modality pairs belonging to the same sample, while applying relaxed mapping for pairs not belonging to the same sample or category. Additionally, to prevent information loss during distribution mapping, we introduce `cyclic rectified flow' to ensure the transferred features can be translated back to the original features, allowing
multimodal representations to learn sufficient modality-specific information. After distribution
alignment, our approach achieves very competitive results on multiple tasks of
multimodal affective computing even with a simple fusion method, and visualizations verify that it can effectively reduce the modality gap.
Read the full abstract ▾
Computer Vision
Mapping Networks
Lord Sen and Shyamapada Mukherjee
The escalating parameter counts in modern deep learning models pose a fundamental challenge to efficient training and resolution of overfitting. We address this by introducing the Mapping Networks which replace the high dimensional weight space by a compact, trainable latent vector based on the hypothesis that the trained parameters of large networks reside on smooth, low-dimensional
manifolds. Henceforth, the Mapping
Theorem enforced by a dedicated Mapping Loss, shows the existence of a mapping from this latent space to the target weight space both theoretically and in practice. Mapping Networks significantly reduce overfitting and achieve comparable to better performance than target network across complex
vision and sequence tasks, including Image Classification, Deepfake Detection etc, with $\mathbf{99.5\%}$, i.e., around $500\times$ reduction in trainable parameters.
Read the full abstract ▾
Computer Vision
StreetTree: A Large-Scale Global Benchmark for Fine-Grained Tree Species Classification
Jiapeng Li, Yingjing Huang, Fan Zhang et al. (+1 more)
The fine grained classification of street trees is a crucial task for urban planning, streetscape management, and the assessment of urban ecosystem services. However, progress in this field has been hindered by the lack of large scale, geographically diverse, and publicly available
benchmark datasets specifically designed for street trees. To address this critical gap, we introduce StreetTree, the world's first large scale
benchmark dataset dedicated to fine grained street tree classification. The
dataset contains over 12 million images covering more than 8,300 common street tree species, collected from urban streetscapes across 133 countries spanning five continents, and supplemented with expert verified observational data. StreetTree poses challenges for pretrained
vision models under complex urban environments including high inter species visual similarity, long tailed natural distributions, significant intra class variations caused by seasonal changes, and diverse imaging conditions such as lighting, occlusions from buildings, and varying camera angles. In addition, we provide a hierarchical taxonomy (order, family, genus, and species) to support research in hierarchical classification and representation learning. Through extensive experiments with various
vision models, we establish solid baselines and reveal the limitations of existing methods in handling such real world complexities. We believe that StreetTree will serve as a key resource for driving new advancements at the intersection of computer
vision and urban science.
Read the full abstract ▾
All authors Jiapeng Li, Yingjing Huang, Fan Zhang, Yu liu
Computer Vision
Keep it SymPL: Symbolic Projective Layout for Allocentric Spatial Reasoning in Vision-Language Models
Jaeyun Jang, Seunghui Shin, Taeho Park et al. (+1 more)
Perspective-aware spatial
reasoning involves understanding spatial relationships from specific viewpoints-either
egocentric (observer-centered) or allocentric (object-centered). While
vision-
language models (VLMs) perform well in
egocentric settings, their performance deteriorates when
reasoning from allocentric viewpoints, where spatial relations must be inferred from the perspective of objects within the scene. In this study, we address this underexplored challenge by introducing Symbolic Projective Layout (SymPL), a framework that reformulates allocentric
reasoning into symbolic-layout forms that VLMs inherently handle well. By leveraging four key factors-projection, abstraction, bipartition, and localization-SymPL converts allocentric questions into structured symbolic-layout representations. Extensive experiments demonstrate that this reformulation substantially improves performance in both allocentric and
egocentric tasks, enhances robustness under visual illusions and multi-view scenarios, and that each component contributes critically to these gains. These results show that SymPL provides an effective and principled approach for addressing complex perspective-aware spatial
reasoning.
Read the full abstract ▾
All authors Jaeyun Jang, Seunghui Shin, Taeho Park, Hyoseok Hwang