Computer Vision
An Image Dataset of Common Skin Diseases of Bangladesh and Benchmarking Performance with Machine Learning Models
Sazzad Hossain, Saiful Islam, Muhammad Ibrahim et al. (+3 more)
Skin diseases are a major public health concern worldwide, and their detection is often challenging without access to dermatological expertise. In countries like Bangladesh, which is highly populated, the number of qualified skin specialists and diagnostic instruments is insufficient to meet the demand. Due to the lack of proper detection and treatment of skin diseases, that may lead to severe health consequences including death. Common properties of skin diseases are, changing the color, texture, and pattern of skin and in this era of artificial intelligence and machine learning, we are able to detect skin diseases by using image processing and computer
vision techniques. In response to this challenge, we develop a publicly available
dataset focused on common skin disease detection using machine learning techniques. We focus on five prevalent skin diseases in Bangladesh: Contact Dermatitis, Vitiligo, Eczema, Scabies, and Tinea Ringworm. The
dataset consists of 1612 images (of which, 250 are distinct while others are augmented), collected directly from patients at the outpatient department of Faridpur Medical College, Faridpur, Bangladesh. The data comprises of 302, 381, 301, 316, and 312 images of Dermatitis, Eczema, Scabies, Tinea Ringworm, and Vitiligo, respectively. Although the data are collected regionally, the selected diseases are common across many countries especially in South Asia, making the
dataset potentially valuable for global applications in machine learning-based dermatology. We also apply several machine learning and deep learning models on the
dataset and report classification performance. We expect that this research would garner
attention from machine learning and deep learning researchers and practitioners working in the field of automated disease diagnosis.
Read the full abstract ▾
All authors Sazzad Hossain, Saiful Islam, Muhammad Ibrahim, Md. Rasel Ahmed, Md Shuayb, Ahmedul Kabir
Computer Vision
Training-free Detection and 6D Pose Estimation of Unseen Surgical Instruments
Jonas Hein, Lilian Calvet, Matthias Seibold et al. (+3 more)
Purpose: Accurate detection and 6D pose estimation of
surgical instruments are crucial for many computer-assisted interventions. However, supervised methods lack flexibility for new or unseen tools and require extensive annotated data. This work introduces a training-free pipeline for accurate multi-view 6D pose estimation of unseen
surgical instruments, which only requires a textured CAD model as prior knowledge. Methods: Our pipeline consists of two main stages. First, for detection, we generate object mask proposals in each view and score their similarity to rendered templates using a pre-trained feature extractor. Detections are matched across views, triangulated into 3D instance candidates, and filtered using multi-view
geometric consistency. Second, for pose estimation, a set of pose hypotheses is iteratively refined and scored using feature-metric scores with cross-view
attention. The best hypothesis undergoes a final refinement using a novel multi-view, occlusion-aware contour registration, which minimizes reprojection errors of unoccluded contour points. Results: The proposed method was rigorously evaluated on real-world
surgical data from the MVPSP
dataset. The method achieves millimeter-accurate pose estimates that are on par with supervised methods under controlled conditions, while maintaining full generalization to unseen instruments. These results demonstrate the feasibility of training-free, marker-less detection and tracking in
surgical scenes, and highlight the unique challenges in
surgical environments. Conclusion: We present a novel and flexible pipeline that effectively combines state-of-the-art foundational models, multi-view geometry, and contour-based refinement for high-accuracy 6D pose estimation of
surgical instruments without task-specific training. This approach enables robust instrument tracking and scene understanding in dynamic clinical environments.
Read the full abstract ▾
All authors Jonas Hein, Lilian Calvet, Matthias Seibold, Siyu Tang, Marc Pollefeys, Philipp Fürnstahl
Computer Vision
SDD-YOLO: A Small-Target Detection Framework for Ground-to-Air Anti-UAV Surveillance with Edge-Efficient Deployment
Pengyu Chen, Haotian Sa, Yiwei Hu et al. (+2 more)
Detecting small unmanned aerial vehicles (
UAVs) from a ground-to-air (G2A) perspective presents significant challenges, including extremely low pixel occupancy, cluttered aerial backgrounds, and strict real-time constraints. Existing YOLO-based detectors are primarily optimized for general object detection and often lack adequate feature resolution for sub-pixel targets, while introducing complexities during deployment. In this paper, we propose SDD-YOLO, a small-target detection framework tailored for G2A anti-
UAV surveillance. To capture fine-grained spatial details critical for micro-targets, SDD-YOLO introduces a P2 high-resolution detection head operating at 4 times downsampling. Furthermore, we integrate the recent architectural advancements from YOLO26, including a DFL-free, NMS-free architecture for streamlined inference, and the MuSGD hybrid training strategy with ProgLoss and STAL, which substantially mitigates gradient oscillation on sparse small-target signals. To support our evaluation, we construct DroneSOD-30K, a large-scale G2A
dataset comprising approximately 30,000 annotated images covering diverse meteorological conditions. Experiments demonstrate that SDD-YOLO-n achieves a mAP@0.5 of 86.0% on DroneSOD-30K, surpassing the YOLOv5n baseline by 7.8 percentage points. Extensive inference analysis shows our model attains 226 FPS on an NVIDIA RTX 5090 and 35 FPS on an Intel Xeon CPU, demonstrating exceptional efficiency for future edge deployment.
Read the full abstract ▾
All authors Pengyu Chen, Haotian Sa, Yiwei Hu, Yuhan Cheng, Junbo Wang
Computer Vision
Free-Lunch Long Video Generation via Layer-Adaptive O.O.D Correction
Jiahao Tian, Chenxi Song, Wei Cheng et al. (+1 more)
Generating long
videos using pre-trained
video diffusion models, which are typically trained on short clips, presents a significant challenge. Directly applying these models for long-
video inference often leads to a notable degradation in visual quality. This paper identifies that this issue primarily stems from two out-of-distribution (O.O.D) problems: frame-level relative position O.O.D and context-length O.O.D. To address these challenges, we propose FreeLOC, a novel training-free, layer-adaptive framework that introduces two core techniques:
Video-based Relative Position Re-encoding (VRPR) for frame-level relative position O.O.D, a multi-granularity strategy that hierarchically re-encodes temporal relative positions to align with the model's pre-trained distribution, and Tiered Sparse
Attention (TSA) for context-length O.O.D, which preserves both local detail and long-range dependencies by structuring
attention density across different temporal scales. Crucially, we introduce a layer-adaptive probing mechanism that identifies the sensitivity of each
transformer layer to these O.O.D issues, allowing for the selective and efficient application of our methods. Extensive experiments demonstrate that our approach significantly outperforms existing training-free methods, achieving state-of-the-art results in both temporal consistency and visual quality. Code is available at https://github.com/Westlake-AGI-Lab/FreeLOC.
Read the full abstract ▾
All authors Jiahao Tian, Chenxi Song, Wei Cheng, Chi Zhang
Computer Vision
Probabilistic Concept Graph Reasoning for Multimodal Misinformation Detection
Ruichao Yang, Wei Gao, Xiaobin Zhu et al. (+5 more)
Multimodal misinformation poses an escalating challenge that often evades traditional detectors, which are opaque black boxes and fragile against new manipulation tactics. We present Probabilistic Concept Graph
Reasoning (PCGR), an interpretable and evolvable framework that reframes
multimodal misinformation detection (MMD) as structured and concept-based
reasoning. PCGR follows a build-then-infer paradigm, which first constructs a graph of human-understandable concept nodes, including novel high-level concepts automatically discovered and validated by
multimodal large
language models (MLLMs), and then applies hierarchical
attention over this concept graph to infer claim veracity. This design produces interpretable
reasoning chains linking evidence to conclusions. Experiments demonstrate that PCGR achieves state-of-the-art MMD accuracy and robustness to emerging manipulation types, outperforming prior methods in both coarse detection and fine-grained manipulation recognition.
Read the full abstract ▾
All authors Ruichao Yang, Wei Gao, Xiaobin Zhu, Jing Ma, Hongzhan Lin, Ziyang Luo, Bo-Wen Zhang, Xu-Cheng Yin
Computer Vision
CIV-DG: Conditional Instrumental Variables for Domain Generalization in Medical Imaging
Shaojin Bai, Yuting Su and Weizhi Nie
Cross-site generalizability in medical AI is fundamentally compromised by selection bias, a structural mechanism where patient demographics (e.g., age, severity) non-randomly dictate hospital assignment. Conventional Domain Generalization (DG) paradigms, which predominantly target image-level distribution shifts, fail to address the resulting spurious correlations between site-specific variations and diagnostic labels. To surmount this identifiability barrier, we propose CIV-DG, a causal framework that leverages Conditional Instrumental Variables to disentangle pathological semantics from scanner-induced artifacts. By relaxing the strict random assignment assumption of standard IV methods, CIV-DG accommodates complex clinical scenarios where hospital selection is endogenously driven by patient demographics. We instantiate this theory via a Deep Generalized Method of Moments (DeepGMM) architecture, employing a conditional critic to minimize moment violations and enforce instrument-error orthogonality within demographic strata. Extensive experiments on the Camelyon17
benchmark and large-scale Chest X-Ray
datasets demonstrate that CIV-DG significantly outperforms leading baselines, validating the efficacy of conditional causal mechanisms in resolving structural confounding for robust medical AI.
Read the full abstract ▾
Computer Vision
TacSIm: A Dataset and Benchmark for Football Tactical Style Imitation
Peng Wen, Yuting Wang and Qiurui Wang
Current football imitation research primarily aims to opti mize reward-based objectives, such as goals scored or win rate proxies, paying less
attention to accurately replicat ing real-world team tactical behaviors. We introduce Tac SIm, a large-scale
dataset and
benchmark for Tactical Style Imitation in football. TacSIm imitates the acitons of all 11 players in one team in the given broadcast footage of Pre mier League matches under a single broadcast view. Under a offensive or defensive broadcast footage, TacSIm projects the beginning positions and actions of all 22 players from both sides onto a standard pitch coordinate system. Tac SIm offers an explicit style imitation task and evaluation protocols. Tactics style imitation is measured by using spatial occupancy similarity and movement vector similarity in defined time, supporting the evaluation of spatial and tem poral similarities for one team. We run multiple baseline methods in a unified virtual environment to generate full team behaviors, enabling both quantitative and visual as sessment of tactical coordination. By using unified data and metrics from broadcast to simulation, TacSIm estab lishes a rigorous
benchmark for measuring and modeling style-aligned tactical imitation task in football.
Read the full abstract ▾
Computer Vision
CardioDiT: Latent Diffusion Transformers for 4D Cardiac MRI Synthesis
Marvin Seyfarth, Sarah Kaye Müller, Arman Ghanaat et al. (+7 more)
Latent
diffusion models (LDMs) have recently achieved strong performance in 3D medical image synthesis. However, modalities like cine cardiac MRI (CMR), representing a temporally synchronized 3D volume across the cardiac cycle, add an additional dimension that most generative approaches do not model directly. Instead, they factorize space and time or enforce temporal consistency through auxiliary mechanisms such as anatomical masks. Such strategies introduce structural biases that may limit global context integration and lead to subtle spatiotemporal discontinuities or physiologically inconsistent cardiac dynamics. We investigate whether a unified 4D generative model can learn continuous cardiac dynamics without architectural factorization. We propose CardioDiT, a fully 4D latent
diffusion framework for short-axis cine CMR synthesis based on
diffusion transformers. A spatiotemporal VQ-VAE encodes 2D+t slices into compact latents, which a
diffusion transformer then models jointly as complete 3D+t volumes, coupling space and time throughout the generative process. We evaluate CardioDiT on public CMR
datasets and a larger private cohort, comparing it to baselines with progressively stronger spatiotemporal coupling. Results show improved inter-slice consistency, temporally coherent motion, and realistic cardiac function distributions, suggesting that explicit 4D modeling with a
diffusion transformer provides a principled foundation for spatiotemporal cardiac image synthesis. Code and models trained on public data are available at https://github.com/Cardio-AI/cardiodit.
Read the full abstract ▾
All authors Marvin Seyfarth, Sarah Kaye Müller, Arman Ghanaat, Isabelle Ayx, Fabian Fastenrath, Philipp Wild, Alexander Hertel, Theano Papavassiliu, Salman Ul Hassan Dar, Sandy Engelhardt
Computer Vision
AnyID: Ultra-Fidelity Universal Identity-Preserving Video Generation from Any Visual References
Jiahao Wang, Hualian Sheng, Sijia Cai et al. (+5 more)
Identity-preserving
video generation offers powerful tools for creative expression, allowing users to customize
videos featuring their beloved characters. However, prevailing methods are typically designed and optimized for a single identity reference. This underlying assumption restricts creative flexibility by inadequately accommodating diverse real-world input formats. Relying on a single source also constitutes an ill-posed scenario, causing an inherently ambiguous setting that makes it difficult for the model to faithfully reproduce an identity across novel contexts. To address these issues, we present AnyID, an ultra-fidelity identity-preservation
video generation framework that features two core contributions. First, we introduce a scalable omni-referenced architecture that effectively unifies heterogeneous identity inputs (e.g., faces, portraits, and
videos) into a cohesive representation. Second, we propose a primary-referenced generation paradigm, which designates one reference as a canonical anchor and uses a novel differential
prompt to enable precise, attribute-level controllability. We conduct training on a large-scale, meticulously curated
dataset to ensure robustness and high fidelity, and then perform a final fine-tuning stage using reinforcement learning. This process leverages a preference
dataset constructed from human evaluations, where annotators performed pairwise comparisons of
videos based on two key criteria: identity fidelity and
prompt controllability. Extensive evaluations validate that AnyID achieves ultra-high identity fidelity as well as superior attribute-level controllability across different task settings.
Read the full abstract ▾
All authors Jiahao Wang, Hualian Sheng, Sijia Cai, Yuxiao Yang, Weizhan Zhang, Caixia Yan, Bing Deng, Jieping Ye
Computer Vision
VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers
Marvin Seyfarth, Salman Ul Hassan Dar, Yannik Frisch et al. (+4 more)
Diffusion models have become a leading approach for high-fidelity medical image synthesis. However, most existing methods for 3D medical image generation rely on convolutional U-Net backbones within latent
diffusion frameworks. While effective, these architectures impose strong locality biases and limited receptive fields, which may constrain scalability, global context integration, and flexible conditioning. In this work, we introduce VolDiT, the first purely
transformer-based 3D
Diffusion Transformer for volumetric medical image synthesis. Our approach extends
diffusion transformers to native 3D data through volumetric patch
embeddings and global self-
attention operating directly over 3D
tokens. To enable structured control, we propose a timestep-gated control adapter that maps
segmentation masks into learnable control
tokens that modulate
transformer layers during denoising. This
token-level conditioning mechanism allows precise spatial guidance while preserving the modeling advantages of
transformer architectures. We evaluate our model on high-resolution 3D medical image synthesis tasks and compare it to state-of-the-art 3D latent
diffusion models based on U-Nets. Results demonstrate improved global coherence, superior generative fidelity, and enhanced controllability. Our findings suggest that fully transformerbased
diffusion models provide a flexible foundation for volumetric medical image synthesis. The code and models trained on public data are available at https://github.com/Cardio-AI/voldit.
Read the full abstract ▾
All authors Marvin Seyfarth, Salman Ul Hassan Dar, Yannik Frisch, Philipp Wild, Norbert Frey, Florian André, Sandy Engelhardt
Computer Vision
Bilingual Text-to-Motion Generation: A New Benchmark and Baselines
Wanjiang Weng, Xiaofeng Tan, Xiangbo Shu et al. (+3 more)
Text-to-motion generation holds significant potential for cross-linguistic applications, yet it is hindered by the lack of bilingual
datasets and the poor cross-lingual semantic understanding of existing
language models. To address these gaps, we introduce BiHumanML3D, the first bilingual text-to-motion
benchmark, constructed via LLM-assisted annotation and rigorous manual correction. Furthermore, we propose a simple yet effective baseline, Bilingual Motion
Diffusion (BiMD), featuring Cross-Lingual
Alignment (CLA). CLA explicitly aligns semantic representations across
languages, creating a robust conditional space that enables high-quality motion generation from bilingual inputs, including zero-shot code-switching scenarios. Extensive experiments demonstrate that BiMD with CLA achieves an FID of 0.045 vs. 0.169 and R@3 of 82.8% vs. 80.8%, significantly outperforms monolingual
diffusion models and translation baselines on BiHumanML3D, underscoring the critical necessity and reliability of our
dataset and the effectiveness of our
alignment strategy for cross-lingual motion synthesis. The
dataset and code are released at \href{https://wengwanjiang.github.io/BilingualT2M-page}{https://wengwanjiang.github.io/BilingualT2M-page}
Read the full abstract ▾
All authors Wanjiang Weng, Xiaofeng Tan, Xiangbo Shu, Guo-Sen Xie, Pan Zhou, Hongsong Wang
Computer Vision
AG-EgoPose: Spatially Anchored Residual Correction with Action Context for Monocular Egocentric 3D Pose Estimation
Md Mushfiqur Azam, John Quarles and Kevin Desai
Monocular egocentric 3D pose estimation is difficult because severe foreshortening, self-occlusion, and a restricted field of view often remove the image evidence needed to recover the camera wearer's body. Temporal context can resolve these ambiguities, but unconstrained fusion may also corrupt joints that are already localized reliably from the current frame. We present AG-EgoPose, an action-guided framework that treats temporal information as a bounded-gate residual correction to a spatial pose estimate. A frozen heatmap network produces joint heatmaps and feature-pyramid features, from which a spatial
transformer estimates an independently supervised 3D pose. The heatmaps further guide joint-local pooling of frozen DINOv2 patch
tokens, while DINOv2 frame
tokens are processed by an Ego4D-pretrained ActionFormer to obtain action-scale temporal context. Each spatial joint
token, augmented with its local appearance feature, attends to the temporal sequence and predicts a zero-initialized 3D residual. A per-joint gate, derived from heatmap location and confidence statistics, bounds the residual weight. AG-EgoPose improves over the strongest evaluated baselines by 10.1% PA-MPJPE on EgoPW, and by 9.0% MPJPE and 6.3% PA-MPJPE on SceneEgo after fine-tuning. Code will be made publicly available at https://github.com/
Read the full abstract ▾
Computer Vision
Knowledge-Guided Adversarial Training for Infrared Object Detection via Thermal Radiation Modeling
Shiji Zhao, Shukun Xiong, Maoxun Yuan et al. (+6 more)
In complex environments, infrared object detection exhibits broad applicability and stability across diverse scenarios. However, infrared object detection is vulnerable to both common corruptions and
adversarial examples, leading to potential security risks. To improve the robustness of infrared object detection, current methods mostly adopt a data-driven ideology, which only superficially drives the network to fit the training data without specifically considering the unique characteristics of infrared images, resulting in limited robustness. In this paper, we revisit infrared physical knowledge and find that relative thermal radiation relations between different classes can be regarded as a reliable knowledge source under the complex scenarios of
adversarial examples and common corruptions. Thus, we theoretically model thermal radiation relations based on the rank order of gray values for different classes, and further quantify the stability of various inter-class thermal radiation relations. Based on the above theoretical framework, we propose Knowledge-Guided
Adversarial Training (KGAT) for infrared object detection, in which infrared physical knowledge is embedded into the
adversarial training process, and the predicted results are optimized to be consistent with the actual physical laws. Extensive experiments on three infrared
datasets and six mainstream infrared object detection models demonstrate that KGAT effectively enhances both clean accuracy and robustness against
adversarial attacks and common corruptions.
Read the full abstract ▾
All authors Shiji Zhao, Shukun Xiong, Maoxun Yuan, Yao Huang, Ranjie Duan, Qing Guo, Jiansheng Chen, Haibin Duan, Xingxing Wei
Computer Vision
ET-SAM: Efficient Point Prompt Prediction in SAM for Unified Scene Text Detection and Layout Analysis
Xike Zhang, Maoyuan Ye, Juhua Liu et al. (+1 more)
Previous works based on Segment Anything Model (SAM) have achieved promising performance in unified scene text detection and layout analysis. However, the typical reliance on pixel-level text
segmentation for sampling thousands of foreground points as
prompts leads to unsatisfied inference latency and limited data utilization. To address above issues, we propose ET-SAM, an Efficient framework with two decoders for unified scene Text detection and layout analysis based on SAM. Technically, we customize a lightweight point decoder that produces word heatmaps for achieving a few foreground points, thereby eliminating excessive point
prompts and accelerating inference. Without the dependence on pixel-level
segmentation, we further design a joint training strategy to leverage existing data with heterogeneous text-level annotations. Specifically, the
datasets with multi-level, word-level only, and line-level only annotations are combined in parallel as a unified training set. For these
datasets, we introduce three corresponding sets of learnable task
prompts in both the point decoder and hierarchical mask decoder to mitigate discrepancies across
datasets.Extensive experiments demonstrate that, compared to the previous SAM-based architecture, ET-SAM achieves about 3$\times$ inference acceleration while obtaining competitive performance on HierText, and improves an average of 11.0% F-score on Total-Text, CTW1500, and ICDAR15.
Read the full abstract ▾
All authors Xike Zhang, Maoyuan Ye, Juhua Liu, Bo Du
Computer Vision
Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds
Bin Yang, Mohamed Abdelsamad, Miao Zhang et al. (+1 more)
Recent advances in self-supervised learning (SSL) for point clouds have substantially improved 3D scene understanding without human annotations. Existing approaches emphasize semantic awareness by enforcing feature consistency across augmented views or by masked scene modeling. However, the resulting representations transfer poorly to instance localization, and often require full finetuning for strong performance. Instance awareness is a fundamental component of 3D perception, thus bridging this gap is crucial for progressing toward true 3D foundation models that support all downstream tasks on 3D data. In this work, we introduce PointINS, an instance-oriented self-supervised framework that enriches point cloud representations through geometry-aware learning. PointINS employs an orthogonal offset branch to jointly learn high-level semantic understanding and
geometric reasoning, yielding instance awareness. We identify two consistent properties essential for robust instance localization and formulate them as complementary regularization strategies, Offset Distribution Regularization (ODR), which aligns predicted offsets with empirically observed
geometric priors, and Spatial Clustering Regularization (SCR), which enforces local coherence by regularizing offsets with pseudo-instance masks. Through extensive experiments across five
datasets, PointINS achieves on average +3.5% mAP improvement for indoor instance
segmentation and +4.1% PQ gain for outdoor panoptic
segmentation, paving the way for scalable 3D foundation models.
Read the full abstract ▾
All authors Bin Yang, Mohamed Abdelsamad, Miao Zhang, Alexandru Paul Condurache
Computer Vision
SportSkills: Physical Skill Learning from Sports Instructional Videos
Kumar Ashutosh, Chi Hsuan Wu and Kristen Grauman
Current large-scale
video datasets focus on general human activity, but lack depth of coverage on fine-grained activities needed to address physical skill learning. We introduce SportSkills, the first large-scale sports
dataset geared towards physical skill learning with in-the-wild
video. SportSkills has more than 360k instructional
videos containing more than 630k visual demonstrations paired with instructional narrations explaining the know-how behind the actions from 55 varied sports. Through a suite of experiments, we show that SportSkills unlocks the ability to understand fine-grained differences between physical actions. Our representation achieves gains of up to 4x with the same model trained on traditional activity-centric
datasets. Crucially, building on SportSkills, we introduce the first large-scale task formulation of mistake-conditioned instructional
video retrieval, bridging representation learning and actionable feedback generation (e.g., "here's my execution of a skill; which
video clip should I watch to improve it?"). Formal evaluations by professional coaches show our
retrieval approach significantly advances the ability of
video models to personalize visual instructions for a user query.
Read the full abstract ▾
Computer Vision
A Semantically Disentangled Unified Model for Multi-category 3D Anomaly Detection
SuYeon Kim, Wongyu Lee and MyeongAh Cho
3D anomaly detection targets the detection and localization of defects in 3D point clouds trained solely on normal data. While a unified model improves scalability by learning across multiple categories, it often suffers from Inter-Category
Entanglement (ICE)-where latent features from different categories overlap, causing the model to adopt incorrect semantic priors during reconstruction and ultimately yielding unreliable anomaly scores. To address this issue, we propose the Semantically Disentangled Unified Model for 3D Anomaly Detection, which reconstructs features conditioned on disentangled semantic representations. Our framework consists of three key components: (i) Coarse-to-Fine Global Tokenization for forming instance-level semantic identity, (ii) Category-Conditioned Contrastive Learning for disentangling category semantics, and (iii) a Geometry-Guided Decoder for semantically consistent reconstruction. Extensive experiments on Real3D-AD and Anomaly-ShapeNet demonstrate that our method achieves state-of-the-art for both unified and category-specific models, improving object-level AUROC by 2.8% and 9.1%, respectively, while enhancing the reliability of unified 3D anomaly detection.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Vision Hopfield Memory Networks for Image Recognition
Jianfeng Wang, Amine M'Charrak, Luk Koska et al. (+6 more)
Recent
vision backbones, such as
Transformer families and state-space models like Mamba, have achieved remarkable progress on image recognition. Despite their empirical success, these architectures remain far from the computational principles of the human
brain, often demanding enormous amounts of training data while offering limited interpretability. We propose the
Vision Hopfield
Memory Network (V-HMN), a
brain-inspired
vision backbone that integrates hierarchical
memory mechanisms across layers with iterative refinement updates. Specifically, V-HMN incorporates local Hopfield modules that provide associative
memory dynamics at the image patch level, global Hopfield modules that function as episodic
memory for contextual modulation, and a predictive-coding-inspired refinement rule for iterative error correction. By organizing these
memory-based modules hierarchically, V-HMN captures both local and global dynamics in a unified framework.
Memory retrieval exposes the relationship between inputs and stored patterns, providing a prototype-based form of interpretability through explicit
memory retrieval, while the reuse of stored patterns improves data efficiency. This
brain-inspired design therefore enhances data efficiency and provides a prototype-based form of interpretability compared to existing self-
attention- or state-space-based approaches. We conducted extensive experiments on public image classification
benchmarks. V-HMN achieves strong performance on small- and medium-scale
benchmarks, and remains competitive with widely adopted backbone architectures on ImageNet despite minimal architectural tuning, while offering improved data efficiency and a prototype-based form of interpretability. These findings highlight the potential of V-HMN as a
memory-centric alternative to standard
vision backbones, thereby bridging
brain-inspired computation with modern machine learning.
Read the full abstract ▾
All authors Jianfeng Wang, Amine M'Charrak, Luk Koska, Xiangtao Wang, Daniel Petriceanu, Ruizhi Wang, Michael Bumbar, Luca Pinchetti, Thomas Lukasiewicz
Computer Vision
Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models
Chengyu Fang, Heng Guo, Zheng Jiang et al. (+3 more)
Multimodal large
language models are promising for clinical visual question answering tasks, but scaling to 3D imaging is hindered by high computational costs. Prior methods often rely on 2D slices or fixed-length
token compression, disrupting volumetric continuity and obscuring subtle findings. We present
Photon, a framework that represents 3D medical volumes with
token sequences of variable length.
Photon introduces instruction-conditioned
token scheduling and surrogate gradient propagation to adaptively reduce
tokens during both training and inference, which lowers computational cost while mitigating the
attention dilution caused by redundant
tokens. It incorporates a custom backpropagation rule with gradient restoration to enable differentiable optimization despite discrete
token drop. To stabilize
token compression and ensure reliable use of visual evidence,
Photon further applies regularization objectives that mitigate
language-only bias and improve reliability. Experiments on diverse medical visual question answering tasks show that
Photon achieves state-of-the-art accuracy while reducing resource usage and accelerating both training and inference.
Read the full abstract ▾
All authors Chengyu Fang, Heng Guo, Zheng Jiang, Chunming He, Xiu Li, Minfeng Xu
Computer Vision
Learning to Rank Caption Chains for Video-Text Alignment
Ansel Blume, Burak Uzkent, Shalini Chaudhuri et al. (+1 more)
Direct preference optimization (DPO) is an effective technique to train
language models to generate preferred over dispreferred responses. However, this binary "winner-takes-all" approach is suboptimal for
vision-
language models whose response quality is highly dependent on visual content. In particular, a response may still be faithful to the visual inputs even if it is less preferable than an alternative. The standard Bradley-Terry DPO formulation lacks this nuance, upweighting winning responses without sufficient regard for whether the "losing" response still maintains high visual fidelity. In this work, we investigate ranking optimization as an alternative that more precisely situates responses' faithfulness to visual inputs. We focus on
video-text
alignment using detailed
video captions, proposing a method to generate challenging, totally ordered caption chains at scale through repeated caption degradation. Our results show ranking optimization outperforms binary DPO for long-form content generation and assessment, and importantly, we find that these approaches require finetuning of the
vision encoder to be effective, challenging the view of DPO as purely a
language-reweighting process.
Read the full abstract ▾
All authors Ansel Blume, Burak Uzkent, Shalini Chaudhuri, Garin Kessler
Computer Vision
FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation
Hongxu Ma, Guang Li, Shijie Wang et al. (+5 more)
Dataset distillation (DD) compresses a large training set into a small synthetic set, reducing storage and training cost, and has shown strong results on general
benchmarks. Decoupled DD further improves efficiency by splitting the pipeline into pretraining, sample
distillation, and soft-label generation. However, existing decoupled methods largely rely on coarse class-label supervision and optimize samples within each class in a nearly identical manner. On fine-grained
datasets, this often yields distilled samples that (i) retain large intra-class variation with subtle inter-class differences and (ii) become overly similar within the same class, limiting localized discriminative cues and hurting recognition. To solve the above-mentioned problems, we propose FD$^{2}$, a dedicated framework for Fine-grained
Dataset Distillation. FD$^{2}$ localizes discriminative regions and constructs fine-grained representations for
distillation. During pretraining, counterfactual
attention learning aggregates discriminative representations to update class prototypes. During
distillation, a fine-grained characteristic constraint aligns each sample with its class prototype while repelling others, and a similarity constraint diversifies
attention across same-class samples. Experiments on multiple fine-grained and general
datasets show that FD$^{2}$ integrates seamlessly with decoupled DD and improves performance in most settings, indicating strong transferability. Code is available at https://github.com/Guang000/FD2.
Read the full abstract ▾
All authors Hongxu Ma, Guang Li, Shijie Wang, Dongzhan Zhou, Baoli Sun, Takahiro Ogawa, Miki Haseyama, Zhihui Wang
Computer Vision
SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment
Sahibzada Adil Shahzad, Ammarah Hashmi, Junichi Yamagishi et al. (+5 more)
Multimodal deepfakes can exhibit subtle visual artifacts and cross-modal inconsistencies, which remain challenging to detect, especially when detectors are trained primarily on curated synthetic forgeries. Such synthetic dependence can introduce
dataset and generator bias, limiting scalability and robustness to unseen manipulations. We propose SAVe, a self-supervised audio-visual deepfake detection framework that learns entirely on authentic
videos. SAVe generates on-the-fly, identity-preserving, region-aware self-blended pseudo-manipulations to emulate tampering artifacts, enabling the model to learn complementary visual cues across multiple facial granularities. To capture cross-modal evidence, SAVe also models lip-
speech synchronization via an audio-visual
alignment component that detects temporal misalignment patterns characteristic of audio-visual forgeries. Experiments on FakeAVCeleb and AV-LipSync-TIMIT demonstrate competitive in-domain performance and strong cross-
dataset generalization, highlighting self-supervised learning as a scalable paradigm for
multimodal deepfake detection.
Read the full abstract ▾
All authors Sahibzada Adil Shahzad, Ammarah Hashmi, Junichi Yamagishi, Yusuke Yasuda, Yu Tsao, Chia-Wen Lin, Yan-Tsung Peng, Hsin-Min Wang
Computer Vision
EgoXtreme: A Dataset for Robust Object Pose Estimation in Egocentric Views under Extreme Conditions
Taegyoon Yoon, Yegyu Han, Seojin Ji et al. (+4 more)
Smart glass is emerging as an useful device since it provides plenty of insights under hands-busy, eyes-on-task situations. To understand the context of the wearer, 6D object pose estimation in
egocentric view is becoming essential. However, existing 6D object pose estimation
benchmarks fail to capture the challenges of real-world
egocentric applications, which are often dominated by severe motion blur, dynamic illumination, and visual obstructions. This discrepancy creates a significant gap between controlled lab data and chaotic real-world application. To bridge this gap, we introduce EgoXtreme, a new large-scale 6D pose estimation
dataset captured entirely from an
egocentric perspective. EgoXtreme features three challenging scenarios - industrial maintenance, sports, and emergency rescue - designed to introduce severe perceptual ambiguities through extreme lighting, heavy motion blur, and smoke. Evaluations of state-of-the-art generalizable pose
estimators on EgoXtreme indicate that their generalization fails to hold in extreme conditions, especially under low light. We further demonstrate that simply applying image restoration (e.g., deblurring) offers no positive improvement for extreme conditions. While performance gain has appeared in tracking-based approach, implying using temporal information in fast-motion scenarios is meaningful. We conclude that EgoXtreme is an essential resource for developing and evaluating the next generation of pose estimation models robust enough for real-world
egocentric vision. The
dataset and code are available at https://taegyoun88.github.io/EgoXtreme/
Read the full abstract ▾
All authors Taegyoon Yoon, Yegyu Han, Seojin Ji, Jaewoo Park, Sojeong Kim, Taein Kwon, Hyung-Sin Kim
Computer Vision
Robust Principal Component Completion
Yinjian Wang, Wei Li, Yuanyuan Gui et al. (+2 more)
Robust principal component analysis (RPCA) seeks a low-rank component and a sparse component from their summation. Yet, in many applications of interest, the sparse foreground actually replaces, or occludes, elements from the low-rank background. To address this mismatch, a new framework is proposed in which the sparse component is identified indirectly through determining its support. This approach, called robust principal component completion (RPCC), is solved via variational
Bayesian inference applied to a fully probabilistic
Bayesian sparse tensor factorization. Convergence to a hard classifier for the support is shown, thereby eliminating the post-hoc thresholding required of most prior RPCA-driven approaches. Experimental results reveal that the proposed approach delivers near-optimal estimates on synthetic data as well as robust foreground-extraction and anomaly-detection performance on real color
video and hyperspectral
datasets, respectively. Source implementation and Appendices are available at https://github.com/WongYinJ/BCP-RPCC.
Read the full abstract ▾
All authors Yinjian Wang, Wei Li, Yuanyuan Gui, James E. Fowler, Gemine Vivone
Computer Vision
Denoise and Align: Towards Source-Free UDA for Robust Panoramic Semantic Segmentation
Yaowen Chang, Zhen Cao, Xu Zheng et al. (+2 more)
Panoramic semantic
segmentation is pivotal for comprehensive 360{\deg} scene understanding in critical applications like autonomous driving and virtual reality. However, progress in this domain is constrained by two key challenges: the severe
geometric distortions inherent in panoramic projections and the prohibitive cost of dense annotation. While Unsupervised Domain Adaptation (UDA) from label-rich pinhole-camera
datasets offers a viable alternative, many real-world tasks impose a stricter source-free (SFUDA) constraint where source data is inaccessible for
privacy or proprietary reasons. This constraint significantly amplifies the core problems of domain shift, leading to unreliable pseudo-labels and dramatic performance degradation, particularly for minority classes. To overcome these limitations, we propose the DAPASS framework. DAPASS introduces two synergistic modules to robustly transfer knowledge without source data. First, our Panoramic Confidence-Guided Denoising (PCGD) module generates high-fidelity, class-balanced pseudo-labels by enforcing perturbation consistency and incorporating neighborhood-level confidence to filter noise. Second, a Contextual Resolution
Adversarial Module (CRAM) explicitly addresses scale variance and distortion by adversarially aligning fine-grained details from high-resolution crops with global semantics from low-resolution contexts. DAPASS achieves state-of-the-art performances on outdoor (Cityscapes-to-DensePASS) and indoor (Stanford2D3D)
benchmarks, yielding 55.04% (+2.05%) and 70.38% (+1.54%) mIoU, respectively.
Read the full abstract ▾
All authors Yaowen Chang, Zhen Cao, Xu Zheng, Xiaoxin Mi, Zhen Dong
Computer Vision
AirSplat: Alignment and Rating for Robust Feed-Forward 3D Gaussian Splatting
Minh-Quan Viet Bui, Jaeho Moon and Munchurl Kim
While 3D
Vision Foundation Models (3DVFMs) have demonstrated remarkable zero-shot capabilities in visual geometry estimation, their direct application to generalizable novel view synthesis (NVS) remains challenging. In this paper, we propose AirSplat, a novel training framework that effectively adapts the robust
geometric priors of 3DVFMs into high-fidelity, pose-free NVS. Our approach introduces two key technical contributions: (1) Self-Consistent Pose
Alignment (SCPA), a training-time feedback loop that ensures pixel-aligned supervision to resolve pose-geometry discrepancy; and (2) Rating-based Opacity Matching (ROM), which leverages the local 3D geometry consistency knowledge from a sparse-view NVS teacher model to filter out degraded primitives. Experimental results on large-scale
benchmarks demonstrate that our method significantly outperforms state-of-the-art pose-free NVS approaches in reconstruction quality. Our AirSplat highlights the potential of adapting 3DVFMs to enable simultaneous visual geometry estimation and high-quality view synthesis.
Read the full abstract ▾
Computer Vision
AnyDoc: Enhancing Document Generation via Large-Scale HTML/CSS Data Synthesis and Height-Aware Reinforcement Optimization
Jiawei Lin, Wanrong Zhu, Vlad I Morariu et al. (+1 more)
Document generation has gained growing
attention in the field of AI-driven content creation. In this work, we push its boundaries by introducing AnyDoc, a framework capable of handling multiple generation tasks across a wide spectrum of document categories, all represented in a unified HTML/CSS format. To overcome the limited coverage and scale of existing human-crafted document
datasets, AnyDoc first establishes a scalable data synthesis pipeline to automatically generate documents in HTML/CSS form. This pipeline yields DocHTML, a large-scale
dataset containing 265,206 document samples, while spanning 111 categories and 32 distinct styles. Additionally, all documents are equipped with comprehensive metadata, including design intentions, HTML/CSS source code, visual assets, and rendered screenshots. Building on the curated
dataset, AnyDoc fine-tunes multi-modal large
language models (MLLMs) to achieve three practical document generation tasks: intention-to-document, document derendering, and element-to-document. To address the content overflow issue observed during fine-tuning, AnyDoc further incorporates a height-aware reinforcement learning (HARL) post-training procedure. By defining a reward function based on the difference between predicted and target document heights, overflow is penalized and gradually mitigated during HARL, thereby enhancing overall performance. Qualitative and quantitative experiments demonstrate that AnyDoc outperforms both general-purpose MLLMs and task-specific baselines across all three tasks.
Read the full abstract ▾
All authors Jiawei Lin, Wanrong Zhu, Vlad I Morariu, Christopher Tensmeyer
Computer Vision
MoireMix: A Formula-Based Data Augmentation for Improving Image Classification Robustness
Yuto Matsuo, Yoshihiro Fukuhara, Yuki M. Asano et al. (+3 more)
Data augmentation is a key technique for improving the robustness of image classification models. However, many recent approaches rely on
diffusion-based synthesis or complex feature mixing strategies, which introduce substantial computational overhead or require external
datasets. In this work, we explore a different direction: procedural augmentation based on analytic interference patterns. Unlike conventional augmentation methods that rely on
stochastic noise, feature mixing, or generative models, our approach exploits Moire interference to generate structured perturbations spanning a wide range of spatial frequencies. We propose a lightweight augmentation method that procedurally generates Moire textures on-the-fly using a closed-form mathematical formulation. The patterns are synthesized directly in
memory with negligible computational cost (0.0026 seconds per image), mixed with training images during training, and immediately discarded, enabling a storage-free augmentation pipeline without external data. Extensive experiments with
Vision Transformers demonstrate that the proposed method consistently improves robustness across multiple
benchmarks, including ImageNet-C, ImageNet-R, and
adversarial benchmarks, outperforming standard augmentation baselines and existing external-data-free augmentation approaches. These results suggest that analytic interference patterns provide a practical and efficient alternative to data-driven generative augmentation methods.
Read the full abstract ▾
All authors Yuto Matsuo, Yoshihiro Fukuhara, Yuki M. Asano, Rintaro Yanagi, Hirokatsu Kataoka, Akio Nakamura
Computer Vision
MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning
Chenglong Wang, Yifu Huo, Yang Gan et al. (+8 more)
Recent advances in
multimodal reward modeling have been largely driven by a paradigm shift from discriminative to generative approaches. Building on this progress, recent studies have further employed reinforcement learning from verifiable rewards (RLVR) to enhance
multimodal reward models (MRMs). Despite their success, RLVR-based training typically relies on labeled
multimodal preference data, which are costly and labor-intensive to obtain, making it difficult to scale MRM training. To overcome this limitation, we propose a Multi-Stage Reinforcement Learning (MSRL) approach, which can achieve scalable RL for MRMs with limited
multimodal data. MSRL replaces the conventional RLVR-based training paradigm by first learning a generalizable reward
reasoning capability from large-scale textual preference data, and then progressively transferring this capability to
multimodal tasks through caption-based and fully
multimodal reinforcement-learning stages. Furthermore, we introduce a cross-modal knowledge
distillation approach to improve preference generalization within MSRL. Extensive experiments demonstrate that MSRL effectively scales the RLVR-based training of generative MRMs and substantially improves their performance across both visual understanding and visual generation tasks (e.g., from 66.6% to 75.9% on VL-RewardBench and from 70.2% to 75.7% on GenAI-Bench), without requiring additional
multimodal preference annotations. Our code is available at: https://github.com/wangclnlp/MSRL.
Read the full abstract ▾
All authors Chenglong Wang, Yifu Huo, Yang Gan, Qiaozhi He, Qi Meng, Bei Li, Yan Wang, Junfu Liu, Tianhua Zhou, Jingbo Zhu, Tong Xiao
Computer Vision
Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning
Yuqiao Zeng, Xu Wang, Tengfei Liang et al. (+3 more)
Multimodal learning integrates complementary information from different modalities such as image, text, and audio to improve model performance, but its success relies on large-scale labeled data, which is costly to obtain. Active learning (AL) mitigates this challenge by selectively annotating informative samples. In
multimodal settings, many approaches implicitly assume that modality importance is stable across rounds and keep selection rules fixed at the fusion stage, which leaves them insensitive to the dynamic nature of
multimodal learning, where the relative value of modalities and the difficulty of instances shift as training proceeds. To address this issue, we propose RL-MBA, a reinforcement-learning framework for modality-balanced, difficulty-aware
multimodal active learning. RL-MBA models sample selection as a Markov Decision Process, where the policy adapts to modality contributions, uncertainty, and diversity, and the reward encourages accuracy gains and balance. Two key components drive this adaptability: (1) Adaptive Modality Contribution Balancing (AMCB), which dynamically adjusts modality weights via reinforcement feedback, and (2) Evidential Fusion for DifficultyAware Policy Adjustment (EFDA), which estimates sample difficulty via uncertainty-based evidential fusion to prioritize informative samples. Experiments on Food101, KineticsSound, and VGGSound demonstrate that RL-MBA consistently outperforms strong baselines, improving both classification accuracy and modality
fairness under limited labeling budgets.
Read the full abstract ▾
All authors Yuqiao Zeng, Xu Wang, Tengfei Liang, Yiqing Hao, Yi Jin, Hui Yu