Computer Vision
Automating construction safety inspections using a multi-modal vision-language RAG framework
Chenxin Wang, Elyas Asadi Shamsabadi, Zhaohui Chen et al. (+3 more)
Conventional construction safety inspection methods are often inefficient as they require navigating through large volume of information. Recent advances in large
vision-
language models (LVLMs) provide opportunities to automate safety inspections through enhanced visual and linguistic understanding. However, existing applications face limitations including irrelevant or unspecific responses, restricted modal inputs and
hallucinations. Utilisation of Large
Language Models (LLMs) for this purpose is constrained by availability of training data and frequently lack real-time adaptability. This study introduces SiteShield, a multi-modal LVLM-based
Retrieval-Augmented Generation (RAG) framework for automating construction safety inspection reports by integrating visual and audio inputs. Using real-world data, SiteShield outperformed unimodal LLMs without RAG with an F1 score of 0.82, hamming loss of 0.04, precision of 0.76, and recall of 0.96. The findings indicate that SiteShield offers a novel pathway to enhance information
retrieval and efficiency in generating safety reports.
Read the full abstract ▾
All authors Chenxin Wang, Elyas Asadi Shamsabadi, Zhaohui Chen, Luming Shen, Alireza Ahmadian Fard Fini, Daniel Dias-da-Costa
Computer Vision
Turning Drift into Constraint: Robust Reasoning Alignment in Non-Stationary Multi-Stream Environments
Xiaoyu Yang, En Yu, Wei Duan et al. (+1 more)
This paper identifies a critical yet underexplored challenge in
reasoning alignment from multiple multi-modal large
language models (MLLMs): In non-stationary environments, the diverse
reasoning distributions of source models often evolve unpredictably, transmitting systematic biases and drift to the target model. To address this, we formulate multi-source
reasoning alignment as a constraint satisfaction problem under concept drift theory. We propose Autonomous Preference Optimization (APO), a novel framework that treats inter-model divergences not as noise, but as dynamic negative constraints. APO operates via a two-stage protocol: first, supervised bootstrapping projects the target model into the capability union of source models; second, constraint-aware optimization synthesizes a consistent consensus
manifold by explicitly suppressing drifting trajectories via a multi-negative Plackett-Luce objective. Extensive experiments on chest X-ray interpretation demonstrate that our 7B model achieves superior robustness, outperforming even proprietary source models in average accuracy. Furthermore, we release CXR-MAX, a large-scale
benchmark comprising 170,982
reasoning trajectories from seven large-scale MLLMs to facilitate research on
reasoning alignment under drift. Code and data are available at: https://github.com/XiaoyuYoung/APO.
Read the full abstract ▾
All authors Xiaoyu Yang, En Yu, Wei Duan, Jie Lu
Computer Vision · Cross-listed from eess.AS
MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
Umberto Cappellazzo, Minsu Kim, Pingchuan Ma et al. (+4 more)
Large
language models (LLMs) have recently shown strong potential in audio-visual
speech recognition (AVSR), but their high computational demands and sensitivity to
token granularity limit their practicality in resource-constrained settings.
Token compression methods can reduce inference cost, but they require fixing a compression rate in advance and produce a single fixed-length output, offering no flexibility to balance information density and efficiency at inference time. Matryoshka representation learning (MRL) addresses this by enabling a single model to operate across multiple
token granularities, allowing compression rates to be adjusted dynamically. However, current MRL-based methods treat each scale independently during training, limiting cross-scale generalization, robustness at high compression, and interpretability. To overcome these limitations, we propose MoME (Mixture of Matryoshka Experts), a novel framework that integrates sparse Mixture-of-Experts (MoE) into MRL-based LLMs for AVSR. MoME augments a frozen LLM with top-k routed and shared experts, allowing dynamic capacity allocation across scales and modalities. A shared router promotes consistent expert activation across granularities, enabling compressed sequences to benefit from representations learned at lower compression. Experiments on LRS2 and LRS3 demonstrate that MoME achieves state-of-the-art performance across AVSR, ASR, and VSR tasks, while requiring significantly fewer parameters and maintaining robustness under noise. MoME unifies the adaptability of MRL with the efficiency of MoE, offering a scalable and interpretable solution for resource-aware
speech recognition.
Read the full abstract ▾
All authors Umberto Cappellazzo, Minsu Kim, Pingchuan Ma, Honglie Chen, Xubo Liu, Stavros Petridis, Maja Pantic
Computer Vision · Cross-listed from cs.IR
Projection and Quantisation: A Unifying View of Learning to Hash, from Random Projections to the RAG Era
Sean Moran
Approximate nearest-neighbour search underpins large-scale
retrieval and
retrieval-augmented generation, yet its methods are studied in communities that seldom read one another. We argue that they form one field with three design choices. We develop the projection-quantisation-organisation lens: every method places its projections, places its quantisation thresholds, and organises the resulting codes for search. We test the lens with a reproducible measurement, released as the open BitBudget
benchmark, and report three findings. First, the quantisation axis delivers the largest
memory savings: a one-bit code with full-precision re-ranking matches uncompressed quality for six of seven embedders, the scanned code one thirty-second of the float's size. Second, the orderings the lens anticipates, including a learned-
embedding regime where binary codes overtake an inverted-file product quantiser at a matched byte budget, recur as the
embedding is enlarged. Third, given class labels, an eight-byte supervised code more than doubles the
retrieval quality of the two-kilobyte task-agnostic float it replaces. We also recast the semantic identifiers of generative
retrieval as quantisation codes. The main contribution is a single, tested account of compact-code search, from random projections to the
retrieval-augmented era.
Read the full abstract ▾
Computer Vision
Joint Learning of Pose Regression and Denoising Diffusion with Score Scaling Sampling for Category-level 6D Pose Estimation
Seunghyun Lee and Tae-Kyun Kim
Latest
diffusion models have shown promising results in category-level 6D object pose estimation by modeling the conditional pose distribution with depth image input. The existing methods, however, suffer from slow convergence during training, learning its encoder with the
diffusion denoising network in end-to-end fashion, and require an additional network that evaluates sampled pose hypotheses to filter out low-quality pose candidates. In this paper, we propose a novel pipeline that tackles these limitations by two key components. First, the proposed method pretrains the encoder with the direct pose regression head, and jointly learns the networks via the regression head and the denoising
diffusion head, significantly accelerating training convergence while achieving higher accuracy. Second, sampling guidance via time-dependent score scaling is proposed s.t. the exploration-exploitation trade-off is effectively taken, eliminating the need for the additional evaluation network. The sampling guidance maintains multi-modal characteristics of symmetric objects at early denoising steps while ensuring high-quality pose generation at final steps. Extensive experiments on multiple
benchmarks including REAL275, HouseCat6D, and ROPE, demonstrate that the proposed method, simple yet effective, achieves state-of-the-art accuracies even with single-pose inference, while being more efficient in both training and inference.
Read the full abstract ▾
Computer Vision
Learning Efficient Meshflow and Optical Flow from Event Cameras
Xinglong Luo, Ao Luo, Kunming Luo et al. (+4 more)
In this paper, we explore the problem of event-based meshflow estimation, a novel task that involves predicting a spatially smooth sparse motion field from event cameras. To start, we review the state-of-the-art in event-based flow estimation, highlighting two key areas for further research: i) the lack of meshflow-specific event
datasets and methods, and ii) the underexplored challenge of event data density. First, we generate a large-scale High-Resolution Event Meshflow (HREM)
dataset, which showcases its superiority by encompassing the merits of high resolution at 1280x720, handling dynamic objects and complex motion patterns, and offering both optical flow and meshflow labels. These aspects have not been fully explored in previous works. Besides, we propose Efficient Event-based MeshFlow (EEMFlow) network, a lightweight model featuring a specially crafted encoder-decoder architecture to facilitate swift and accurate meshflow estimation. Furthermore, we upgrade EEMFlow network to support dense event optical flow, in which a Confidence-induced Detail Completion (CDC) module is proposed to preserve sharp motion boundaries. We conduct comprehensive experiments to show the exceptional performance and runtime efficiency (30x faster) of our EEMFlow model compared to the recent state-of-the-art flow method. As an extension, we expand HREM into HREM+, a multi-density event
dataset contributing to a thorough study of the robustness of existing methods across data with varying densities, and propose an Adaptive Density Module (ADM) to adjust the density of input event data to a more optimal range, enhancing the model's generalization ability. We empirically demonstrate that ADM helps to significantly improve the performance of EEMFlow and EEMFlow+ by 8% and 10%, respectively. Code and
dataset are released at https://github.com/boomluo02/EEMFlowPlus.
Read the full abstract ▾
All authors Xinglong Luo, Ao Luo, Kunming Luo, Zhengning Wang, Ping Tan, Bing Zeng, Shuaicheng Liu
Computer Vision
TOPO-Bench: An Open-Source Topological Mapping Evaluation Framework with Quantifiable Perceptual Aliasing
Jiaming Wang, Jizhuo Chen, Diwen Liu et al. (+5 more)
Topological mapping offers a compact and robust representation for
navigation, but progress in the field is hindered by the lack of standardized evaluation metrics,
datasets, and protocols. Existing systems are assessed using different environments and criteria, preventing fair and reproducible comparisons. Moreover, a key challenge - perceptual aliasing - remains under-quantified, despite its strong influence on system performance. We address these gaps by (1) formalizing topological consistency as the fundamental property of topological maps and showing that localization accuracy provides an efficient and interpretable surrogate metric, and (2) proposing the first quantitative measure of
dataset ambiguity to enable fair comparisons across environments. To support this protocol, we curate a diverse
benchmark dataset with calibrated ambiguity levels, implement and release deep-learned baseline systems, and evaluate them alongside classical methods. Our experiments and analysis yield new insights into the limitations of current approaches under perceptual aliasing. All
datasets, baselines, and evaluation tools are fully open-sourced to foster consistent and reproducible research in topological mapping.
Read the full abstract ▾
All authors Jiaming Wang, Jizhuo Chen, Diwen Liu, Jiaxuan Da, Jiamo Hu, Zhiwei Xue, Linh Kästner, Harold Soh
Computer Vision · Cross-listed from cs.LG
Using predefined vector systems as latent space configuration for neural network supervised training on data with arbitrarily large number of classes
Nikita Gabdullin
Supervised learning (SL) methods are indispensable for neural network (NN) training used to perform classification tasks. While resulting in very high accuracy, SL training often requires making NN parameter number dependent on the number of classes, limiting their applicability when the number of classes is extremely large or unknown in advance. In this paper we propose a methodology that allows one to train the same NN architecture regardless of the number of classes. This is achieved by using predefined vector systems as the target latent space configuration (LSC) during NN training. We discuss the desired properties of target configurations and choose randomly perturbed vectors of An root system for our experiments. These vectors are used to successfully train encoders and visual
transformers (ViT) on Cinic-10 and ImageNet-1K in low- and high-dimensional cases by matching NN predictions with the predefined vectors. Finally, ViT is trained on a
dataset with 1.28 million classes illustrating the applicability of the method to training on
datasets with extremely large number of classes. In addition, potential applications of LSC in lifelong learning and NN
distillation are discussed illustrating versatility of the proposed methodology.
Read the full abstract ▾
Computer Vision
Diffusion Low Rank Hybrid Reconstruction for Sparse View Medical Imaging
Zongyin Deng, Qing Zhou, Yuhao Fang et al. (+4 more)
This work presents TV-LoRA, a novel method for low-dose sparse-view CT reconstruction that combines a
diffusion generative prior (NCSN++ with SDE modeling) and multi-regularization constraints, including anisotropic TV and
nuclear norm (LoRA), within an ADMM framework. To address ill-posedness and texture loss under extremely sparse views, TV-LoRA integrates generative and physical constraints, and utilizes a 2D slice-based strategy with FFT acceleration and tensor-parallel optimization for efficient inference. Experiments on AAPM-2016, CTHD, and LIDC
datasets with $N_{\mathrm{view}}=8,4,2$ show that TV-LoRA consistently surpasses
benchmarks in SSIM, texture recovery, edge clarity, and artifact suppression, demonstrating strong robustness and generalizability. Ablation studies confirm the complementary effects of LoRA regularization and
diffusion priors, while the FFT-PCG module provides a speedup. Overall,
Diffusion + TV-LoRA achieves high-fidelity, efficient 3D CT reconstruction and broad clinical applicability in low-dose, sparse-sampling scenarios.
Read the full abstract ▾
All authors Zongyin Deng, Qing Zhou, Yuhao Fang, Zijian Wang, Yao Lu, Ye Zhang, Chun Li
Computer Vision
QuantDemoire: Quantization with Outlier Aware for Image Demoiréing
Zheng Chen, Kewei Zhang, Xiaoyang Liu et al. (+4 more)
Demoiréing aims to remove moiré artifacts that often occur in images. While recent deep learning-based methods have achieved promising results, they typically require substantial computational resources, limiting their deployment on edge devices. Model quantization offers a compelling solution. However, directly applying existing quantization methods to demoiréing models introduces severe performance degradation. The main reasons are distribution outliers and weakened representations in smooth regions. To address these issues, we propose QuantDemoire, a post-training quantization framework tailored to demoiréing. It contains two key components. First, we introduce an outlier-aware quantizer to reduce errors from outliers. It uses sampling-based range estimation to reduce activation outliers and keeps a few extreme weights in FP16 with negligible cost. Second, we design a frequency-aware calibration strategy. It emphasizes low- and mid-frequency components during fine-tuning, which mitigates banding artifacts caused by low-bit quantization. Experiments validate that our QuantDemoire achieves large reductions in parameters and computation while maintaining quality. Meanwhile, it outperforms existing quantization methods by over 4 dB on W4A4. We implement a CUDA kernel for the mixed-precision branch and achieve a 2.3$\times$ end-to-end speedup on an RTX 4090 GPU. Code is released at: https://github.com/zhengchen1999/QuantDemoire.
Read the full abstract ▾
All authors Zheng Chen, Kewei Zhang, Xiaoyang Liu, Weihang Zhang, Mengfan Wang, Yifan Fu, Yulun Zhang
Computer Vision
Ordinal Encoding as a Regularizer in Binary Loss for Solar Flare Prediction
Chetraj Pandey, Jinsu Hong, Anli Ji et al. (+2 more)
The prediction of
solar flares is typically formulated as a binary classification task, distinguishing events as either
Flare (FL) or No-
Flare (NF) according to a specified threshold (for example, greater than or equal to C-class, M-class, or X-class). However, this binary framework neglects the inherent ordinal relationships among the sub-classes contained within each category (FL and NF). Several studies on
solar flare prediction have empirically shown that the most frequent misclassifications occur near this prediction threshold. This suggests that the models struggle to differentiate events that are similar in intensity but fall on opposite sides of the binary threshold. To mitigate this limitation, we propose a modified loss function that integrates the ordinal information among the sub-classes of the binarized
flare labels into the conventional binary cross-
entropy (BCE) loss. This approach serves as an ordinality-aware, data-driven regularization method that penalizes the incorrect predictions of
flare events in close proximity to the prediction threshold more heavily than those away from the boundary during model optimization. By incorporating ordinal weighting into the loss function, we aim to enhance the model's learning process by leveraging the ordinal characteristics of the data, thereby improving its overall performance.
Read the full abstract ▾
All authors Chetraj Pandey, Jinsu Hong, Anli Ji, Rafal A. Angryk, Berkay Aydin
Computer Vision
MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generation
Zhenyu Pan, Yucheng Lu and Han Liu
We present MetaFind, a scene-aware tri-modal compositional
retrieval framework designed to enhance scene generation in the metaverse by retrieving 3D assets from large-scale repositories. MetaFind addresses two core challenges: (i) inconsistent asset
retrieval that overlooks spatial, semantic, and stylistic constraints, and (ii) the absence of a standardized
retrieval paradigm specifically tailored for 3D asset
retrieval, as existing approaches mainly rely on general-purpose 3D shape representation models. Our key innovation is a flexible
retrieval mechanism that supports arbitrary combinations of text, image, and 3D modalities as queries, enhancing spatial
reasoning and style consistency by jointly modeling object-level features (including appearance) and scene-level layout structures. Methodologically, MetaFind introduces a plug-and-play equivariant layout encoder ESSGNN that captures spatial relationships and object appearance features, ensuring retrieved 3D assets are contextually and stylistically coherent with the existing scene, regardless of coordinate frame transformations. The framework supports iterative scene construction by continuously adapting
retrieval results to current scene updates. Empirical evaluations demonstrate the improved spatial and stylistic consistency of MetaFind in various
retrieval tasks compared to baseline methods.
Read the full abstract ▾
Computer Vision
Quantization Range Estimation for Convolutional Neural Networks
Bingtao Yang, Yujia Wang, Mengzhi Jiao et al. (+1 more)
Post-training quantization for reducing the storage of deep neural network models has been demonstrated to be an effective way in various tasks. However, low-bit quantization while maintaining model accuracy is a challenging problem. In this paper, we present a range estimation method to improve the quantization performance for post-training quantization. We model the range estimation into an optimization problem of minimizing quantization errors by layer-wise local minima. We prove this problem is locally convex and present an efficient search
algorithm to find the optimal solution. We propose the application of the above search
algorithm to the transformed weights space to do further improvement in practice. Our experiments demonstrate that our method outperforms state-of-the-art performance generally on top-1 accuracy for image classification tasks on the ResNet series models and Inception-v3 model. The experimental results show that the proposed method has almost no loss of top-1 accuracy in 8-bit and 6-bit settings for image classifications, and the accuracy of 4-bit quantization is also significantly improved. The code is available at https://github.com/codeiscommitting/REQuant.
Read the full abstract ▾
All authors Bingtao Yang, Yujia Wang, Mengzhi Jiao, Hongwei Huo
Computer Vision
GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding
Bin Lei, Nuo Xu, Ali Payani et al. (+4 more)
Multimodal large
language models (MLLMs) have markedly expanded the competence of graphical user-interface (GUI) systems, propelling them beyond controlled simulations into complex, real-world environments across diverse platforms. However, practical usefulness is still bounded by the reliability of visual grounding, i.e., mapping textual references to exact on-screen elements. This limitation prevents the system from accurately performing pointer-level actions such as clicking or dragging. To address it, we introduce GUI-Spotlight – a model trained for image-grounded
reasoning that dynamically invokes multiple specialized tools to iteratively narrow its focus to the relevant region of the screen, thereby substantially improving visual grounding accuracy. On the ScreenSpot-Pro
benchmark, GUI-Spotlight trained with only 18.5K training samples achieves 52.8% accuracy, surpassing V2P-7B (50.6% with 9.6M training samples) and GTA-1-7B (50.1% with 1.56M training samples).
Read the full abstract ▾
All authors Bin Lei, Nuo Xu, Ali Payani, Mingyi Hong, Chunhua Liao, Yu Cao, Caiwen Ding
Computer Vision
Prompt-to-Prompt: Text-Based Image Editing Via Cross-Attention Mechanisms – The Research of Hyperparameters and Novel Mechanisms to Enhance Existing Frameworks
Linn Bieske and Carla Lorente
Recent advances in image editing have shifted from manual pixel manipulation to employing deep learning methods like stable
diffusion models, which now leverage cross-
attention mechanisms for text-driven control. This transition has simplified the editing process but also introduced variability in results, such as inconsistent hair color changes. Our research aims to enhance the precision and reliability of
prompt-to-
prompt image editing frameworks by exploring and optimizing hyperparameters. We present a comprehensive study of the "word swap" method, develop an "
attention re-weight method" for better adaptability, and propose the "CL P2P" framework to address existing limitations like cycle inconsistency. This work contributes to understanding and improving the interaction between hyperparameter settings and the architectural choices of neural network models, specifically their
attention mechanisms, which significantly influence the composition and quality of the generated images.
Read the full abstract ▾
Computer Vision
Enhancing Fake News Video Detection via LLM-Driven Creative Process Simulation
Yuyan Bu, Qiang Sheng, Juan Cao et al. (+4 more)
The emergence of fake news on short
video platforms has become a new significant societal concern, necessitating automatic
video-news-specific detection. Current detectors primarily rely on pattern-based features to separate fake news
videos from real ones. However, limited and less diversified training data lead to biased patterns and hinder their performance. This weakness stems from the complex many-to-many relationships between
video material segments and fabricated news events in real-world scenarios: a single
video clip can be utilized in multiple ways to create different fake narratives, while a single fabricated event often combines multiple distinct
video segments. However, existing
datasets do not adequately reflect such relationships due to the difficulty of collecting and annotating large-scale real-world data, resulting in sparse coverage and non-comprehensive learning of the characteristics of potential fake news
video creation. To address this issue, we propose a data augmentation framework, AgentAug, that generates diverse fake news
videos by simulating typical creative processes. AgentAug implements multiple LLM-driven pipelines of four fabrication categories for news
video creation, combined with an active learning strategy based on uncertainty sampling to select the potentially useful augmented samples during training. Experimental results on two
benchmark datasets demonstrate that AgentAug consistently improves the performance of short
video fake news detectors.
Read the full abstract ▾
All authors Yuyan Bu, Qiang Sheng, Juan Cao, Shaofei Wang, Peng Qi, Yuhui Shi, Beizhe Hu
Computer Vision
Video-in-the-Loop: Span-Grounded Long Video QA with Interleaved Reasoning
Chendong Wang, Donglin Bai, Yifan Yang et al. (+11 more)
We present
Video-in-the-Loop (ViTL), a two-stage long-
video QA framework that preserves a fixed
token budget by first localizing question-relevant interval(s) with a low-fps skim and then answering via span-aware reallocation of visual
tokens at higher effective frame rate, emitting an interleaved output with both spans and the final option for direct attribution. We also introduce \dataname{}, which converts description based event graphs into span-grounded multiple-choice QA by pairing each question with ground-truth time span(s) and related
reasoning. ViTL is trained end-to-end with an interleaved group-relative objective that couples temporal IoU for localization with answer correctness, allowing credit to flow from answers back to spans without increasing compute. Under fixed
token budgets, ViTL attains up to 8.6% with 50% less frame input on long-
video QA and temporal grounding (e.g., Charades-STA, ActivityNet-Captions) and ablations show that span-aware
token reallocation consistently surpasses uniform sampling. Together, \dataname{} and ViTL provide an interpretable, compute-efficient recipe for scalable long-
video QA.
Read the full abstract ▾
All authors Chendong Wang, Donglin Bai, Yifan Yang, Xiao Jin, Anlan Zhang, Rui Wang, Shiqi Jiang, Yuqing Yang, Hao Wu, Qi Dai, Chong Luo, Ting Cao, Lili Qiu, Suman Banerjee
Computer Vision
Fit Pixels, Get Labels: Meta-learned Implicit Networks for Image Segmentation
Kushal Vyas, Ashok Veeraraghavan and Guha Balakrishnan
Implicit neural representations (INRs) have achieved remarkable successes in learning expressive yet compact signal representations. However, they are not naturally amenable to predictive tasks such as
segmentation, where they must learn semantic structures over a distribution of signals. In this study, we introduce MetaSeg, a meta-learning framework to train INRs for medical image
segmentation. MetaSeg uses an underlying INR that simultaneously predicts per pixel intensity values and class labels. It then uses a meta-learning procedure to find optimal initial parameters for this INR over a training
dataset of images and
segmentation maps, such that the INR can simply be fine-tuned to fit pixels of an unseen test image, and automatically decode its class labels. We evaluated MetaSeg on 2D and 3D
brain MRI
segmentation tasks and report Dice scores comparable to commonly used U-Net models, but with $90\%$ fewer parameters. MetaSeg offers a fresh, scalable alternative to traditional resource-heavy architectures such as U-Nets and
vision transformers for medical image
segmentation. Our project is available at https://kushalvyas.github.io/metaseg.html .
Read the full abstract ▾
Computer Vision · Cross-listed from cs.IR
Visual Lifelog Retrieval through Captioning-Enhanced Interpretation
Yu-Fei Shih, An-Zi Yen, Hen-Hsen Huang et al. (+1 more)
People often struggle to remember specific details of past experiences, which can lead to the need to revisit these
memories. Consequently, lifelog
retrieval has emerged as a crucial application. Various studies have explored methods to facilitate rapid access to personal lifelogs for
memory recall assistance. In this paper, we propose a Captioning-Integrated Visual Lifelog (CIVIL)
Retrieval System for extracting specific images from a user's visual lifelog based on textual queries. Unlike traditional
embedding-based methods, our system first generates captions for visual lifelogs and then utilizes a text
embedding model to project both the captions and user queries into a shared vector space. Visual lifelogs, captured through wearable cameras, provide a first-person viewpoint, necessitating the interpretation of the activities of the individual behind the camera rather than merely describing the scene. To address this, we introduce three distinct approaches: the single caption method, the collective caption method, and the merged caption method, each designed to interpret the life experiences of lifeloggers. Experimental results show that our method effectively describes first-person visual images, enhancing the outcomes of lifelog
retrieval. Furthermore, we construct a textual
dataset that converts visual lifelogs into captions, thereby reconstructing personal life experiences.
Read the full abstract ▾
All authors Yu-Fei Shih, An-Zi Yen, Hen-Hsen Huang, Hsin-Hsi Chen
Computer Vision
Enhancing OCR for Sino-Vietnamese Language Processing via Fine-tuned PaddleOCRv5
Minh Hoang Nguyen and Su Nguyen Thiet
Recognizing and processing Classical Chinese (Han-Nom) texts play a vital role in digitizing Vietnamese historical documents and enabling cross-lingual semantic research. However, existing OCR systems struggle with degraded scans, non-standard glyphs, and handwriting variations common in ancient sources. In this work, we propose a fine-tuning approach for PaddleOCRv5 to improve character recognition on Han-Nom texts. We retrain the text recognition module using a curated subset of ancient Vietnamese Chinese manuscripts, supported by a full training pipeline covering preprocessing, LMDB conversion, evaluation, and visualization. Experimental results show a significant improvement over the base model, with exact accuracy increasing from 37.5 percent to 50.0 percent, particularly under noisy image conditions. Furthermore, we develop an interactive demo that visually compares pre- and post-fine-tuning recognition results, facilitating downstream applications such as Han-Vietnamese semantic
alignment, machine translation, and historical linguistics research. The demo is available at https://huggingface.co/spaces/MinhDS/Fine-tuned-PaddleOCRv5
Read the full abstract ▾
Computer Vision
Keep It on a Leash: Controllable Pseudo-label Generation Towards Realistic Long-Tailed Semi-Supervised Learning
Yaxin Hou, Bo Han, Yuheng Jia et al. (+2 more)
Current long-tailed semi-supervised learning methods assume that labeled data exhibit a long-tailed distribution, and unlabeled data adhere to a typical predefined distribution (i.e., long-tailed, uniform, or inverse long-tailed). However, the distribution of the unlabeled data is generally unknown and may follow an arbitrary distribution. To tackle this challenge, we propose a Controllable Pseudo-label Generation (CPG) framework, expanding the labeled
dataset with the progressively identified reliable pseudo-labels from the unlabeled
dataset and training the model on the updated labeled
dataset with a known distribution, making it unaffected by the unlabeled data distribution. Specifically, CPG operates through a controllable self-reinforcing optimization cycle: (i) at each training step, our dynamic controllable filtering mechanism selectively incorporates reliable pseudo-labels from the unlabeled
dataset into the labeled
dataset, ensuring that the updated labeled
dataset follows a known distribution; (ii) we then construct a Bayes-optimal classifier using logit adjustment based on the updated labeled data distribution; (iii) this improved classifier subsequently helps identify more reliable pseudo-labels in the next training step. We further theoretically prove that this optimization cycle can significantly reduce the generalization error under some conditions. Additionally, we propose a class-aware adaptive augmentation module to further improve the representation of minority classes, and an auxiliary branch to maximize data utilization by leveraging all labeled and unlabeled samples. Comprehensive evaluations on various commonly used
benchmark datasets show that CPG achieves consistent improvements, surpassing state-of-the-art methods by up to $\textbf{15.97%}$ in accuracy. The code is available at https://github.com/yaxinhou/CPG.
Read the full abstract ▾
All authors Yaxin Hou, Bo Han, Yuheng Jia, Hui Liu, Junhui Hou