Computer Vision
CHUG: Crowdsourced User-Generated HDR Video Quality Dataset
Shreshth Saini, Alan C. Bovik, Neil Birkbeck et al. (+2 more)
High Dynamic Range (HDR)
videos enhance visual experiences with superior brightness, contrast, and color depth. The surge of User-Generated Content (UGC) on platforms like YouTube and TikTok introduces unique challenges for HDR
video quality assessment (VQA) due to diverse capture conditions, editing artifacts, and compression distortions. Existing HDR-VQA
datasets primarily focus on professionally generated content (PGC), leaving a gap in understanding real-world UGC-HDR degradations. To address this, we introduce CHUG: Crowdsourced User-Generated HDR
Video Quality
Dataset, the first large-scale subjective study on UGC-HDR quality. CHUG comprises 856 UGC-HDR source
videos, transcoded across multiple resolutions and bitrates to simulate real-world scenarios, totaling 5,992
videos. A large-scale study via Amazon Mechanical Turk collected 211,848 perceptual ratings. CHUG provides a
benchmark for analyzing UGC-specific distortions in HDR
videos. We anticipate CHUG will advance No-Reference (NR) HDR-VQA research by offering a large-scale, diverse, and real-world UGC
dataset. The
dataset is publicly available at: https://shreshthsaini.github.io/CHUG/.
Read the full abstract ▾
All authors Shreshth Saini, Alan C. Bovik, Neil Birkbeck, Yilin Wang, Balu Adsumilli
Computer Vision
Fast Self-Supervised depth and mask aware Association for Multi-Object Tracking
Milad Khanchi, Maria Amer and Charalambos Poullis
Multi-object tracking (MOT) methods often rely on Intersection-over-Union (IoU) for association. However, this becomes unreliable when objects are similar or occluded. Also, computing IoU for
segmentation masks is computationally expensive. In this work, we use
segmentation masks to capture object shapes, but we do not compute
segmentation IoU. Instead, we fuse depth and mask features and pass them through a compact encoder trained self-supervised. This encoder produces stable object representations, which we use as an additional similarity cue alongside bounding box IoU and re-identification features for matching. We obtain depth maps from a zero-shot depth
estimator and object masks from a promptable visual
segmentation model to obtain fine-grained spatial cues. Our MOT method is the first to use the self-supervised encoder to refine
segmentation masks without computing masks IoU. MOT can be divided into joint detection-ReID (JDR) and tracking-by-detection (TBD) models. The latter are computationally more efficient. Experiments of our TBD method on challenging
benchmarks with non-linear motion, occlusion, and crowded scenes, such as SportsMOT and DanceTrack, show that our method outperforms the TBD state-of-the-art on most metrics, while achieving competitive performance on simpler
benchmarks with linear motion, such as MOT17.
Read the full abstract ▾
Computer Vision
Cluster-Aware Prompt Ensemble Learning for Few-Shot Vision-Language Model Adaptation
Zhi Chen, Xin Yu, Xiaohui Tao et al. (+2 more)
Vision-
language models (VLMs) such as CLIP achieve zero-shot transfer across various tasks by pre-training on numerous image-text pairs. These models often benefit from using an ensemble of context
prompts to represent a class. Despite being effective, conventional
prompt ensembling that averages textual features of context
prompts often yields suboptimal results. This is because feature averaging shifts the class centroids away from the true class distribution. To address this issue, we propose the Cluster-Aware
Prompt Ensemble Learning (CAPEL) framework, which preserves the cluster nature of context
prompts. CAPEL classifies images into one of several class clusters, each represented by a distinct
prompt. Instead of ensembling
prompts in the feature space, we perform ensembling in the classification logits space, aligning better with the visual feature distribution. To further optimize
prompt fine-tuning while maintaining cluster-specific discriminative power, we introduce a cluster-preserving regularization term. This ensures that
prompts remain distinct and specialized for different clusters, preventing collapse into a uniform direction. Additionally, we integrate an adaptive
prompt weighting technique to dynamically adjust the
attention weights for flawed or ambiguous
prompts, ensuring robust performance across diverse
datasets and tasks.
Read the full abstract ▾
All authors Zhi Chen, Xin Yu, Xiaohui Tao, Yan Li, Zi Huang
Computer Vision · Cross-listed from cs.IR
MTMD: A Multi-Task Multi-Domain Framework for Unified Ad Lightweight Ranking at Pinterest
Xiao Yang, Peifeng Yin, Abe Engle et al. (+2 more)
The lightweight ad ranking layer, living after the
retrieval stage and before the fine ranker, plays a critical role in the success of a cascaded ad recommendation system. Due to the fact that there are multiple optimization tasks depending on the ad domain, e.g., Click Through Rate (CTR) for click ads and Conversion Rate (CVR) for conversion ads, as well as multiple surfaces where an ad is served (home feed, search, or related item recommendation) with diverse ad products (shopping or standard ad); it is an essentially challenging problem in industry on how to do joint holistic optimization in the lightweight ranker, such that the overall platform's value, advertiser's value, and user's value are maximized. Deep Neural Network (DNN)-based multitask learning (MTL) can handle multiple goals naturally, with each prediction head mapping to a particular optimization goal. However, in practice, it is unclear how to unify data from different surfaces and ad products into a single model. It is critical to learn domain-specialized knowledge and explicitly transfer knowledge between domains to make MTL effective. We present a Multi-Task Multi-Domain (MTMD) architecture under the classic Two-Tower paradigm, with the following key contributions: 1) handle different prediction tasks, ad products, and ad serving surfaces in a unified framework; 2) propose a novel mixture-of-expert architecture to learn both specialized knowledge each domain and common knowledge shared between domains; 3) propose a domain adaption module to encourage knowledge transfer between experts; 4) constrain the modeling of different prediction tasks. MTMD improves the offline loss value by 12% to 36%, mapping to 2% online reduction in cost per click. We have deployed this single MTMD framework into production for Pinterest ad recommendation replacing 9 production models.
Read the full abstract ▾
All authors Xiao Yang, Peifeng Yin, Abe Engle, Jinfeng Zhuang, Ling Leng
Computer Vision · Cross-listed from cs.CL
Ruizhe Zhu
The widespread application of large
vision language models has significantly raised safety concerns. In this project, we investigate text
prompt injection, a simple yet effective method to mislead these models. We developed an
algorithm for this type of attack and demonstrated its effectiveness and efficiency through experiments. Compared to other attack methods, our approach is particularly effective for large models without high demand for computational resources.
Read the full abstract ▾
Computer Vision
Cell Instance Segmentation: The Devil Is in the Boundaries
Peixian Liang, Yifan Ding, Yizhe Zhang et al. (+9 more)
State-of-the-art (SOTA) methods for
cell instance
segmentation are based on deep learning (DL) semantic
segmentation approaches, focusing on distinguishing foreground pixels from background pixels. In order to identify
cell instances from foreground pixels (e.g., pixel clustering), most methods decompose instance information into pixel-wise objectives, such as distances to foreground-background boundaries (distance maps), heat gradients with the center point as heat source (heat
diffusion maps), and distances from the center point to foreground-background boundaries with fixed angles (
star-shaped polygons). However, pixel-wise objectives may lose significant
geometric properties of the
cell instances, such as shape,
curvature, and convexity, which require a collection of pixels to represent. To address this challenge, we present a novel pixel clustering method, called Ceb (for
Cell boundaries), to leverage
cell boundary features and labels to divide foreground pixels into
cell instances. Starting with probability maps generated from semantic
segmentation, Ceb first extracts potential foreground-foreground boundaries with a revised Watershed
algorithm. For each boundary candidate, a boundary feature representation (called boundary signature) is constructed by sampling pixels from the current foreground-foreground boundary as well as the neighboring background-foreground boundaries. Next, a boundary classifier is used to predict its binary boundary label based on the corresponding boundary signature. Finally,
cell instances are obtained by dividing or merging neighboring regions based on the predicted boundary labels. Extensive experiments on six
datasets demonstrate that Ceb outperforms existing pixel clustering methods on semantic
segmentation probability maps. Moreover, Ceb achieves highly competitive performance compared to SOTA
cell instance
segmentation methods.
Read the full abstract ▾
All authors Peixian Liang, Yifan Ding, Yizhe Zhang, Jianxu Chen, Hao Zheng, Hongxiao Wang, Yejia Zhang, Guangyu Meng, Tim Weninger, Michael Niemier, X. Sharon Hu, Danny Z Chen
Computer Vision · Cross-listed from cs.LG
Harnessing Self-Supervised Deep Learning and Geostationary Remote Sensing for Advancing Wildfire and Associated Air Quality Monitoring: Improved Smoke and Fire Front Masking using GOES and TEMPO Radiance Data
Nicholas LaHaye, Thilanka Munashinge, Hugo Lee et al. (+4 more)
This work demonstrates the possibilities for improving wildfire and air quality management in the western United States by leveraging the unprecedented hourly data from NASA's TEMPO satellite mission and advances in self-supervised deep learning. Here we demonstrate the efficacy of deep learning for mapping the near real-time hourly spread of wildfire fronts and smoke plumes using an innovative self-supervised deep learning-system: successfully distinguishing smoke plumes from clouds using GOES-18 and TEMPO data, strong agreement across the smoke and fire masks generated from different sensing modalities as well as significant improvement over operational products for the same cases.
Read the full abstract ▾
All authors Nicholas LaHaye, Thilanka Munashinge, Hugo Lee, Xiaohua Pan, Gonzalo Gonzalez Abad, Hazem Mahmoud, Jennifer Wei
Computer Vision
Exploration of Incremental Synthetic Non-Morphed Images for Single Morphing Attack Detection
David Benavente-Rios, Juan Ruiz Rodriguez and Gustavo Gatica
This paper investigates the use of synthetic face data to enhance Single-Morphing Attack Detection (S-MAD), addressing the limitations of availability of large-scale
datasets of bona fide images due to
privacy concerns. Various morphing tools and cross-
dataset evaluation schemes were utilized to conduct this study. An incremental testing protocol was implemented to assess the generalization capabilities as more and more synthetic images were added. The results of the experiments show that generalization can be improved by carefully incorporating a controlled number of synthetic images into existing
datasets or by gradually adding bona fide images during training. However, indiscriminate use of synthetic data can lead to sub-optimal performance. Evenmore, the use of only synthetic data (morphed and non-morphed images) achieves the highest Equal Error Rate (EER), which means in operational scenarios the best option is not relying only on synthetic data for S-MAD.
Read the full abstract ▾
Computer Vision
Post Processing of image segmentation using Conditional Random Fields
Aashish Dhawan, Pankaj Bodani and Vishal Garg
The output of image the
segmentation process is usually not very clear due to low quality features of Satellite images. The purpose of this study is to find a suitable Conditional Random Field (CRF) to achieve better clarity in a segmented image. We started with different types of CRFs and studied them as to why they are or are not suitable for our purpose. We evaluated our approach on two different
datasets - Satellite imagery having low quality features and high quality Aerial photographs. During the study we experimented with various CRFs to find which CRF gives the best results on images and compared our results on these
datasets to show the pitfalls and potentials of different approaches.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Decomposer Networks: Deep Component Analysis and Synthesis
Mohsen Joneidi
We propose the Decomposer Networks (DecompNet), a semantic autoencoder that factorizes an input into multiple interpretable components. Unlike classical autoencoders that compress an input into a single latent representation, the Decomposer Network maintains N parallel branches, each assigned a residual input defined as the original signal minus the reconstructions of all other branches. By unrolling a Gauss–Seidel style block-coordinate descent into a differentiable network, DecompNet enforce explicit competition among components, yielding parsimonious, semantically meaningful representations. We situate our model relative to linear decomposition methods (PCA, NMF), deep unrolled optimization, and object-centric architectures (MONet, IODINE, Slot
Attention), and highlight its novelty as the first semantic autoencoder to implement an all-but-one residual update rule.
Read the full abstract ▾
Computer Vision
Task-Aware Resolution Optimization for Visual Large Language Models
Weiqing Luo, Zhen Tan, Yifan Li et al. (+4 more)
Real-world
vision-
language applications demand varying levels of perceptual granularity. However, most existing visual large
language models (VLLMs), such as LLaVA, pre-assume a fixed resolution for downstream tasks, which leads to subpar performance. To address this problem, we first conduct a comprehensive and pioneering investigation into the resolution preferences of different
vision-
language tasks, revealing a correlation between resolution preferences with image complexity, and uncertainty variance of the VLLM at different image input resolutions. Building on this insight, we propose an empirical formula to determine the optimal resolution for a given
vision-
language task, combining these two factors. Second, based on rigorous experiments, we propose a novel parameter-efficient fine-tuning technique to extend the visual input resolution of pre-trained VLLMs to the identified optimal resolution. Extensive experiments on various
vision-
language tasks validate the effectiveness of our method.
Read the full abstract ▾
All authors Weiqing Luo, Zhen Tan, Yifan Li, Xinyu Zhao, Kwonjoon Lee, Behzad Dariush, Tianlong Chen
Computer Vision · Cross-listed from cs.RO
Cross-Sensor Touch Generation
Samanta Rodriguez, Yiming Dou, Miquel Oller et al. (+2 more)
Today's visuo-tactile sensors come in many shapes and sizes, making it challenging to develop general-purpose tactile representations. This is because most models are tied to a specific sensor design. To address this challenge, we propose two approaches to cross-sensor image generation. The first is an end-to-end method that leverages paired data (Touch2Touch). The second method builds an intermediate depth representation and does not require paired data (T2D2: Touch-to-Depth-to-Touch). Both methods enable the use of sensor-specific models across multiple sensors via the cross-sensor touch generation process. Together, these models offer flexible solutions for sensor translation, depending on data availability and application needs. We demonstrate their effectiveness on downstream tasks such as in-hand pose estimation and behavior cloning, successfully transferring models trained on one sensor to another. Project page: https://samantabelen.github.io/cross_sensor_touch_generation.
Read the full abstract ▾
All authors Samanta Rodriguez, Yiming Dou, Miquel Oller, Andrew Owens, Nima Fazeli
Computer Vision
Towards Understanding Ambiguity Resolution in Multimodal Inference of Meaning
Yufei Wang, Adriana Kovashka, Loretta Fernández et al. (+2 more)
We investigate a new setting for foreign
language learning, where learners infer the meaning of unfamiliar words in a
multimodal context of a sentence describing a paired image. We conduct studies with human participants using different image-text pairs. We analyze the features of the data (i.e., images and texts) that make it easier for participants to infer the meaning of a masked or unfamiliar word, and what
language backgrounds of the participants correlate with success. We find only some intuitive features have strong correlations with participant performance, prompting the need for further investigating of predictive features for success in these tasks. We also analyze the ability of AI systems to reason about participant performance, and discover promising future directions for improving this
reasoning ability.
Read the full abstract ▾
All authors Yufei Wang, Adriana Kovashka, Loretta Fernández, Marc N. Coutanche, Seth Wiener
Computer Vision · Cross-listed from cs.LG
Causality $\neq$ Decodability, and Vice Versa: Lessons from Interpreting Counting ViTs
Lianghuan Huang and Yingshan Chang
Mechanistic interpretability seeks to uncover how internal components of neural networks give rise to predictions. A persistent challenge, however, is disentangling two often conflated notions: decodability–the recoverability of information from hidden states–and causality–the extent to which those states functionally influence outputs. In this work, we investigate their relationship in
vision transformers (ViTs) fine-tuned for object counting. Using activation patching, we test the causal role of spatial and CLS
tokens by transplanting activations across clean-corrupted image pairs. In parallel, we train linear probes to assess the decodability of count information at different depths. Our results reveal systematic mismatches: middle-layer object
tokens exert strong causal influence despite being weakly decodable, whereas final-layer object
tokens support accurate decoding yet are functionally inert. Similarly, the CLS
token becomes decodable in mid-layers but only acquires causal power in the final layers. These findings highlight that decodability and causality reflect complementary dimensions of representation–what information is present versus what is used–and that their divergence can expose hidden computational
circuits.
Read the full abstract ▾
Computer Vision
Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping
Dwip Dalal, Gautam Vashishtha, Utkarsh Mishra et al. (+6 more)
Multimodal large
language models (MLLMs) often miss small details and spatial relations in cluttered scenes, leading to errors in fine-grained perceptual grounding. We introduce AttWarp, a lightweight method that allocates more resolution to query-relevant content while compressing less informative areas, all while preserving global context. At test time, the approach uses an MLLM's cross-modal
attention to perform rectilinear warping of the input image, reallocating spatial resolution toward regions the model deems important, without changing model weights or architecture. This
attention-guided warping preserves all original image information but redistributes it non-uniformly, so small objects and subtle relationships become easier for the same model to read while the global layout remains intact. Across five
benchmarks (TextVQA, GQA, DocVQA, POPE, MMMU) and four MLLMs (LLaVA, Qwen-VL, InternVL, and InstructBLIP), AttWarp consistently improves accuracy, strengthens compositional
reasoning, and reduces
hallucinations, outperforming four competitive baselines that manipulate raw images at test time. Together, these results show that
attention-guided warping prioritizes information relevant to the query while preserving context, and that the same MLLMs perform better when given such warped inputs.
Read the full abstract ▾
All authors Dwip Dalal, Gautam Vashishtha, Utkarsh Mishra, Jeonghwan Kim, Madhav Kanda, Hyeonjeong Ha, Svetlana Lazebnik, Heng Ji, Unnat Jain
Computer Vision · Cross-listed from cs.LG
Reliable Active Learning from Unreliable Labels via Neural Collapse Geometry
Atharv Goel, Sharat Agarwal, Saket Anand et al. (+1 more)
Active Learning (AL) promises to reduce annotation cost by prioritizing informative samples, yet its reliability is undermined when labels are noisy or when the data distribution shifts. In practice, annotators make mistakes, rare categories are ambiguous, and conventional AL heuristics (uncertainty, diversity) often amplify such errors by repeatedly selecting mislabeled or redundant samples. We propose Reliable Active Learning via Neural Collapse Geometry (NCAL-R), a framework that leverages the emergent
geometric regularities of deep networks to counteract unreliable supervision. Our method introduces two complementary signals: (i) a Class-Mean
Alignment Perturbation score, which quantifies how candidate samples structurally stabilize or distort inter-class geometry, and (ii) a Feature Fluctuation score, which captures temporal instability of representations across training checkpoints. By combining these signals, NCAL-R prioritizes samples that both preserve class separation and highlight ambiguous regions, mitigating the effect of noisy or redundant labels. Experiments on ImageNet-100 and CIFAR100 show that NCAL-R consistently outperforms standard AL baselines, achieving higher accuracy with fewer labels, improved robustness under synthetic label noise, and stronger generalization to out-of-distribution data. These results suggest that incorporating
geometric reliability criteria into acquisition decisions can make Active Learning less brittle to annotation errors and distribution shifts, a key step toward trustworthy deployment in real-world labeling pipelines. Our code is available at https://github.com/
Vision-IIITD/NCAL.
Read the full abstract ▾
All authors Atharv Goel, Sharat Agarwal, Saket Anand, Chetan Arora
Computer Vision · Cross-listed from cs.CL
VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation
Yubo Sun, Chunyi Peng, Yukun Yan et al. (+5 more)
Visual
Retrieval-Augmented Generation (VRAG) has emerged as a promising paradigm for equipping
Vision-
Language Models (VLMs) with external visual evidence, enabling them to go beyond parametric knowledge when answering visually grounded questions. However, in such multi-image settings, VLMs still often suffer from visual
hallucinations and struggle to accurately identify the question-relevant evidence needed for reliable
reasoning. Existing methods usually lack an explicit cross-image evidence collection process, and also provide limited credit assignment when jointly optimizing perception and
reasoning. To address this issue, we propose EVisRAG, an evidence-guided visual
retrieval-augmented framework for multi-image
reasoning. EVisRAG first observes the retrieved images, records question-relevant visual evidence from each image, and then performs
reasoning and answer generation based on the aggregated evidence. We further introduce RS-GRPO, which aligns reward signals with
token spans from different stages, improving training stability and strengthening the joint optimization of evidence localization and
reasoning. Experiments on multiple visual question answering
benchmarks show that EVisRAG consistently outperforms the backbone VLM by an average of about 19%, while substantially reducing visual
hallucinations. These results demonstrate that explicit evidence collection and scoped reward design are effective for improving visual grounding and
reasoning reliability in multi-image settings. Codes and data are available at https://github.com/OpenBMB/VisRAG
Read the full abstract ▾
All authors Yubo Sun, Chunyi Peng, Yukun Yan, Shi Yu, Zhenghao Liu, Sen Mei, Chi Chen, Maosong Sun
Computer Vision
Multi Camera Connected Vision System with Multi View Analytics: A Comprehensive Survey
Muhammad Munsif, Waqas Ahmad, Amjid Ali et al. (+3 more)
Connected
Vision Systems (CVS) are transforming a variety of applications, including autonomous vehicles, smart cities, surveillance, and human-
robot interaction. These systems harness multi-view multi-camera (MVMC) data to provide enhanced situational awareness through the integration of MVMC tracking, re-identification (Re-ID), and action understanding (AU). However, deploying CVS in real-world, dynamic environments presents a number of challenges, particularly in addressing occlusions, diverse viewpoints, and environmental variability. Existing surveys have focused primarily on isolated tasks such as tracking, Re-ID, and AU, often neglecting their integration into a cohesive system. These reviews typically emphasize single-view setups, overlooking the complexities and opportunities provided by multi-camera collaboration and multi-view data analysis. To the best of our knowledge, this survey is the first to offer a comprehensive and integrated review of MVMC that unifies MVMC tracking, Re-ID, and AU into a single framework. We propose a unique taxonomy to better understand the critical components of CVS, dividing it into four key parts: MVMC tracking, Re-ID, AU, and combined methods. We systematically arrange and summarize the state-of-the-art
datasets, methodologies, results, and evaluation metrics, providing a structured view of the field's progression. Furthermore, we identify and discuss the open research questions and challenges, along with emerging technologies such as lifelong learning,
privacy, and
federated learning, that need to be addressed for future advancements. The paper concludes by outlining key research directions for enhancing the robustness, efficiency, and adaptability of CVS in complex, real-world applications. We hope this survey will inspire innovative solutions and guide future research toward the next generation of intelligent and adaptive CVS.
Read the full abstract ▾
All authors Muhammad Munsif, Waqas Ahmad, Amjid Ali, Mohib Ullah, Adnan Hussain, Sung Wook Baik
Computer Vision
Adaptive Fusion Network with Temporal-Ranked and Motion-Intensity Dynamic Images for Micro-expression Recognition
Thi Bich Phuong Man, Luu Tu Nguyen, Vu Tram Anh Khuong et al. (+2 more)
Micro-expressions (MEs) are subtle, transient facial changes with very low intensity, almost imperceptible to the naked eye, yet they reveal a person genuine
emotion. They are of great value in lie detection, behavioral analysis, and psychological assessment. This paper proposes a novel MER method with two main contributions. First, we propose two complementary representations - Temporal-ranked dynamic image, which emphasizes temporal progression, and Motion-intensity dynamic image, which highlights subtle motions through a frame reordering mechanism incorporating motion intensity. Second, we propose an Adaptive fusion network, which automatically learns to optimally integrate these two representations, thereby enhancing discriminative ME features while suppressing noise. Experiments on three
benchmark datasets (CASME-II, SAMM and MMEW) demonstrate the superiority of the proposed method. Specifically, AFN achieves 93.95 Accuracy and 0.897 UF1 on CASME-II, setting a new state-of-the-art
benchmark. On SAMM, the method attains 82.47 Accuracy and 0.665 UF1, demonstrating more balanced recognition across classes. On MMEW, the model achieves 76.00 Accuracy, further confirming its generalization ability. The obtained results show that both the input and the proposed architecture play important roles in improving the performance of MER. Moreover, they provide a solid foundation for further research and practical applications in the fields of affective computing, lie detection, and human-computer interaction.
Read the full abstract ▾
All authors Thi Bich Phuong Man, Luu Tu Nguyen, Vu Tram Anh Khuong, Thanh Ha Le, Thi Duyen Ngo
Computer Vision · Cross-listed from cs.CL
Layout-Aware Parsing Meets Efficient LLMs: A Unified, Scalable Framework for Resume Information Extraction and Evaluation
Fanwei Zhu, Jinke Yu, Zulong Chen et al. (+6 more)
Automated resume information extraction is critical for scaling talent acquisition, yet its real-world deployment faces three major challenges: the extreme heterogeneity of resume layouts and content, the high cost and latency of large
language models (LLMs), and the lack of standardized
datasets and evaluation tools. In this work, we present a layout-aware and efficiency-optimized framework for automated extraction and evaluation that addresses all three challenges. Our system combines a fine-tuned layout parser to normalize diverse document formats, an inference-efficient LLM extractor based on parallel prompting and instruction tuning, and a robust two-stage automated evaluation framework supported by new
benchmark datasets. Extensive experiments show that our framework significantly outperforms strong baselines in both accuracy and efficiency. In particular, we demonstrate that a fine-tuned compact 0.6B LLM achieves top-tier accuracy while significantly reducing inference latency and computational cost. The system is fully deployed in Alibaba's intelligent HR platform, supporting real-time applications across its business units.
Read the full abstract ▾
All authors Fanwei Zhu, Jinke Yu, Zulong Chen, Ying Zhou, Junhao Ji, Zhibo Yang, Yuxue Zhang, Haoyuan Hu, Zhenghao Liu
Computer Vision
StreamingVLM: Real-Time Understanding for Infinite Video Streams
Ruyi Xu, Guangxuan Xiao, Yukang Chen et al. (+3 more)
Vision-
language models (VLMs) could power real-time assistants and autonomous
agents, but they face a critical challenge: understanding near-infinite
video streams without escalating latency and
memory usage. Processing entire
videos with full
attention leads to quadratic computational costs and poor performance on long
videos. Meanwhile, simple sliding window methods are also flawed, as they either break coherence or suffer from high latency due to redundant recomputation. In this paper, we introduce StreamingVLM, a model designed for real-time, stable understanding of infinite visual input. Our approach is a unified framework that aligns training with streaming inference. During inference, we maintain a compact KV cache by reusing states of
attention sinks, a short window of recent
vision tokens, and a long window of recent text
tokens. This streaming ability is instilled via a simple supervised fine-tuning (SFT) strategy that applies full
attention on short, overlapped
video chunks, which effectively mimics the inference-time
attention pattern without training on prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, a new
benchmark with
videos averaging over two hours that requires dense, per-second
alignment between frames and text. On Inf-Streams-Eval, StreamingVLM achieves a 66.18% win rate against GPT-4O mini and maintains stable, real-time performance at up to 8 FPS on a single NVIDIA H100. Notably, our SFT strategy also enhances general VQA abilities without any VQA-specific fine-tuning, improving performance on LongVideoBench by +4.30 and OVOBench Realtime by +5.96. Code is available at https://github.com/mit-han-lab/streaming-vlm.
Read the full abstract ▾
All authors Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Yao Lu, Song Han
Computer Vision
VITA-VLA: Efficiently Teaching Vision-Language Models to Act via Action Expert Distillation
Shaoqi Dong, Chaoyou Fu, Haihan Gao et al. (+12 more)
Vision-
Language Action (VLA) models significantly advance robotic manipulation by leveraging the strong perception capabilities of pretrained
vision-
language models (VLMs). By integrating action modules into these pretrained models, VLA methods exhibit improved generalization. However, training them from scratch is costly. In this work, we propose a simple yet effective
distillation-based framework that equips VLMs with action-execution capability by transferring knowledge from pretrained small action models. Our architecture retains the original VLM structure, adding only an action
token and a state encoder to incorporate physical inputs. To distill action knowledge, we adopt a two-stage training strategy. First, we perform lightweight
alignment by mapping VLM hidden states into the action space of the small action model, enabling effective reuse of its pretrained action decoder and avoiding expensive pretraining. Second, we selectively fine-tune the
language model, state encoder, and action modules, enabling the system to integrate
multimodal inputs with precise action generation. Specifically, the action
token provides the VLM with a direct handle for predicting future actions, while the state encoder allows the model to incorporate
robot dynamics not captured by
vision alone. This design yields substantial efficiency gains over training large VLA models from scratch. Compared with previous state-of-the-art methods, our method achieves 97.3% average success rate on LIBERO (11.8% improvement) and 93.5% on LIBERO-LONG (24.5% improvement). In real-world experiments across five manipulation tasks, our method consistently outperforms the teacher model, achieving 82.0% success rate (17% improvement), which demonstrate that action
distillation effectively enables VLMs to generate precise actions while substantially reducing training costs.
Read the full abstract ▾
All authors Shaoqi Dong, Chaoyou Fu, Haihan Gao, Yi-Fan Zhang, Chi Yan, Chu Wu, Xiaoyu Liu, Yunhang Shen, Jing Huo, Deqiang Jiang, Haoyu Cao, Yang Gao, Xing Sun, Ran He, Caifeng Shan
Computer Vision
SpaceVista: All-Scale Visual Spatial Reasoning from mm to km
Peiwen Sun, Shiqiang Lang, Dongming Wu et al. (+8 more)
With the current surge in spatial
reasoning explorations, researchers have made significant progress in understanding indoor scenes, but still struggle with diverse applications such as robotics and autonomous driving. This paper aims to advance all-scale spatial
reasoning across diverse scenarios by tackling two key challenges: 1) the heavy reliance on indoor 3D scans and labor-intensive manual annotations for
dataset curation; 2) the absence of effective all-scale scene modeling, which often leads to overfitting to individual scenes. In this paper, we introduce a holistic solution that integrates a structured spatial
reasoning knowledge system, scale-aware modeling, and a progressive training paradigm, as the first attempt to broaden the all-scale spatial intelligence of MLLMs to the best of our knowledge. Using a task-specific, specialist-driven automated pipeline, we curate over 38K
video scenes across 5 spatial scales to create SpaceVista-1M, a
dataset comprising approximately 1M spatial QA pairs spanning 19 diverse task types. While specialist models can inject useful domain knowledge, they are not reliable for evaluation. We then build an all-scale
benchmark with precise annotations by manually recording, retrieving, and assembling
video-based data. However, naive training with SpaceVista-1M often yields suboptimal results due to the potential knowledge conflict. Accordingly, we introduce SpaceVista-7B, a spatial
reasoning model that accepts dense inputs beyond semantics and uses scale as an anchor for scale-aware experts and progressive rewards. Finally, extensive evaluations across 5
benchmarks, including our SpaceVista-Bench, demonstrate competitive performance, showcasing strong generalization across all scales and scenarios. Our
dataset, model, and
benchmark will be released on https://peiwensun2000.github.io/mm2km .
Read the full abstract ▾
All authors Peiwen Sun, Shiqiang Lang, Dongming Wu, Yi Ding, Kaituo Feng, Huadai Liu, Zhen Ye, Rui Liu, Yun-Hui Liu, Jianan Wang, Xiangyu Yue
Computer Vision · Cross-listed from cs.LG
STaTS: Structure-Aware Temporal Sequence Summarization via Statistical Window Merging
Disharee Bhowmick, Ranjith Ramanathan and Sathyanarayanan N. Aakur
Time series data often contain latent temporal structure, transitions between locally stationary regimes, repeated motifs, and bursts of variability, that are rarely leveraged in standard representation learning pipelines. Existing models typically operate on raw or fixed-window sequences, treating all time steps as equally informative, which leads to inefficiencies, poor robustness, and limited scalability in long or noisy sequences. We propose STaTS, a lightweight, unsupervised framework for Structure-Aware Temporal Summarization that adaptively compresses both univariate and multivariate time series into compact, information-preserving
token sequences. STaTS detects change points across multiple temporal resolutions using a BIC-based statistical divergence criterion, then summarizes each segment using simple functions like the mean or generative models such as GMMs. This process achieves up to 30x sequence compression while retaining core temporal dynamics. STaTS operates as a model-agnostic preprocessor and can be integrated with existing unsupervised time series encoders without retraining. Extensive experiments on 150+
datasets, including classification tasks on the UCR-85, UCR-128, and UEA-30 archives, and
forecasting on ETTh1 and ETTh2, ETTm1, and Electricity, demonstrate that STaTS enables 85-90% of the full-model performance while offering dramatic reductions in computational cost. Moreover, STaTS improves robustness under noise and preserves discriminative structure, outperforming uniform and clustering-based compression baselines. These results position STaTS as a principled, general-purpose solution for efficient, structure-aware time series modeling.
Read the full abstract ▾
Computer Vision
Vision Language Models: A Survey of 26K Papers
Fengming Lin
We present a transparent, reproducible measurement of research trends across 26,104 accepted papers from CVPR, ICLR, and NeurIPS spanning 2023-2025. Titles and abstracts are normalized, phrase-protected, and matched against a hand-crafted lexicon to assign up to 35 topical labels and mine fine-grained cues about tasks, architectures, training regimes, objectives,
datasets, and co-mentioned modalities. The analysis quantifies three macro shifts: (1) a sharp rise of
multimodal vision-
language-LLM work, which increasingly reframes classic perception as instruction following and multi-step
reasoning; (2) steady expansion of generative methods, with
diffusion research consolidating around controllability,
distillation, and speed; and (3) resilient 3D and
video activity, with composition moving from NeRFs to Gaussian splatting and a growing emphasis on human- and
agent-centric understanding. Within VLMs, parameter-efficient adaptation like prompting/adapters/LoRA and lightweight
vision-
language bridges dominate; training practice shifts from building encoders from scratch to instruction tuning and finetuning strong backbones; contrastive objectives recede relative to cross-
entropy/ranking and
distillation. Cross-venue comparisons show CVPR has a stronger 3D footprint and ICLR the highest VLM share, while reliability themes such as efficiency or robustness diffuse across areas. We release the lexicon and methodology to enable
auditing and extension. Limitations include lexicon recall and abstract-only scope, but the longitudinal signals are consistent across venues and years.
Read the full abstract ▾
Computer Vision
FSP-DETR: Few-Shot Prototypical Parasitic Ova Detection
Shubham Trehan, Udhav Ramachandran, Akash Rao et al. (+2 more)
Object detection in biomedical settings is fundamentally constrained by the scarcity of labeled data and the frequent emergence of novel or rare categories. We present FSP-DETR, a unified detection framework that enables robust few-shot detection, open-set recognition, and generalization to unseen biomedical tasks within a single model. Built upon a class-agnostic DETR backbone, our approach constructs class prototypes from original support images and learns an
embedding space using augmented views and a lightweight
transformer decoder. Training jointly optimizes a prototype matching loss, an
alignment-based separation loss, and a KL divergence regularization to improve discriminative feature learning and calibration under scarce supervision. Unlike prior work that tackles these tasks in isolation, FSP-DETR enables inference-time flexibility to support unseen class recognition, background rejection, and cross-task adaptation without retraining. We also introduce a new ova species detection
benchmark with 20 parasite classes and establish standardized evaluation protocols. Extensive experiments across ova, blood
cell, and malaria detection tasks demonstrate that FSP-DETR significantly outperforms prior few-shot and prototype-based detectors, especially in low-shot and open-set scenarios.
Read the full abstract ▾
All authors Shubham Trehan, Udhav Ramachandran, Akash Rao, Ruth Scimeca, Sathyanarayanan N. Aakur
Computer Vision · Cross-listed from cs.CL
Dyna-Mind: Learning to Simulate from Experience for Better AI Agents
Xiao Yu, Baolin Peng, Michel Galley et al. (+6 more)
Reasoning models have recently shown remarkable progress in domains such as math and coding. However, their expert-level abilities in math and coding contrast sharply with their performance in long-horizon, interactive tasks such as web
navigation and computer/phone-use. Inspired by literature on human cognition, we argue that current AI
agents need ”vicarious trial and error” - the capacity to mentally simulate alternative futures before acting - in order to enhance their understanding and performance in complex interactive environments. We introduce Dyna-Mind, a two-stage training framework that explicitly teaches (V)LM
agents to integrate such simulation into their
reasoning. In stage 1, we introduce
Reasoning with Simulations (ReSim), which trains the
agent to generate structured
reasoning traces from expanded search trees built from real experience gathered through environment interactions. ReSim thus grounds the
agent's
reasoning in faithful world dynamics and equips it with the ability to anticipate future states in its
reasoning. In stage 2, we propose Dyna-GRPO, an online reinforcement learning method to further strengthen the
agent's simulation and decision-making ability by using both outcome rewards and intermediate states as feedback from real rollouts. Experiments on two synthetic
benchmarks (Sokoban and ALFWorld) and one realistic
benchmark (AndroidWorld) demonstrate that (1) ReSim effectively infuses simulation ability into AI
agents, and (2) Dyna-GRPO leverages outcome and interaction-level signals to learn better policies for long-horizon, planning-intensive tasks. Together, these results highlight the central role of simulation in enabling AI
agents to reason, plan, and act more effectively in the ever more challenging environments.
Read the full abstract ▾
All authors Xiao Yu, Baolin Peng, Michel Galley, Hao Cheng, Qianhui Wu, Janardhan Kulkarni, Suman Nath, Zhou Yu, Jianfeng Gao
Computer Vision
TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Control
Minkyoung Cho, Ruben Ohana, Christian Jacobsen et al. (+4 more)
Current controllable
diffusion models typically rely on fixed architectures that modify intermediate activations to inject guidance conditioned on a new modality. This approach uses a static conditioning strategy for a dynamic, multi-stage denoising process, limiting the model's ability to adapt its response as the generation evolves from coarse structure to fine detail. We introduce TC-LoRA (Temporally Modulated Conditional LoRA), a new paradigm that enables dynamic, context-aware control by conditioning the model's weights directly. Our framework uses a hypernetwork to generate LoRA adapters on-the-fly, tailoring weight modifications for the frozen backbone at each
diffusion step based on time and the user's condition. This mechanism enables the model to learn and execute an explicit, adaptive strategy for applying conditional guidance throughout the entire generation process. Through experiments on various data domains, we demonstrate that this dynamic, parametric control significantly enhances generative fidelity and adherence to spatial conditions compared to static, activation-based methods. TC-LoRA establishes an alternative approach in which the model's conditioning strategy is modified through a deeper functional adaptation of its weights, allowing control to align with the dynamic demands of the task and generative stage.
Read the full abstract ▾
All authors Minkyoung Cho, Ruben Ohana, Christian Jacobsen, Adityan Jothi, Min-Hung Chen, Z. Morley Mao, Ethem Can
Computer Vision
FLOWING: Implicit Neural Flows for Structure-Preserving Morphing
Arthur Bizzi, Matias Grynberg, Vitor Matias et al. (+7 more)
Morphing is a long-standing problem in
vision and computer graphics, requiring a time-dependent warping for feature
alignment and a blending for smooth interpolation. Recently, multilayer perceptrons (MLPs) have been explored as implicit neural representations (INRs) for modeling such deformations, due to their meshlessness and differentiability; however, extracting coherent and accurate morphings from standard MLPs typically relies on costly regularizations, which often lead to unstable training and prevent effective feature
alignment. To overcome these limitations, we propose FLOWING (FLOW morphING), a framework that recasts warping as the construction of a differential vector flow, naturally ensuring continuity, invertibility, and temporal coherence by encoding structural flow properties directly into the network architectures. This flow-centric approach yields principled and stable transformations, enabling accurate and structure-preserving morphing of both 2D images and 3D shapes. Extensive experiments across a range of applications - including face and image morphing, as well as Gaussian Splatting morphing - show that FLOWING achieves state-of-the-art morphing quality with faster convergence. Code and pretrained models are available at http://schardong.github.io/flowing.
Read the full abstract ▾
All authors Arthur Bizzi, Matias Grynberg, Vitor Matias, Daniel Perazzo, João Paulo Lima, Luiz Velho, Nuno Gonçalves, João Pereira, Guilherme Schardong, Tiago Novello
Computer Vision
PRNet: Original Information Is All You Have
PeiHuang Zheng, Yunlong Zhao, Zheng Cui et al. (+1 more)
Small object detection in aerial images suffers from severe information degradation during feature extraction due to limited pixel representations, where shallow spatial details fail to align effectively with semantic information, leading to frequent misses and false positives. Existing FPN-based methods attempt to mitigate these losses through post-processing enhancements, but the reconstructed details often deviate from the original image information, impeding their fusion with semantic content. To address this limitation, we propose PRNet, a real-time detection framework that prioritizes the preservation and efficient utilization of primitive shallow spatial features to enhance small object representations. PRNet achieves this via two modules:the Progressive Refinement Neck (PRN) for spatial-semantic
alignment through backbone reuse and iterative refinement, and the Enhanced SliceSamp (ESSamp) for preserving shallow information during downsampling via optimized rearrangement and convolution. Extensive experiments on the VisDrone, AI-TOD, and UAVDT
datasets demonstrate that PRNet outperforms state-of-the-art methods under comparable computational constraints, achieving superior accuracy-efficiency trade-offs.
Read the full abstract ▾
All authors PeiHuang Zheng, Yunlong Zhao, Zheng Cui, Yang Li