Computer Vision · Cross-listed from cs.RO
Adaptive Illumination Control for Robot Perception
Yash Turkar, Shekoufeh Sadeghi and Karthik Dantu
Robot perception under low light or high dynamic range is usually improved downstream - via more robust feature extraction, image enhancement, or closed-loop exposure control. However, all of these approaches are limited by the image captured these conditions. An alternate approach is to utilize a programmable onboard light that adds to ambient illumination and improves captured images. However, it is not straightforward to predict its impact on image formation. Illumination interacts nonlinearly with depth, surface reflectance, and scene geometry. It can both reveal structure and induce failure modes such as specular highlights and saturation. We introduce Lightning, a closed-loop illumination-control framework for visual SLAM that combines relighting, offline optimization, and imitation learning. This is performed in three stages. First, we train a Co-Located Illumination Decomposition (CLID) relighting model that decomposes a
robot observation into an ambient component and a light-contribution field. CLID enables physically consistent synthesis of the same scene under alternative light intensities and thereby creates dense multi-intensity training data without requiring us to repeatedly re-run trajectories. Second, using these synthesized candidates, we formulate an offline Optimal Intensity Schedule (OIS) problem that selects illumination levels over a sequence trading off SLAM-relevant image utility against power consumption and temporal smoothness. Third, we distill this ideal solution into a real-time controller through behavior cloning, producing an Illumination Control Policy (ILC) that generalizes beyond the initial training distribution and runs online on a mobile
robot to command discrete light-intensity levels. Across our evaluation, Lightning substantially improves SLAM trajectory robustness while reducing unnecessary illumination power.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Frequency-Enhanced Hilbert Scanning Mamba for Short-Term Arctic Sea Ice Concentration Prediction
Feng Gao, Zheng Gong, Wenli Liu et al. (+4 more)
While Mamba models offer efficient sequence modeling, vanilla versions struggle with temporal correlations and boundary details in Arctic sea ice concentration (SIC) prediction. To address these limitations, we propose Frequency-enhanced
Hilbert scanning Mamba Framework (FH-Mamba) for short-term Arctic SIC prediction. Specifically, we introduce a 3D
Hilbert scan mechanism that traverses the 3D spatiotemporal grid along a locality-preserving path, ensuring that adjacent indices in the flattened sequence correspond to neighboring voxels in both spatial and temporal dimensions. Additionally, we incorporate wavelet transform to amplify high-frequency details and we also design a Hybrid Shuffle
Attention module to adaptively aggregate sequence and frequency features. Experiments conducted on the OSI-450a1 and AMSR2
datasets demonstrate that our FH-Mamba achieves superior prediction performance compared with state-of-the-art baselines. The results confirm the effectiveness of
Hilbert scanning and frequency-aware
attention in improving both temporal consistency and edge reconstruction for Arctic SIC
forecasting. Our codes are publicly available at https://github.com/oucailab/FH-Mamba.
Read the full abstract ▾
All authors Feng Gao, Zheng Gong, Wenli Liu, Yanhai Gan, Zhuoran Zheng, Junyu Dong, Qian Du
Computer Vision
SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning
Jintao Zhang, Kai Jiang, Chendong Xiang et al. (+5 more)
Many training-free sparse
attention methods are effective for accelerating
diffusion models. Recently, several works suggest that making sparse
attention trainable can further increase sparsity while preserving generation quality. We study three key questions: (1) when do the two common masking rules, i.e., Top-k and Top-p, fail, and how can we avoid these failures? (2) why can trainable sparse
attention reach higher sparsity than training-free methods? (3) what are the limitations of fine-tuning sparse
attention using the
diffusion loss, and how can we address them? Based on this analysis, we propose SpargeAttention2, a trainable sparse
attention method that achieves high sparsity without degrading generation quality. SpargeAttention2 includes (i) a hybrid masking rule that combines Top-k and Top-p for more robust masking at high sparsity, (ii) an efficient trainable sparse
attention implementation, and (iii) a
distillation-inspired fine-tuning objective to better preserve generation quality during fine-tuning using sparse
attention. Experiments on
video diffusion models show that SpargeAttention2 reaches 95%
attention sparsity and a 16.2x
attention speedup while maintaining generation quality, consistently outperforming prior sparse
attention methods.
Read the full abstract ▾
All authors Jintao Zhang, Kai Jiang, Chendong Xiang, Weiqi Feng, Yuezhou Hu, Haocheng Xi, Jianfei Chen, Jun Zhu
Computer Vision
Benchmarking Video Foundation Models for Remote Parkinson's Disease Screening
Md Saiful Islam, Ekram Hossain, Abdelrahman Abdelkader et al. (+11 more)
Video-based assessments offer a scalable pathway for remote Parkinson's disease (PD) screening. While traditional approaches rely on handcrafted features mimicking clinical scales, recent advances in
video foundation models (VFMs) enable representation learning without task-specific customization. However, the comparative effectiveness of different VFM architectures across diverse clinical tasks remains poorly understood. We present a large-scale systematic study using a novel
video dataset from 1,888 participants (727 with PD), comprising 32,847
videos across 16 standardized clinical tasks. We evaluate seven state-of-the-art VFMs – including VideoPrism, V-JEPA, ViViT, and VideoMAE – to determine their robustness in clinical screening. By evaluating frozen
embeddings with a linear classification head, we demonstrate that task saliency is highly model-dependent: VideoPrism excels in capturing visual
speech kinematics (no audio) and facial expressivity, while V-JEPA proves superior for upper-limb motor tasks. Notably, TimeSformer remains highly competitive for rhythmic tasks like finger tapping. Our experiments yield AUCs of 76.4 - 85.3% and accuracies of 71.5 - 80.6%. While high specificity (up to 90.3%) suggests strong potential for ruling out healthy individuals, the lower sensitivity (43.2 - 57.3%) highlights the need for task-aware calibration and integration of multiple tasks and modalities. Overall, this work establishes a rigorous baseline for VFM-based PD screening and provides a roadmap for selecting suitable tasks and architectures in remote neurological monitoring. Code and anonymized structured data are publicly available: https://anonymous.4open.science/r/parkinson\_
video\_benchmarking-A2C5
Read the full abstract ▾
All authors Md Saiful Islam, Ekram Hossain, Abdelrahman Abdelkader, Tariq Adnan, Fazla Rabbi Mashrur, Sooyong Park, Praveen Kumar, Qasim Sudais, Natalia Chunga, Nami Shah, Jan Freyberg, Christopher Kanan, Ruth Schneider, Ehsan Hoque
Computer Vision
GLIMPSE : Real-Time Text Recognition and Contextual Understanding for VQA in Wearables
Akhil Ramachandran, Ankit Arun, Ashish Shenoy et al. (+8 more)
Video Large
Language Models (
Video LLMs) have shown remarkable progress in understanding and
reasoning about visual content, particularly in tasks involving text recognition and text-based visual question answering (Text VQA). However, deploying Text VQA on wearable devices faces a fundamental tension: text recognition requires high-resolution
video, but streaming high-quality
video drains battery and causes thermal throttling. Moreover, existing models struggle to maintain coherent temporal context when processing text across multiple frames in real-time streams. We observe that text recognition and visual
reasoning have asymmetric resolution requirements - OCR needs fine detail while scene understanding tolerates coarse features. We exploit this asymmetry with a hybrid architecture that performs selective high-resolution OCR on-device while streaming low-resolution
video for visual context. On a
benchmark of text-based VQA samples across five task categories, our system achieves 72% accuracy at 0.49x the power consumption of full-resolution streaming, enabling sustained VQA sessions on resource-constrained wearables without sacrificing text understanding quality.
Read the full abstract ▾
All authors Akhil Ramachandran, Ankit Arun, Ashish Shenoy, Abhay Harpale, Srihari Jayakumar, Debojeet Chatterjee, Mohsen Moslehpour, Pierce Chuang, Yichao Lu, Vikas Bhardwaj, Peyman Heidari
Computer Vision · Cross-listed from cs.RO
FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation
Huajian Zeng, Lingyun Chen, Jiaqi Yang et al. (+4 more)
Recent
vision-
language-action (VLA) models can generate plausible end-effector motions, yet they often fail in long-horizon, contact-rich tasks because the underlying hand-object interaction (HOI) structure is not explicitly represented. An embodiment-agnostic interaction representation that captures this structure would make manipulation behaviors easier to validate and transfer across
robots. We propose FlowHOI, a two-stage flow-matching framework that generates semantically grounded, temporally coherent HOI sequences, comprising hand poses, object poses, and hand-object contact states, conditioned on an
egocentric observation, a
language instruction, and a 3D Gaussian splatting (3DGS) scene reconstruction. We decouple geometry-centric grasping from semantics-centric manipulation, conditioning the latter on compact 3D scene
tokens and employing a motion-text
alignment loss to semantically ground the generated interactions in both the physical scene layout and the
language instruction. To address the scarcity of high-fidelity HOI supervision, we introduce a reconstruction pipeline that recovers aligned hand-object trajectories and meshes from large-scale
egocentric videos, yielding an HOI prior for robust generation. Across the GRAB and HOT3D
benchmarks, FlowHOI achieves the highest action recognition accuracy and a 1.7$\times$ higher physics simulation success rate than the strongest
diffusion-based baseline, while delivering a 40$\times$ inference speedup. We further demonstrate real-
robot execution on four dexterous manipulation tasks, illustrating the feasibility of retargeting generated HOI representations to real-
robot execution pipelines.
Read the full abstract ▾
All authors Huajian Zeng, Lingyun Chen, Jiaqi Yang, Yuantai Zhang, Fan Shi, Peidong Liu, Xingxing Zuo
Computer Vision
Learning on the Fly: Replay-Based Continual Object Perception for Indoor Drones
Sebastian-Ion Nae, Mihai-Eugen Barbu, Sebastian Mocanu et al. (+1 more)
Autonomous
agents such as indoor drones must learn new object classes in real-time while limiting catastrophic forgetting, motivating Class-Incremental Learning (CIL). However, most unmanned aerial vehicle (
UAV)
datasets focus on outdoor scenes and offer limited temporally coherent indoor
videos. We introduce an indoor
dataset of $14,400$ frames capturing inter-drone and ground vehicle footage, annotated via a semi-automatic workflow with a $98.6\%$ first-pass labeling agreement before final manual verification. Using this
dataset, we
benchmark 3 replay-based CIL strategies: Experience Replay (ER), Maximally Interfered
Retrieval (MIR), and Forgetting-Aware Replay (FAR), using YOLOv11-nano as a resource-efficient detector for deployment-constrained
UAV platforms. Under tight
memory budgets ($5-10\%$ replay), FAR performs better than the rest, achieving an average accuracy (ACC, $mAP_{50-95}$ across increments) of $82.96\%$ with $5\%$ replay. Gradient-weighted class activation mapping (Grad-CAM) analysis shows
attention shifts across classes in mixed scenes, which is associated with reduced localization quality for drones. The experiments further demonstrate that replay-based continual learning can be effectively applied to edge aerial systems. Overall, this work contributes an indoor
UAV video dataset with preserved temporal coherence and an evaluation of replay-based CIL under limited replay budgets. Project page: https://spacetime-
vision-robotics-laboratory.github.io/learning-on-the-fly-cl
Read the full abstract ▾
All authors Sebastian-Ion Nae, Mihai-Eugen Barbu, Sebastian Mocanu, Marius Leordeanu
Computer Vision
Handling Supervision Scarcity in Chest X-ray Classification: Long-Tailed and Zero-Shot Learning
Ha-Hieu Pham, Hai-Dang Nguyen, Thanh-Huy Nguyen et al. (+4 more)
Chest X-Ray (CXR) classification in clinical practice is often limited by imperfect supervision, arising from (i) extreme long-tailed multi-label disease distributions and (ii) missing annotations for rare or previously unseen findings. The CXR-LT 2026 challenge addresses these issues on a PadChest-based
benchmark with a 36-class label space split into 30 in-distribution classes for training and 6 out-of-distribution (OOD) classes for zero-shot evaluation. We present task-specific solutions tailored to the distinct supervision regimes. For Task 1 (long-tailed multi-label classification), we adopt an imbalance-aware multi-label learning strategy to improve recognition of tail classes while maintaining stable performance on frequent findings. For Task 2 (zero-shot OOD recognition), we propose a prediction approach that produces scores for unseen disease categories without using any supervised labels or examples from the OOD classes during training. Evaluated with macro-averaged mean Average Precision (mAP), our method achieves strong performance on both tasks, ranking first on the public leaderboard of the development phase. Code and pre-trained models are available at https://github.com/hieuphamha19/CXR_LT.
Read the full abstract ▾
All authors Ha-Hieu Pham, Hai-Dang Nguyen, Thanh-Huy Nguyen, Min Xu, Ulas Bagci, Trung-Nghia Le, Huy-Hieu Pham
Computer Vision · Cross-listed from eess.IV
FUTON: Fourier Tensor Networks for Implicit Neural Representations
Pooya Ashtari, Pourya Behmandpoor, Nikos Deligiannis et al. (+1 more)
Implicit neural representations (INRs) have emerged as powerful tools for encoding signals, yet dominant MLP-based designs often suffer from slow convergence, and how their accuracy depends on their size is known only empirically. We introduce FUTON (
Fourier Tensor Network), which models signals as generalized
Fourier series whose coefficients are parameterized by a low-rank tensor decomposition, read out by a linear or MLP head. FUTON combines complementary inductive biases:
Fourier bases capture smoothness and periodicity, while the low-rank parameterization enforces low-dimensional spectral structure. Beyond universal approximation, we show that with a linear head, the best FUTON's error splits exactly into the truncation of the signal's series and the best low-rank error of its coefficients, which we bracket from the samples before training, with a FUTON that attains the upper bound. On image fitting, occupancy volumes, and novel view synthesis, against twelve INRs of matched size, FUTON achieves the best average quality, reaches the final quality of each task's strongest baseline 4-34x sooner than that baseline, infers fastest, and trains in the least
memory.
Read the full abstract ▾
All authors Pooya Ashtari, Pourya Behmandpoor, Nikos Deligiannis, Aleksandra Pizurica
Computer Vision
LAF-YOLOv10 with Partial Convolution Backbone, Attention-Guided Feature Pyramid, Auxiliary P2 Head, and Wise-IoU Loss for Small Object Detection in Drone Aerial Imagery
Sohail Ali Farooqui, Zuhair Ahmed Khan Taha, Mohammed Mudassir Uddin et al. (+1 more)
Unmanned aerial vehicles serve as primary sensing platforms for surveillance, traffic monitoring, and disaster response, making aerial object detection a central problem in applied computer
vision. Current detectors struggle with
UAV-specific challenges: targets spanning only a few pixels, cluttered backgrounds, heavy occlusion, and strict onboard computational budgets. This study introduces LAF-YOLOv10, built on YOLOv10n, integrating four complementary techniques to improve small-object detection in drone imagery. A Partial Convolution C2f (PC-C2f) module restricts spatial convolution to one quarter of backbone channels, reducing redundant computation while preserving discriminative capacity. An
Attention-Guided Feature Pyramid Network (AG-FPN) inserts Squeeze-and-Excitation channel gates before multi-scale fusion and replaces nearest-neighbor upsampling with DySample for content-aware interpolation. An auxiliary P2 detection head at 160$\times$160 resolution extends localization to objects below 8$\times$8 pixels, while the P5 head is removed to redistribute parameters. Wise-IoU v3 replaces CIoU for bounding box regression, attenuating gradients from noisy annotations in crowded aerial scenes. The four modules address non-overlapping bottlenecks: PC-C2f compresses backbone computation, AG-FPN refines cross-scale fusion, the P2 head recovers spatial resolution, and Wise-IoU stabilizes regression under label noise. No individual component is novel; the contribution is the joint integration within a single YOLOv10 framework. Across three training runs (seeds 42, 123, 256), LAF-YOLOv10 achieves 35.1$\pm$0.3% mAP@0.5 on VisDrone-DET2019 with 2.3\,M parameters, exceeding YOLOv10n by 3.3 points. Cross-
dataset evaluation on UAVDT yields 35.8$\pm$0.4% mAP@0.5.
Benchmarks on NVIDIA Jetson Orin Nano confirm 24.3 FPS at FP16, demonstrating viability for embedded
UAV deployment.
Read the full abstract ▾
All authors Sohail Ali Farooqui, Zuhair Ahmed Khan Taha, Mohammed Mudassir Uddin, Shahnawaz Alam
Computer Vision
An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generation
Giang Son Nguyen, Zi Pong Lim, Sarthak Ketanbhai Modi et al. (+2 more)
Vision-
Language Models (VLMs) are increasingly used in document processing pipelines to convert flowchart images into structured code (e.g., Mermaid). In production, these systems process arbitrary inputs for which no ground-truth code exists, making output quality difficult to assess. We propose a reference-free evaluation framework that monitors flowchart image-to-code generation quality at inference time, using only the input image and the generated output. The framework introduces two automated metrics: $\text{Recall}{\text{OCR}}$, which estimates content coverage by extracting text from the input image via OCR as a proxy reference, and $\text{Precision}{\text{VE}}$, which detects hallucinated elements through Visual Entailment against the original image. Their harmonic mean, $\text{F1}{\text{OCR-VE}}$, provides a unified quality score. Validation on the FlowVQA
dataset shows strong agreement with ground-truth metrics (average Pearson's $r = 0.97$, $0.91$, and $0.94$ for Recall, Precision, and F1, respectively), confirming the framework's reliability as a practical, reference-free alternative for continuous quality monitoring in production settings.
Read the full abstract ▾
All authors Giang Son Nguyen, Zi Pong Lim, Sarthak Ketanbhai Modi, Yon Shin Teo, Wenya Wang
Computer Vision
The Diffusion Duet: Harmonizing Dual Channels with Wavelet Suppression for Image Separation
Jingwei Li and Wei Pu
Blind image separation (BIS) refers to the inverse problem of simultaneously estimating and restoring multiple independent source images from a single observation image under conditions of unknown mixing mode and without prior knowledge of the source images. Traditional methods relying on statistical independence assumptions or CNN/GAN variants struggle to characterize complex feature distributions in real scenes, leading to estimation bias, texture distortion, and artifact residue under strong noise and nonlinear mixing. This paper innovatively introduces
diffusion models into dual-channel BIS, proposing an efficient Dual-Channel
Diffusion Separation Model (DCDSM). DCDSM leverages
diffusion models' powerful generative capability to learn source image feature distributions and reconstruct feature structures effectively. A novel Wavelet Suppression Module (WSM) is designed within the dual-branch reverse denoising process, forming an interactive separation network that enhances detail separation by exploiting the mutual coupling noise characteristic between source images. Extensive experiments on synthetic
datasets containing rain/snow and complex mixtures demonstrate that DCDSM achieves state-of-the-art performance: 1) In image restoration tasks, it obtains PSNR/SSIM values of 35.0023 dB/0.9549 and 29.8108 dB/0.9243 for rain and snow removal respectively, outperforming Histoformer and LDRCNet by 1.2570 dB/0.9272 dB (PSNR) and 0.0262/0.0289 (SSIM) on average; 2) For complex mixture separation, the restored dual-source images achieve average PSNR and SSIM of 25.0049 dB and 0.7997, surpassing comparative methods by 4.1249 dB and 0.0926. Both subjective and objective evaluations confirm DCDSM's superiority in addressing rain/snow residue removal and detail preservation challenges.
Read the full abstract ▾
Computer Vision
AdaCorrection: Adaptive Offset Cache Correction for Accurate Diffusion Transformers
Dong Liu, Yanxuan Yu, Ben Lengerich et al. (+1 more)
Diffusion Transformers (DiTs) achieve state-of-the-art performance in high-fidelity image and
video generation but suffer from expensive inference due to their iterative denoising structure. While prior methods accelerate sampling by caching intermediate features, they rely on static reuse schedules or coarse-grained heuristics, which often lead to temporal drift and cache misalignment that significantly degrade generation quality. We introduce AdaCorrection, an adaptive offset cache correction framework that maintains high generation fidelity while enabling efficient cache reuse across
Transformer layers during
diffusion inference. At each timestep, AdaCorrection estimates cache validity with lightweight spatio-temporal signals and adaptively blends cached and fresh activations. This correction is computed on-the-fly without additional supervision or retraining. Our approach achieves strong generation quality with minimal computational overhead, maintaining near-original FID while providing moderate acceleration. Experiments on image and
video diffusion benchmarks show that AdaCorrection consistently improves generation performance. Our code has been integrated into FastCache-xDiT at https://github.com/NoakLiu/FastCache-xDiT.
Read the full abstract ▾
All authors Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu
Computer Vision
Using Deep Learning to Generate Semantically Correct Hindi Captions
Wasim Akram Khan and Anil Kumar Vuppala
Automated image captioning using the content from the image is very appealing when done by harnessing the capability of computer
vision and natural
language processing. Extensive research has been done in the field with a major focus on the English
language which gives the scope for further developments in the same with consideration of popular foreign
languages. This research utilizes distinct models for translating the image caption into Hindi, the fourth most popular
language across the world. Exploring the multi-modal architectures this research comprises local visual features, global visual features,
attention mechanisms, and pre-trained models. Using google cloud translator on the image
dataset from Flickr8k, Hindi image descriptions have been generated. Pre-trained CNNs like VGG16, ResNet50, and Inception V3 helped in retrieving image characteristics, while the uni-directional and bi-directional techniques of text encoding are used for the text encoding process. An additional
Attention layer helps to generate a weight vector and, by multiplying it, combine image characteristics from each time step into a sentence-level feature vector. Bilingual evaluation understudy scores are used to compare the research outcome. Many experiments that serve as a baseline are done for the comparative analysis of the research. An image with a score of BLEU-1 is considered sufficient, whereas one with a score of BLEU-4 is considered to have fluid image captioning. For both BLEU scores, the
attention-based bidirectional LSTM with VGG16 produced the best results of 0.59 and 0.19 respectively. The experiments conclude that researchs ability to produce relevant, semantically accurate image captions in Hindi. The research accomplishes the goals and future research can be guided by this research model.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videos
Albert J. Zhai, Kuo-Hao Zeng, Jiasen Lu et al. (+3 more)
The ability to learn manipulation skills by watching
videos of humans has the potential to unlock a new source of highly scalable data for
robot learning. Here, we tackle prehensile manipulation, in which tasks involve grasping an object before performing various post-grasp motions. Human
videos offer strong signals for learning the post-grasp motions, but they are less useful for learning the prerequisite grasping behaviors, especially for
robots without human-like hands. A promising way forward is to use a modular policy design, leveraging a dedicated grasp generator to produce stable grasps. However, arbitrary stable grasps are often not task-compatible, hindering the
robot's ability to perform the desired downstream motion. To address this challenge, we present Perceive-Simulate-Imitate (PSI), a framework for training a modular manipulation policy using human
video motion data processed by paired grasp-trajectory filtering in simulation. This simulation step extends the trajectory data with grasp suitability labels, which allows for supervised learning of task-oriented grasping capabilities. We show through real-world experiments that our framework can be used to learn precise manipulation skills efficiently without any
robot data, resulting in significantly more robust performance than using a grasp generator naively.
Read the full abstract ▾
All authors Albert J. Zhai, Kuo-Hao Zeng, Jiasen Lu, Ali Farhadi, Shenlong Wang, Wei-Chiu Ma
Computer Vision
Conversational Image Segmentation: Grounding Abstract Concepts with Scalable Supervision
Aadarsh Sahoo and Georgia Gkioxari
Conversational image
segmentation grounds abstract, intent-driven concepts into pixel-accurate masks. Prior work on referring image grounding focuses on categorical and spatial queries (e.g., "left-most apple") and overlooks functional and physical
reasoning (e.g., "where can I safely store the knife?"). We address this gap and introduce Conversational Image
Segmentation (CIS) and ConverSeg, a
benchmark spanning entities, spatial relations, intent, affordances, functions, safety, and physical
reasoning. We also present ConverSeg-Net, which fuses strong
segmentation priors with
language understanding, and an AI-powered data engine that generates
prompt-mask pairs without human supervision. We show that current
language-guided
segmentation models are inadequate for CIS, while ConverSeg-Net trained on our data engine achieves significant gains on ConverSeg and maintains strong performance on existing
language-guided
segmentation benchmarks. Project webpage: https://glab-caltech.github.io/converseg/
Read the full abstract ▾
Computer Vision
CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling
Sayan Deb Sarkar, Rémi Pautrat, Ondrej Miksik et al. (+4 more)
Video Language Models (VideoLMs) enable AI systems to understand temporal dynamics in
videos. To fit within the maximum context window constraint, current methods use keyframe sampling which often misses both macro-level events and micro-level details due to the sparse temporal coverage. Furthermore, processing full images and their
tokens for each frame incurs substantial computational overhead. We address these limitations by leveraging
video codec primitives (specifically motion vectors and residuals) which natively encode
video redundancy and sparsity without requiring expensive full-image encoding for most frames. To this end, we introduce lightweight
transformer-based encoders that aggregate codec primitives and align their representations with image encoder
embeddings through a pre-training strategy that accelerates convergence during end-to-end fine-tuning. Our approach, CoPE-VideoLM, reduces the time-to-first-
token by up to 86% and
token usage by up to 93% compared to standard VideoLMs. Moreover, by varying the keyframe and codec primitive densities we maintain or exceed performance on 14 diverse
video understanding
benchmarks spanning general question answering, temporal and motion
reasoning, long-form understanding, and spatial scene understanding.
Read the full abstract ▾
All authors Sayan Deb Sarkar, Rémi Pautrat, Ondrej Miksik, Marc Pollefeys, Iro Armeni, Mahdi Rad, Mihai Dusmanu
Computer Vision
FlexAM: Flexible Appearance-Motion Decomposition for Versatile Video Generation Control
Mingzhi Sheng, Zekai Gu, Peng Li et al. (+4 more)
Effective and generalizable control in
video generation remains a significant challenge. While many methods rely on ambiguous or task-specific signals, we argue that a fundamental disentanglement of "appearance" and "motion" provides a more robust and scalable pathway. We propose FlexAM, a unified framework built upon a novel 3D control signal. This signal represents
video dynamics as a point cloud, introducing three key enhancements: multi-frequency positional encoding to distinguish fine-grained motion, depth-aware encoding, and a flexible control signal for balancing precision and generalization. This representation allows FlexAM to effectively disentangle appearance and motion, enabling a wide range of tasks including I2V/V2V editing, camera control, and spatial object editing. Extensive experiments demonstrate that FlexAM achieves superior performance across all evaluated tasks.
Read the full abstract ▾
All authors Mingzhi Sheng, Zekai Gu, Peng Li, Cheng Lin, Hao-Xiang Guo, Ying-Cong Chen, Yuan Liu
Computer Vision
Monocular Markerless Motion Capture Enables Quantitative Assessment of Upper Extremity Reachable Workspace
Seth Donahue, J.D. Peiffer, R. Tyler Richardson et al. (+7 more)
To validate a clinically accessible approach for quantifying the Upper Extremity Reachable Workspace (UERW) using a single (
monocular) camera and Artificial Intelligence (AI)-driven Markerless Motion Capture (MMC) for biomechanical analysis. Objective assessment and validation of these techniques for specific clinically oriented tasks are crucial for their adoption in clinical motion analysis. AI-driven
monocular MMC reduces the barriers to adoption in the clinic and has the potential to reduce the overhead for analysis of this common clinical assessment. Nine adult participants with no impairments performed the standardized UERW task, which entails reaching targets distributed across a virtual sphere centered on the torso, with targets displayed in a VR headset. Movements were simultaneously captured using a marker-based motion capture system and a set of eight FLIR cameras. We performed
monocular video analysis on two of these
video camera views to compare a frontal and offset camera configurations. The frontal camera orientation demonstrated strong agreement with the marker-based reference, exhibiting a minimal mean bias of $0.61 \pm 0.12$ % reachspace reached per octanct (mean $\pm$ standard deviation). In contrast, the offset camera view underestimated the percent workspace reached ($-5.66 \pm 0.45$ % reachspace reached). Conclusion: The findings support the feasibility of a frontal
monocular camera configuration for UERW assessment, particularly for anterior workspace evaluation where agreement with marker-based motion capture was highest. The overall performance demonstrates clinical potential for practical, single-camera assessments. This study provides the first validation of
monocular MMC system for the assessment of the UERW task. By reducing technical complexity, this approach enables broader implementation of quantitative upper extremity mobility assessment.
Read the full abstract ▾
All authors Seth Donahue, J.D. Peiffer, R. Tyler Richardson, Yishan Zhong, Shaun Q. Y. Tan, Benoit Marteau, Stephanie R. Russo, May D. Wang, R. James Cotton, Ross Chafetz
Computer Vision
LongStream: Long-Sequence Streaming Autoregressive Visual Geometry
Chong Cheng, Xianda Chen, Tao Xie et al. (+5 more)
Long-sequence streaming 3D reconstruction remains a significant open challenge. Existing autoregressive models often fail when processing long sequences because they anchor poses to the first frame, leading to
attention decay, scale drift, and extrapolation errors. We introduce LongStream, a novel gauge-decoupled streaming visual geometry model for metric-scale scene reconstruction across thousands of frames under a strictly online, future-invisible setting. Our approach is threefold. First, we discard the first-frame anchor and predict keyframe-relative poses. This reformulates long-range extrapolation into a constant-difficulty local task. Second, we introduce orthogonal scale learning. This method fully disentangles geometry from scale estimation to suppress drift. Finally, we identify
attention bias issues in
Transformers, including
attention-sink reliance and long-term KV-cache saturation. We propose cache-consistent training combined with periodic cache refresh. This approach suppresses
attention biases and contamination over ultra-long sequences and reduces the gap between training and inference. Experiments show that LongStream achieves state-of-the-art performance, enabling stable, metric-scale reconstruction over kilometer-scale sequences at 18 FPS. Project Page: https://3dagentworld.github.io/longstream/
Read the full abstract ▾
All authors Chong Cheng, Xianda Chen, Tao Xie, Wei Yin, Weiqiang Ren, Qian Zhang, Xiaoyang Guo, Hao Wang
Computer Vision
Realistic Face Reconstruction from Facial Embeddings via Diffusion Models
Dong Han, Yong Li and Joachim Denzler
With the advancement of face recognition (FR) systems,
privacy-preserving face recognition (PPFR) systems have gained popularity for their accurate recognition, enhanced facial
privacy protection, and robustness to various attacks. However, there are limited studies to further verify
privacy risks by reconstructing realistic high-resolution face images from
embeddings of these systems, especially for PPFR. In this work, we propose the face
embedding mapping (FEM), a general framework that explores Kolmogorov-Arnold Network (KAN) for conducting the
embedding-to-face attack by leveraging pre-trained Identity-Preserving
diffusion model against state-of-the-art (SOTA) FR and PPFR systems. Based on extensive experiments, we verify that reconstructed faces can be used for accessing other real-word FR systems. Besides, the proposed method shows the robustness in reconstructing faces from the partial and protected face
embeddings. Moreover, FEM can be utilized as a tool for evaluating safety of FR and PPFR systems in terms of
privacy leakage. All images used in this work are from public
datasets.
Read the full abstract ▾
Computer Vision
BAAF: Universal Transformation of One-Class Classifiers for Unsupervised Image Anomaly Detection
Declan McIntosh and Alexandra Branzan Albu
Detecting anomalies in images and
video is an essential task for multiple real-world problems, including industrial inspection, computer-assisted diagnosis, and environmental monitoring. Anomaly detection is typically formulated as a one-class classification problem, where the training data consists solely of nominal values, leaving methods built on this assumption susceptible to training label noise. We present Bootstrap Aggregation Anomaly Filtering (BAAF), a method that transforms an arbitrary one-class classifier-based anomaly detector into a fully unsupervised method. This is achieved by leveraging the unique intrinsic properties of anomaly detection: anomalies are uncommon in the sampled data and generally heterogeneous. These properties enable us to design a modified Bootstrap Aggregation method that uses multiple independently trained instances of supervised one-class classifiers to filter the training
dataset for anomalies. This transformation requires no modifications to the underlying anomaly detector; only the algorithmically selected data bags used for training change. We demonstrate empirically that our method can transform a wide variety of one-class classifier-based image anomaly detectors into unsupervised ones. Consequently, we present the first unsupervised logical anomaly detectors for images. We also demonstrate that our method achieves state-of-the-art performance in fully unsupervised anomaly detection on the MVTec AD, ViSA, and MVTec Loco AD
datasets. As improvements to one-class classifiers are made, our method directly transfers those improvements to the unsupervised domain, linking the domains.
Read the full abstract ▾