Computer Vision · Cross-listed from cs.LG
Spectral Evolution Search: Efficient Inference-Time Scaling for Reward-Aligned Image Generation
Jinyan Ye, Zhongjie Duan, Zhiwen Li et al. (+4 more)
Inference-time scaling offers a versatile paradigm for aligning visual generative models with downstream objectives without parameter updates. However, existing approaches that optimize the high-dimensional initial noise suffer from severe inefficiency, as many search directions exert negligible influence on the final generation. We show that this inefficiency is closely related to a spectral bias in generative dynamics: model sensitivity to initial perturbations diminishes rapidly as frequency increases. Building on this insight, we propose Spectral
Evolution Search (SES), a plug-and-play framework for initial noise optimization that executes gradient-free evolutionary search within a low-frequency subspace. Theoretically, we derive the Spectral Scaling Prediction from perturbation propagation dynamics, which explains the systematic differences in the impact of perturbations across frequencies. Extensive experiments demonstrate that SES significantly advances the Pareto frontier of generation quality versus computational cost, consistently outperforming strong baselines under equivalent budgets.
Read the full abstract ▾
All authors Jinyan Ye, Zhongjie Duan, Zhiwen Li, Cen Chen, Daoyuan Chen, Yaliang Li, Yingda Chen
Computer Vision · Cross-listed from cs.GR
WebSplatter: Enabling Cross-Device Efficient Gaussian Splatting in Web Browsers via WebGPU
Yudong Han, Chao Xu, Xiaodan Ye et al. (+3 more)
We present WebSplatter, an end-to-end GPU rendering pipeline for the heterogeneous web ecosystem. Unlike naive ports, WebSplatter introduces a wait-free hierarchical radix sort that circumvents the lack of global atomics in WebGPU, ensuring deterministic execution across diverse hardware. Furthermore, we propose an opacity-aware geometry culling stage that dynamically prunes splats before rasterization, significantly reducing overdraw and peak
memory footprint. Evaluation demonstrates that WebSplatter consistently achieves 1.2$\times$ to 4.5$\times$ speedups over state-of-the-art web viewers.
Read the full abstract ▾
All authors Yudong Han, Chao Xu, Xiaodan Ye, Weichen Bi, Zilong Dong, Yun Ma
Computer Vision
Hand3R: Online 4D Hand-Scene Reconstruction in the Wild
Wendi Hu, Haonan Zhou, Wenhao Hu et al. (+1 more)
For
Embodied AI, jointly reconstructing dynamic hands and the dense scene context is crucial for understanding physical interaction. However, most existing methods recover isolated hands in local coordinates, overlooking the surrounding 3D environment. To address this, we present Hand3R, the first online framework for joint 4D hand-scene reconstruction from
monocular video. Hand3R synergizes a pre-trained hand expert with a 4D scene foundation model via a scene-aware visual prompting mechanism. By injecting high-fidelity hand priors into a persistent scene
memory, our approach enables simultaneous reconstruction of accurate hand meshes and dense metric-scale scene geometry in a single forward pass. Experiments demonstrate that Hand3R bypasses the reliance on offline optimization and delivers competitive performance in both local hand reconstruction and global positioning.
Read the full abstract ▾
All authors Wendi Hu, Haonan Zhou, Wenhao Hu, Gaoang Wang
Computer Vision
From Single Scan to Sequential Consistency: A New Paradigm for LIDAR Relocalization
Minghang Zhu, Zhijing Wang, Yuxin Guo et al. (+3 more)
LiDAR relocalization aims to estimate the global 6-DoF pose of a sensor in the environment. However, existing regression-based approaches are prone to dynamic or ambiguous scenarios, as they either solely rely on single-frame inference or neglect the spatio-temporal consistency across scans. In this paper, we propose TempLoc, a new
LiDAR relocalization framework that enhances the robustness of localization by effectively modeling sequential consistency. Specifically, a Global Coordinate Estimation module is first introduced to predict point-wise global coordinates and associated uncertainties for each
LiDAR scan. A Prior Coordinate Generation module is then presented to estimate inter-frame point correspondences by the
attention mechanism. Lastly, an Uncertainty-Guided Coordinate Fusion module is deployed to integrate both predictions of point correspondence in an end-to-end fashion, yielding a more temporally consistent and accurate global 6-DoF pose. Experimental results on the NCLT and Oxford
Robot-Car
benchmarks show that our TempLoc outperforms stateof-the-art methods by a large margin, demonstrating the effectiveness of temporal-aware correspondence modeling in
LiDAR relocalization. Our code will be released soon.
Read the full abstract ▾
All authors Minghang Zhu, Zhijing Wang, Yuxin Guo, Wen Li, Sheng Ao, Cheng Wang
Computer Vision
LSGQuant: Layer-Sensitivity Guided Quantization for One-Step Diffusion Real-World Video Super-Resolution
Tianxing Wu, Zheng Chen, Cirou Xu et al. (+5 more)
One-Step
Diffusion Models have demonstrated promising capability and fast inference in
video super-resolution (VSR) for real-world. Nevertheless, the substantial model size and high computational cost of
Diffusion Transformers (DiTs) limit downstream applications. While low-bit quantization is a common approach for model compression, the effectiveness of quantized models is challenged by the high dynamic range of input latent and diverse layer behaviors. To deal with these challenges, we introduce LSGQuant, a layer-sensitivity guided quantizing approach for one-step
diffusion-based real-world VSR. Our method incorporates a Dynamic Range Adaptive Quantizer (DRAQ) to fit
video token activations. Furthermore, we estimate layer sensitivity and implement a Variance-Oriented Layer Training Strategy (VOLTS) by analyzing layer-wise statistics in calibration. We also introduce Quantization-Aware Optimization (QAO) to jointly refine the quantized branch and a retained high-precision branch. Extensive experiments demonstrate that our method has nearly performance to origin model with full-precision and significantly exceeds existing quantization techniques. Code is available at: https://github.com/zhengchen1999/LSGQuant.
Read the full abstract ▾
All authors Tianxing Wu, Zheng Chen, Cirou Xu, Bowen Chai, Yong Guo, Yutong Liu, Linghe Kong, Yulun Zhang
Computer Vision
BinaryDemoire: Moiré-Aware Binarization for Image Demoiréing
Zheng Chen, Zhi Yang, Xiaoyang Liu et al. (+5 more)
Image demoiréing aims to remove structured moiré artifacts in recaptured imagery, where degradations are highly frequency-dependent and vary across scales and directions. While recent deep networks achieve high-quality restoration, their full-precision designs remain costly for deployment. Binarization offers an extreme compression regime by quantizing both activations and weights to 1-bit. Yet, it has been rarely studied for demoiréing and performs poorly when naively applied. In this work, we propose BinaryDemoire, a binarized demoiréing framework that explicitly accommodates the frequency structure of moiré degradations. First, we introduce a moiré-aware binary gate (MABG) that extracts lightweight frequency descriptors together with activation statistics. It predicts channel-wise gating coefficients to condition the aggregation of binary convolution responses. Second, we design a shuffle-grouped residual adapter (SGRA) that performs structured sparse shortcut
alignment. It further integrates interleaved mixing to promote information exchange across different channel partitions. Extensive experiments on four
benchmarks demonstrate that the proposed BinaryDemoire surpasses current binarization methods. Code: https://github.com/zhengchen1999/BinaryDemoire.
Read the full abstract ▾
All authors Zheng Chen, Zhi Yang, Xiaoyang Liu, Weihang Zhang, Mengfan Wang, Yifan Fu, Linghe Kong, Yulun Zhang
Computer Vision
Human-in-the-loop Adaptation in Group Activity Feature Learning for Team Sports Video Retrieval
Chihiro Nakatani, Hiroaki Kawashima and Norimichi Ukita
This paper proposes human-in-the-loop adaptation for Group Activity Feature Learning (GAFL) without group activity annotations. This human-in-the-loop adaptation is employed in a group-activity
video retrieval framework to improve its
retrieval performance. Our method initially pre-trains the GAF space based on the similarity of group activities in a self-supervised manner, unlike prior work that classifies
videos into pre-defined group activity classes in a supervised learning manner. Our interactive fine-tuning process updates the GAF space to allow a user to better retrieve
videos similar to query
videos given by the user. In this fine-tuning, our proposed data-efficient
video selection process provides several
videos, which are selected from a
video database, to the user in order to manually label these
videos as positive or negative. These labeled
videos are used to update (i.e., fine-tune) the GAF space, so that the positive and negative
videos move closer to and farther away from the query
videos through contrastive learning. Our comprehensive experimental results on two team sports
datasets validate that our method significantly improves the
retrieval performance. Ablation studies also demonstrate that several components in our human-in-the-loop adaptation contribute to the improvement of the
retrieval performance. Code: https://github.com/chihina/GAFL-FINE-CVIU.
Read the full abstract ▾
Computer Vision
Fully Kolmogorov-Arnold Deep Model in Medical Image Segmentation
Xingyu Qiu, Xinghua Ma, Dong Liang et al. (+4 more)
Deeply stacked KANs are practically impossible due to high training difficulties and substantial
memory requirements. Consequently, existing studies can only incorporate few KAN layers, hindering the comprehensive exploration of KANs. This study overcomes these limitations and introduces the first fully KA-based deep model, demonstrating that KA-based layers can entirely replace traditional architectures in deep learning and achieve superior learning capacity. Specifically, (1) the proposed Share-activation KAN (SaKAN) reformulates Sprecher's variant of Kolmogorov-Arnold representation
theorem, which achieves better optimization due to its simplified parameterization and denser training samples, to ease training difficulty, (2) this paper indicates that spline gradients contribute negligibly to training while consuming huge GPU
memory, thus proposes the Grad-Free Spline to significantly reduce
memory usage and computational overhead. (3) Building on these two innovations, our ALL U-KAN is the first representative implementation of fully KA-based deep model, where the proposed KA and KAonv layers completely replace FC and Conv layers. Extensive evaluations on three medical image
segmentation tasks confirm the superiority of the full KA-based architecture compared to partial KA-based and traditional architectures, achieving all higher
segmentation accuracy. Compared to directly deeply stacked KAN, ALL U-KAN achieves 10 times reduction in parameter count and reduces
memory consumption by more than 20 times, unlocking the new explorations into deep KAN architectures.
Read the full abstract ▾
All authors Xingyu Qiu, Xinghua Ma, Dong Liang, Gongning Luo, Wei Wang, Kuanquan Wang, Shuo Li
Computer Vision
Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesis
Tianhe Wu, Ruibin Li, Lei Zhang et al. (+1 more)
Distribution matching
distillation (DMD) facilitates few-step image generation by aligning a distilled student with a reference multi-step teacher. In practice, however, optimizing DMD can reduce sample diversity in few-step synthesis, and existing remedies typically rely on perceptual or
adversarial regularization, leading to stability and scalability challenges during training. Here, we describe diversity-preserved DMD (DP-DMD), a role-separated
distillation method inspired by the complementary roles of early and late denoising steps. Specifically, the first
distillation step is trained with a teacher-derived target-prediction objective (e.g., v-prediction) to preserve sample diversity, while the remaining steps are optimized with the standard DMD loss to refine perceptual quality. DP-DMD, with no perceptual or
adversarial regularization, no additional modules, and no teacher-generated reference samples, preserves sample diversity while maintaining competitive visual quality under few-step sampling, providing a simple and stable alternative to other DMD variants.
Read the full abstract ▾
All authors Tianhe Wu, Ruibin Li, Lei Zhang, Kede Ma
Computer Vision
FSOD-VFM: Few-Shot Object Detection with Vision Foundation Models and Graph Diffusion
Chen-Bin Feng, Youyang Sha, Longfei Liu et al. (+4 more)
In this paper, we present FSOD-VFM: Few-Shot Object Detectors with
Vision Foundation Models, a framework that leverages
vision foundation models to tackle the challenge of few-shot object detection. FSOD-VFM integrates three key components: a universal proposal network (UPN) for category-agnostic bounding box generation, SAM2 for accurate mask extraction, and DINOv2 features for efficient adaptation to new object categories. Despite the strong generalization capabilities of foundation models, the bounding boxes generated by UPN often suffer from overfragmentation, covering only partial object regions and leading to numerous small, false-positive proposals rather than accurate, complete object detections. To address this issue, we introduce a novel graph-based confidence reweighting method. In our approach, predicted bounding boxes are modeled as nodes in a directed graph, with graph
diffusion operations applied to propagate confidence scores across the network. This reweighting process refines the scores of proposals, assigning higher confidence to whole objects and lower confidence to local, fragmented parts. This strategy improves detection granularity and effectively reduces the occurrence of false-positive bounding box proposals. Through extensive experiments on Pascal-5$^i$, COCO-20$^i$, and CD-FSOD
datasets, we demonstrate that our method substantially outperforms existing approaches, achieving superior performance without requiring additional training. Notably, on the challenging CD-FSOD
dataset, which spans multiple
datasets and domains, our FSOD-VFM achieves 31.6 AP in the 10-shot setting, substantially outperforming previous training-free methods that reach only 21.4 AP. Code is available at: https://intellindust-ai-lab.github.io/projects/FSOD-VFM.
Read the full abstract ▾
All authors Chen-Bin Feng, Youyang Sha, Longfei Liu, Yongjun Yu, Chi Man Vong, Xuanlong Yu, Xi Shen
Computer Vision
SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass
Chen Qian, Xinran Yu, Danyang Li et al. (+4 more)
Visual
token pruning is a promising approach for reducing the computational cost of
vision-
language models (VLMs), and existing methods often rely on early
pruning decisions to improve efficiency. While effective on coarse-grained
reasoning tasks, they suffer from significant performance degradation on tasks requiring fine-grained visual details. Through layer-wise analysis, we reveal substantial discrepancies in visual
token importance across layers, showing that
tokens deemed unimportant at shallow layers can later become highly relevant for text-conditioned
reasoning. To avoid irreversible critical information loss caused by premature
pruning, we introduce a new
pruning paradigm, termed bypass, which preserves unselected visual
tokens and forwards them to subsequent
pruning stages for re-evaluation. Building on this paradigm, we propose SwiftVLM, a simple and training-free method that performs
pruning at model-specific layers with strong visual
token selection capability, while enabling independent
pruning decisions across layers. Experiments across multiple VLMs and
benchmarks demonstrate that SwiftVLM consistently outperforms existing
pruning strategies, achieving superior accuracy-efficiency trade-offs and more faithful visual
token selection behavior.
Read the full abstract ▾
All authors Chen Qian, Xinran Yu, Danyang Li, Guoxuan Chi, Zheng Yang, Qiang Ma, Xin Miao
Computer Vision
Chenxi Zhang, Ziliang Gan, Liyun Zhu et al. (+3 more)
The financial domain poses substantial challenges for
vision-
language models (VLMs) due to specialized chart formats and knowledge-intensive
reasoning requirements. However, existing financial
benchmarks are largely single-turn and rely on a narrow set of question formats, limiting comprehensive evaluation in realistic application scenarios. To address this gap, we propose FinMTM, a multi-turn
multimodal benchmark that expands diversity along both data and task dimensions. On the data side, we curate and annotate 11{,}133 bilingual (Chinese and English) financial QA pairs grounded in financial visuals, including candlestick charts, statistical plots, and report figures. On the task side, FinMTM covers single- and multiple-choice questions, multi-turn open-ended dialogues, and
agent-based tasks. We further design task-specific evaluation protocols, including a set-overlap scoring rule for multiple-choice questions, a weighted combination of turn-level and session-level scores for multi-turn dialogues, and a composite metric that integrates planning quality with final outcomes for
agent tasks. Extensive experimental evaluation of 22 VLMs reveal their limitations in fine-grained visual perception, long-context
reasoning, and complex
agent workflows.
Read the full abstract ▾
All authors Chenxi Zhang, Ziliang Gan, Liyun Zhu, Youwei Pang, Qing Zhang, Rongjunchen Zhang
Computer Vision
Flexible Geometric Guidance for Probabilistic Human Pose Estimation with Diffusion Models
Francis Snelgar, Ming Xu, Stephen Gould et al. (+2 more)
3D human pose estimation from 2D images is a challenging problem due to depth ambiguity and occlusion. Because of these challenges the task is underdetermined, where there exists multiple – possibly infinite – poses that are plausible given the image. Despite this, many prior works assume the existence of a deterministic mapping and estimate a single pose given an image. Furthermore, methods based on machine learning require a large amount of paired 2D-3D data to train and suffer from generalization issues to unseen scenarios. To address both of these issues, we propose a framework for pose estimation using
diffusion models, which enables sampling from a probability distribution over plausible poses which are consistent with a 2D image. Our approach falls under the guidance framework for conditional generation, and guides samples from an unconditional
diffusion model, trained only on 3D data, using the gradients of the heatmaps from a 2D keypoint detector. We evaluate our method on the Human 3.6M
dataset under best-of-$m$ multiple hypothesis evaluation, showing state-of-the-art performance among methods which do not require paired 2D-3D data for training. We additionally evaluate the generalization ability using the MPI-INF-3DHP and 3DPW
datasets and demonstrate competitive performance. Finally, we demonstrate the flexibility of our framework by using it for novel tasks including pose generation and pose completion, without the need to train bespoke conditional models. We make code available at https://github.com/fsnelgar/
diffusion_pose .
Read the full abstract ▾
All authors Francis Snelgar, Ming Xu, Stephen Gould, Liang Zheng, Akshay Asthana
Computer Vision
Feature, Alignment, and Supervision in Category Learning: A Comparative Approach with Children and Neural Networks
Fanxiao Wani Qiu and Oscar Leong
Understanding how humans and machines learn from sparse data is central to cognitive science and machine learning. Using a species-fair design, we compare children and convolutional neural networks (CNNs) in a few-shot semi-supervised category learning task. Both learners are exposed to novel object categories under identical conditions. Learners receive mixtures of labeled and unlabeled exemplars while we vary supervision (1/3/6 labels), target feature (size, shape, pattern), and perceptual
alignment (high/low). We find that children generalize rapidly from minimal labels but show strong feature-specific biases and sensitivity to
alignment. CNNs show a different interaction profile: added supervision improves performance, but both
alignment and feature structure moderate the impact additional supervision has on learning. These results show that human-model comparisons must be drawn under the right conditions, emphasizing interactions among supervision, feature structure, and
alignment rather than overall accuracy.
Read the full abstract ▾
Computer Vision
Beyond Cropping and Rotation: Automated Evolution of Powerful Task-Specific Augmentations with Generative Models
Judah Goldfeder, Shreyes Kaliyur, Vaibhav Sourirajan et al. (+5 more)
Data augmentation has long been a cornerstone for reducing overfitting in
vision models, with methods like AutoAugment automating the design of task-specific augmentations. Recent advances in generative models, such as conditional
diffusion and few-shot NeRFs, offer a new paradigm for data augmentation by synthesizing data with significantly greater diversity and realism. However, unlike traditional augmentations like cropping or rotation, these methods introduce substantial changes that enhance robustness but also risk degrading performance if the augmentations are poorly matched to the task. In this work, we present EvoAug, an automated augmentation learning pipeline, which leverages these generative models alongside an efficient evolutionary
algorithm to learn optimal task-specific augmentations. Our pipeline introduces a novel approach to image augmentation that learns
stochastic augmentation trees that hierarchically compose augmentations, enabling more structured and adaptive transformations. We demonstrate strong performance across fine-grained classification and few-shot learning tasks. Notably, our pipeline discovers augmentations that align with domain knowledge, even in low-data settings. These results highlight the potential of learned generative augmentations, unlocking new possibilities for robust model training.
Read the full abstract ▾
All authors Judah Goldfeder, Shreyes Kaliyur, Vaibhav Sourirajan, Patrick Minwan Puma, Philippe Martin Wyder, Yuhang Hu, Jiong Lin, Hod Lipson
Computer Vision
Gromov Wasserstein Optimal Transport for Semantic Correspondences
Francis Snelgar, Stephen Gould, Ming Xu et al. (+2 more)
Establishing correspondences between image pairs is a long studied problem in computer
vision. With recent large-scale foundation models showing strong zero-shot performance on downstream tasks including classification and
segmentation, there has been interest in using the internal feature maps of these models for the semantic correspondence task. Recent works observe that features from DINOv2 and Stable
Diffusion (SD) are complementary, the former producing accurate but sparse correspondences, while the latter produces spatially consistent correspondences. As a result, current state-of-the-art methods for semantic correspondence involve combining features from both models in an ensemble. While the performance of these methods is impressive, they are computationally expensive, requiring evaluating feature maps from large-scale foundation models. In this work we take a different approach, instead replacing SD features with a superior matching
algorithm which is imbued with the desirable spatial consistency property. Specifically, we replace the standard nearest neighbours matching with an optimal transport
algorithm that includes a Gromov Wasserstein spatial smoothness prior. We show that we can significantly boost the performance of the DINOv2 baseline, and be competitive and sometimes surpassing state-of-the-art methods using Stable
Diffusion features, while being 5–10x more efficient. We make code available at https://github.com/fsnelgar/semantic_matching_gwot .
Read the full abstract ▾
All authors Francis Snelgar, Stephen Gould, Ming Xu, Liang Zheng, Akshay Asthana
Computer Vision · Cross-listed from cs.LG
Neural Predictor-Corrector: Solving Homotopy Problems with Reinforcement Learning
Jiayao Mai, Bangyan Liao, Zhenjun Zhao et al. (+6 more)
The Homotopy paradigm, a general principle for solving challenging problems, appears across diverse domains such as robust optimization, global optimization,
polynomial root-finding, and sampling. Practical solvers for these problems typically follow a predictor-corrector (PC) structure, but rely on hand-crafted heuristics for step sizes and iteration termination, which are often suboptimal and task-specific. To address this, we unify these problems under a single framework, which enables the design of a general neural solver. Building on this unified view, we propose Neural Predictor-Corrector (NPC), which replaces hand-crafted heuristics with automatically learned policies. NPC formulates policy selection as a sequential decision-making problem and leverages reinforcement learning to automatically discover efficient strategies. To further enhance generalization, we introduce an amortized training mechanism, enabling one-time offline training for a class of problems and efficient online inference on new instances. Experiments on four representative homotopy problems demonstrate that our method generalizes effectively to unseen instances. It consistently outperforms classical and specialized baselines in efficiency while demonstrating superior stability across tasks, highlighting the value of unifying homotopy methods into a single neural framework.
Read the full abstract ▾
All authors Jiayao Mai, Bangyan Liao, Zhenjun Zhao, Yingping Zeng, Haoang Li, Javier Civera, Tailin Wu, Yi Zhou, Peidong Liu
Computer Vision
A generalizable large-scale foundation model for musculoskeletal radiographs
Shinn Kim, Soobin Lee, Kyoungseob Shin et al. (+9 more)
Artificial intelligence (AI) has shown promise in detecting and characterizing musculoskeletal diseases from radiographs. However, most existing models remain task-specific, annotation-dependent, and limited in generalizability across diseases and anatomical regions. Although a generalizable foundation model trained on large-scale musculoskeletal radiographs is clinically needed, publicly available
datasets remain limited in size and lack sufficient diversity to enable training across a wide range of musculoskeletal conditions and anatomical sites. Here, we present SKELEX, a large-scale foundation model for musculoskeletal radiographs, trained using self-supervised learning on 1.2 million diverse, condition-rich images. The model was evaluated on 12 downstream diagnostic tasks and generally outperformed baselines in fracture detection, osteoarthritis grading, and bone tumor classification. Furthermore, SKELEX demonstrated zero-shot abnormality localization, producing error maps that identified pathologic regions without task-specific training. Building on this capability, we developed an interpretable, region-guided model for predicting bone tumors, which maintained robust performance on independent external
datasets and was deployed as a publicly accessible web application. Overall, SKELEX provides a scalable, label-efficient, and generalizable AI framework for musculoskeletal imaging, establishing a foundation for both clinical translation and data-efficient research in musculoskeletal radiology.
Read the full abstract ▾
All authors Shinn Kim, Soobin Lee, Kyoungseob Shin, Han-Soo Kim, Yongsung Kim, Minsu Kim, Juhong Nam, Somang Ko, Daeheon Kwon, Wook Huh, Ilkyu Han, Sunghoon Kwon
Computer Vision
Finding Optimal Video Moment without Training: Gaussian Boundary Optimization for Weakly Supervised Video Grounding
Sunoh Kim, Kimin Yun and Daeho Um
Weakly supervised temporal
video grounding aims to localize query-relevant segments in untrimmed
videos using only
video-sentence pairs, without requiring ground-truth segment annotations that specify exact temporal boundaries. Recent approaches tackle this task by utilizing Gaussian-based temporal proposals to represent query-relevant segments. However, their inference strategies rely on heuristic mappings from Gaussian parameters to segment boundaries, resulting in suboptimal localization performance. To address this issue, we propose Gaussian Boundary Optimization (GBO), a novel inference framework that predicts segment boundaries by solving a principled optimization problem that balances proposal coverage and segment compactness. We derive a closed-form solution for this problem and rigorously analyze the optimality conditions under varying penalty regimes. Beyond its theoretical foundations, GBO offers several practical advantages: it is training-free and compatible with both single-Gaussian and mixture-based proposal architectures. Our experiments show that GBO significantly improves localization, achieving state-of-the-art results across standard
benchmarks. Extensive experiments demonstrate the efficiency and generalizability of GBO across various proposal schemes. The code is available at https://github.com/sunoh-kim/gbo.
Read the full abstract ▾
Computer Vision
JRDB-Pose3D: A Multi-person 3D Human Pose and Shape Estimation Dataset for Robotics
Sandika Biswas, Kian Izadpanah and Hamid Rezatofighi
Real-world scenes are inherently crowded. Hence, estimating 3D poses of all nearby humans, tracking their movements over time, and understanding their activities within social and environmental contexts are essential for many applications, such as autonomous driving,
robot perception,
robot navigation, and human-
robot interaction. However, most existing 3D human pose estimation
datasets primarily focus on single-person scenes or are collected in controlled laboratory environments, which restricts their relevance to real-world applications. To bridge this gap, we introduce JRDB-Pose3D, which captures multi-human indoor and outdoor environments from a mobile robotic platform. JRDB-Pose3D provides rich 3D human pose annotations for such complex and dynamic scenes, including SMPL-based pose annotations with consistent body-shape parameters and track IDs for each individual over time. JRDB-Pose3D contains, on average, 5-10 human poses per frame, with some scenes featuring up to 35 individuals simultaneously. The proposed
dataset presents unique challenges, including frequent occlusions, truncated bodies, and out-of-frame body parts, which closely reflect real-world environments. Moreover, JRDB-Pose3D inherits all available annotations from the JRDB
dataset, such as 2D pose, information about social grouping, activities, and interactions, full-scene semantic masks with consistent human- and object-level tracking, and detailed annotations for each individual, such as age, gender, and race, making it a holistic
dataset for a wide range of downstream perception and human-centric understanding tasks.
Read the full abstract ▾
Computer Vision
IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning
Zhichao Sun, Yidong Ma, Gang Liu et al. (+4 more)
Large
Vision-
Language Models (LVLMs) achieve impressive performance across multiple tasks. A significant challenge, however, is their prohibitive inference cost when processing high-resolution visual inputs. While visual
token pruning has emerged as a promising solution, existing methods that primarily focus on semantic relevance often discard
tokens that are crucial for spatial
reasoning. We address this gap through a novel insight into how LVLMs process spatial
reasoning. Specifically, we reveal that LVLMs implicitly establish visual coordinate systems through Rotary Position
Embeddings (RoPE), where specific
token positions serve as implicit visual coordinates (IVC
tokens) that are essential for spatial
reasoning. Based on this insight, we propose IVC-Prune, a training-free,
prompt-aware
pruning strategy that retains both IVC
tokens and semantically relevant foreground
tokens. IVC
tokens are identified by theoretically analyzing the mathematical properties of RoPE, targeting positions at which its rotation
matrices approximate identity
matrix or the $90^\circ$ rotation
matrix. Foreground
tokens are identified through a robust two-stage process: semantic seed discovery followed by contextual refinement via value-vector similarity. Extensive evaluations across four representative LVLMs and twenty diverse
benchmarks show that IVC-Prune reduces visual
tokens by approximately 50% while maintaining $\geq$ 99% of the original performance and even achieving improvements on several
benchmarks. Source codes are available at https://github.com/FireRedTeam/IVC-Prune.
Read the full abstract ▾
All authors Zhichao Sun, Yidong Ma, Gang Liu, Yibo Chen, Xu Tang, Yao Hu, Yongchao Xu
Computer Vision · Cross-listed from cs.LG
SAFE-KD: Risk-Controlled Early-Exit Distillation for Vision Backbones
Salim Khazem
Early-exit networks reduce inference cost by allowing “easy” inputs to stop early, but practical deployment hinges on knowing when early exit is safe. We introduce SAFE-KD, a universal multi-exit wrapper for modern
vision backbones that couples hierarchical
distillation with conformal risk control. SAFE-KD attaches lightweight exit heads at intermediate depths, distills a strong teacher into all exits via Decoupled Knowledge
Distillation (DKD), and enforces deep-to-shallow consistency between exits. At inference, we calibrate per-exit stopping thresholds on a held-out set using conformal risk control (CRC) to guarantee a user-specified selective misclassification risk (among the samples that exit early) under exchangeability. Across multiple
datasets and architectures, SAFE-KD yields improved accuracy compute trade-offs, stronger calibration, and robust performance under corruption while providing finite-sample risk guarantees.
Read the full abstract ▾
Computer Vision
HP-GAN: Harnessing pretrained networks for GAN improvement with FakeTwins and discriminator consistency
Geonhui Son, Jeong Ryong Lee and Dosik Hwang
Generative
Adversarial Networks (GANs) have made significant progress in enhancing the quality of image synthesis. Recent methods frequently leverage pretrained networks to calculate perceptual losses or utilize pretrained feature spaces. In this paper, we extend the capabilities of pretrained networks by incorporating innovative self-supervised learning techniques and enforcing consistency between discriminators during GAN training. Our proposed method, named HP-GAN, effectively exploits neural network priors through two primary strategies: FakeTwins and discriminator consistency. FakeTwins leverages pretrained networks as encoders to compute a self-supervised loss and applies this through the generated images to train the generator, thereby enabling the generation of more diverse and high quality images. Additionally, we introduce a consistency mechanism between discriminators that evaluate feature maps extracted from Convolutional Neural Network (CNN) and
Vision Transformer (ViT) feature networks. Discriminator consistency promotes coherent learning among discriminators and enhances training robustness by aligning their assessments of image quality. Our extensive evaluation across seventeen
datasets-including scenarios with large, small, and limited data, and covering a variety of image domains-demonstrates that HP-GAN consistently outperforms current state-of-the-art methods in terms of Fréchet Inception Distance (FID), achieving significant improvements in image diversity and quality. Code is available at: https://github.com/higun2/HP-GAN.
Read the full abstract ▾
Computer Vision
Bongards at the Boundary of Perception and Reasoning: Programs or Language?
Cassidy Langenfeld, Claas Beger, Gloria Geng et al. (+4 more)
Vision-
Language Models (VLMs) have made great strides in everyday visual tasks, such as captioning a natural image, or answering commonsense questions about such images. But humans possess the puzzling ability to deploy their visual
reasoning abilities in radically new situations, a skill rigorously tested by the classic set of visual
reasoning challenges known as the Bongard problems. We present a neurosymbolic approach to solving these problems: given a hypothesized solution rule for a Bongard problem, we leverage LLMs to generate parameterized programmatic representations for the rule and perform parameter fitting using
Bayesian optimization. We evaluate our method on classifying Bongard problem images given the ground truth rule, as well as on solving the problems from scratch.
Read the full abstract ▾
All authors Cassidy Langenfeld, Claas Beger, Gloria Geng, Wasu Top Piriyakulkij, Keya Hu, Yewen Pu, Kevin Ellis
Computer Vision
MUSE: A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestration
Wenzhang Sun, Zhenyu Wang, Zhangchi Hu et al. (+3 more)
Generating long-form audio-visual stories from a short user
prompt remains challenging due to an intent-execution gap, where high-level narrative intent must be preserved across coherent, shot-level
multimodal generation over long horizons. Existing approaches typically rely on feed-forward pipelines or
prompt-only refinement, which often leads to semantic drift and identity inconsistency as sequences grow longer. We address this challenge by formulating storytelling as a closed-loop constraint enforcement problem and propose MUSE, a multi-
agent framework that coordinates generation through an iterative plan-execute-verify-revise loop. MUSE translates narrative intent into explicit, machine-executable controls over identity, spatial composition, and temporal continuity, and applies targeted
multimodal feedback to correct violations during generation. To evaluate open-ended storytelling without ground-truth references, we introduce MUSEBench, a reference-free evaluation protocol validated by human judgments. Experiments demonstrate that MUSE substantially improves long-horizon narrative coherence, cross-modal identity consistency, and cinematic quality compared with representative baselines.
Read the full abstract ▾
All authors Wenzhang Sun, Zhenyu Wang, Zhangchi Hu, Chunfeng Wang, Hao Li, Wei Chen
Computer Vision
A Vision-Based Analysis of Congestion Pricing in New York City
Mehmet Kerem Turkcan, Jhonatan Tavori, Javad Ghaderi et al. (+3 more)
We examine the impact of New York City's congestion pricing program through automated analysis of traffic camera data. Our computer
vision pipeline processes footage from over 900 cameras distributed throughout Manhattan and New York, comparing traffic patterns from November 2024 through the program's implementation in January 2025 until January 2026. We establish baseline traffic patterns and identify systematic changes in vehicle density across the monitored region.
Read the full abstract ▾
All authors Mehmet Kerem Turkcan, Jhonatan Tavori, Javad Ghaderi, Gil Zussman, Zoran Kostic, Andrew Smyth
Computer Vision
Thinking inside the Convolution for Image Inpainting: Reconstructing Texture via Structure under Global and Local Side
Haipeng Liu, Yang Wang, Biao Qian et al. (+2 more)
Image inpainting has earned substantial progress, owing to the encoder-and-decoder pipeline, which is benefited from the Convolutional Neural Networks (CNNs) with convolutional downsampling to inpaint the masked regions semantically from the known regions within the encoder, coupled with an upsampling process from the decoder for final inpainting output. Recent studies intuitively identify the high-frequency structure and low-frequency texture to be extracted by CNNs from the encoder, and subsequently for a desirable upsampling recovery. However, the existing arts inevitably overlook the information loss for both structure and texture feature maps during the convolutional downsampling process, hence suffer from a non-ideal upsampling output. In this paper, we systematically answer whether and how the structure and texture feature map can mutually help to alleviate the information loss during the convolutional downsampling. Given the structure and texture feature maps, we adopt the statistical normalization and denormalization strategy for the reconstruction guidance during the convolutional downsampling process. The extensive experimental results validate its advantages to the state-of-the-arts over the images from low-to-high resolutions including 256*256 and 512*512, especially holds by substituting all the encoders by ours. Our code is available at https://github.com/htyjers/ConvInpaint-TSGL
Read the full abstract ▾
All authors Haipeng Liu, Yang Wang, Biao Qian, Yong Rui, Meng Wang
Computer Vision
VOILA: Value-of-Information Guided Fidelity Selection for Cost-Aware Multimodal Question Answering
Rahul Atul Bhope, K. R. Jayaram, Vinod Muthusamy et al. (+3 more)
Despite significant costs from retrieving and processing high-fidelity visual inputs, most
multimodal vision-
language systems operate at fixed fidelity levels. We introduce VOILA, a framework for Value-Of-Information-driven adaptive fidelity selection in Visual Question Answering (VQA) that optimizes what information to retrieve before model execution. Given a query, VOILA uses a two-stage pipeline: a gradient-boosted regressor estimates correctness likelihood at each fidelity from question features alone, then an isotonic calibrator refines these probabilities for reliable decision-making. The system selects the minimum-cost fidelity maximizing expected utility given predicted accuracy and
retrieval costs. We evaluate VOILA across three deployment scenarios using five
datasets (VQA-v2, GQA, TextVQA, LoCoMo, FloodNet) and six
Vision-
Language Models (VLMs) with 7B-235B parameters. VOILA consistently achieves 50-60% cost reductions while retaining 90-95% of full-resolution accuracy across diverse query types and model architectures, demonstrating that pre-
retrieval fidelity selection is vital to optimize
multimodal inference under resource constraints.
Read the full abstract ▾
All authors Rahul Atul Bhope, K. R. Jayaram, Vinod Muthusamy, Ritesh Kumar, Vatche Isahagian, Nalini Venkatasubramanian
Computer Vision
Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation
Jiaze Li, Hao Yin, Haoran Xu et al. (+6 more)
Reinforcement learning has emerged as a principled post-training paradigm for Temporal
Video Grounding (TVG) due to its on-policy optimization, yet existing GRPO-based methods remain fundamentally constrained by sparse reward signals and substantial computational overhead. We propose
Video-OPD, an efficient post-training framework for TVG inspired by recent advances in on-policy
distillation.
Video-OPD optimizes trajectories sampled directly from the current policy, thereby preserving
alignment between training and inference distributions, while a frontier teacher supplies dense,
token-level supervision via a reverse KL divergence objective. This formulation preserves the on-policy property critical for mitigating distributional shift, while converting sparse, episode-level feedback into fine-grained, step-wise learning signals. Building on
Video-OPD, we introduce Teacher-Validated Disagreement Focusing (TVDF), a lightweight training curriculum that iteratively prioritizes trajectories that are both teacher-reliable and maximally informative for the student, thereby improving training efficiency. Empirical results demonstrate that
Video-OPD consistently outperforms GRPO while achieving substantially faster convergence and lower computational cost, establishing on-policy
distillation as an effective alternative to conventional reinforcement learning for TVG.
Read the full abstract ▾
All authors Jiaze Li, Hao Yin, Haoran Xu, Boshen Xu, Wenhui Tan, Zewen He, Jianzhong Ju, Zhenbo Luo, Jian Luan
Computer Vision
SharpTimeGS: Sharp and Stable Dynamic Gaussian Splatting via Lifespan Modulation
Zhanfeng Liao, Jiajun Zhang, Hanzhang Tu et al. (+4 more)
Novel view synthesis of dynamic scenes is fundamental to achieving photorealistic 4D reconstruction and immersive visual experiences. Recent progress in Gaussian-based representations has significantly improved real-time rendering quality, yet existing methods still struggle to maintain a balance between long-term static and short-term dynamic regions in both representation and optimization. To address this, we present SharpTimeGS, a lifespan-aware 4D Gaussian framework that achieves temporally adaptive modeling of both static and dynamic regions under a unified representation. Specifically, we introduce a learnable lifespan parameter that reformulates temporal visibility from a Gaussian-shaped decay into a flat-top profile, allowing primitives to remain consistently active over their intended duration and avoiding redundant densification. In addition, the learned lifespan modulates each primitives' motion, reducing drift in long-lived static points while retaining unrestricted motion for short-lived dynamic ones. This effectively decouples motion magnitude from temporal duration, improving long-term stability without compromising dynamic fidelity. Moreover, we design a lifespan-velocity-aware densification strategy that mitigates optimization imbalance between static and dynamic regions by allocating more capacity to regions with pronounced motion while keeping static areas compact and stable. Extensive experiments on multiple
benchmarks demonstrate that our method achieves state-of-the-art performance while supporting real-time rendering up to 4K resolution at 100 FPS on one RTX 4090.
Read the full abstract ▾
All authors Zhanfeng Liao, Jiajun Zhang, Hanzhang Tu, Zhixi Wang, Yunqi Gao, Hongwen Zhang, Yebin Liu