Computer Vision
Quality-Guided Semi-Supervised Learning for Medical Image Segmentation
Kumar Abhishek and Ghassan Hamarneh
Training accurate medical image
segmentation models requires large amounts of densely annotated data, which is costly and time-consuming to obtain. Semi-supervised learning (SSL) alleviates this by learning from both abundant unlabeled data and limited labeled data. However, most modern SSL methods rely on pseudolabels for unlabeled data, and typically assess their reliability through model confidence or uncertainty, measures that are self-referential and lack explicit grounding in
segmentation quality. Instead, we propose a quality-guided SSL framework that trains a dedicated network to estimate
segmentation quality from image-mask pairs. The predictor is trained on variable-quality masks generated through synthetic corruptions augmented with imperfect outputs from partially trained
segmentation models, capturing realistic error patterns encountered during training. We integrate the quality predictor into SSL through two complementary mechanisms: a quality-aware regularization loss and a quality-based pseudolabel sample reweighting scheme. We show that our method serves as a drop-in enhancement to existing SSL frameworks. Extensive experiments across five
datasets and multiple architectures demonstrate consistent improvements over competing SSL methods, advancing the state-of-the-art in semi-supervised medical image
segmentation.
Read the full abstract ▾
Computer Vision
Sensitivity as a Double-Edged Sword: A Trade-off Between Discriminability and Adversarial Robustness
Kai Wang
Modern neural networks are highly susceptible to
adversarial perturbations. In this work, we identify that part of this vulnerability stems from the sensitivity of the widely used fully connected (FC) classifiers to such perturbations. In contrast, simple $\ell_2$ distance-based classifiers exhibit significantly greater robustness. We provide thorough theoretical and empirical analysis showing that while FC classifiers' high sensitivity makes them discriminative, it also makes them vulnerable. Conversely, $\ell_2$-classifiers' insensitivity grants robustness but limits performance. Motivated by this trade-off, we propose a novel $\ell_2$-reclassifier based on a Hybrid Prototype Mixing (HPM) framework. This method retains the discriminative power of FC classifiers while leveraging the robustness of $\ell_2$ distance. It yields $\ell_2$-distance-based predictions by fusing two prototype types: (1) stable,
dataset-level prototypes updated via EMA, and (2) dynamic, batch-level prototypes generated from the FC classifier's predictions using a Straight-Through
Estimator (STE). However, this dynamic, STE-based architecture introduces significant challenges for evaluation, such as gradient obfuscation and forward discontinuity. To address this, we propose a new, rigorous evaluation protocol, the Mixed Surrogate Attack (MSA), which uses multiple surrogates along with powerful AutoAttack to ensure a fair and robust assessment. Extensive experiments demonstrate that our lightweight, plug-and-play module, with minimal fine-tuning, effectively enhances the
adversarial robustness of various existing SOTA adversarially trained models.
Read the full abstract ▾
Computer Vision
FlatVPR: Plug-and-play Geo-linear Residual Adapter for Geometric Rectification of Foundation Model Feature Manifolds
Rai Hisada and Kanji Tanaka
This paper proposes “FlatVPR,” a novel
geometric rectification paradigm that effectively bridges the trade-off between map lightweightness and localization accuracy in visual place recognition (VPR) by enforcing a feature
manifold structure where any descriptor between two adjacent anchors $\mathbf{z}_A$ and $\mathbf{z}_B$ can be accurately reconstructed via linear interpolation $\hat{\mathbf{z}}_{pseudo} = (1-t)\mathbf{z}_A + t\mathbf{z}_B$, where $t \in [0,1]$ denotes the relative position. While state-of-the-art foundation models such as DINOv2-ViT-S/14 provide robust semantic features, their latent
manifolds exhibit prominent
curvature, projecting uniform linear motion in physical space onto highly non-linear trajectories in the feature space, which hinders reliable reconstruction under sparse anchor conditions. To enable the aforementioned interpolation-based reconstruction, we introduce a residual transformation $\hat{\mathbf{z}} = \mathbf{z} + \text{Res}(\mathbf{z})$ to the raw foundation features $\mathbf{z}$, where $\text{Res}(\cdot)$ represents a learnable adapter. Our method explicitly suppresses
manifold curvature using a mathematically grounded Pullback Flatness Loss that minimizes the deviation of intermediate features from the linear segment connecting adjacent anchors, thereby minimizing the intrinsic
curvature of the
manifold. Through this spatial flattening, map construction is formulated within an Expectation-Maximization (EM) framework, decoupled into a continuous M-step for
manifold adaptation and a conceptual E-step for optimal anchor selection guidelines. Experiments on the NCLT
dataset demonstrate that the application of our adapter leads to significant performance improvements even under extremely sparse anchor conditions with 100m intervals and extreme seasonal changes.
Read the full abstract ▾
Computer Vision
Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference
Hyeonwoo Cho, Donghyeon Baek, Yewon Kim et al. (+1 more)
Recent advancements in
Multimodal Large
Language Models (MLLMs) have achieved remarkable success in
vision-
language tasks, yet the quadratic computational complexity arising from the vast number of visual
tokens incurs significant
memory and latency bottlenecks. While visual
token reduction (VTR) strategies have been explored to mitigate this burden, existing methods overlook the positional and attentional consistency between the full and reduced sequences, resulting in a distorted representation. To this end, we propose RESTORE, a novel VTR framework that rectifies the positional and attentional distortions while maintaining efficiency. Specifically, we present a simple yet effective calibration method that restores lost visual
attention by augmenting
attention weights based on relative distances. We also introduce a distinctive anchor selection for
token merging to mitigate information loss during feature averaging. Experimental results on multiple
benchmarks demonstrate that our method consistently improves the accuracy of various reduction methods, achieving state-of-the-art performance while maintaining computational efficiency. Project page is available at https://cvlab.yonsei.ac.kr/projects/RESTORE
Read the full abstract ▾
All authors Hyeonwoo Cho, Donghyeon Baek, Yewon Kim, Bumsub Ham
Computer Vision
Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs
Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie et al. (+1 more)
Vision-
Language models (VLMs), such as CLIP, achieve powerful zero-shot classification. However, their predictions remain sensitive to spurious correlations, where contextual cues dominate over semantic content. Earlier solutions typically rely on fine-tuning or
prompt engineering, which either undermine the advantages of pre-trained models or are prone to
hallucination. In this work, we propose Density-Aware Translation (DAT) that refines image-text similarity scores using a local
geometric density term derived from group reference sets. Our approach is motivated by the phenomenon that CLIP
embeddings exhibit a modality gap and lie on an anisotropic shell in the feature space: common patterns cluster near the mean, while rare patterns are pushed outward. This geometry creates uneven
alignment, where spurious correlations are amplified while semantically meaningful but rare cues are marginalised. To address this, we employ a relative measure to rescale similarities based on
embedding density, suppressing overconfident scores in diffuse regions while preserving dense, semantically consistent matches. Experimental results on
benchmark datasets demonstrate consistent improvements in worst-group and average accuracy, highlighting density-aware translation as a simple and effective calibration mechanism for reliable zero-shot classification using
multimodal models.
Read the full abstract ▾
All authors Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani
Computer Vision · Cross-listed from cs.SD
JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions
Jiashuo Yu, Yao Yao, Boyu Chen et al. (+1 more)
We address the challenge of generating high-fidelity, long-form soundtracks that remain coherent across scene transitions. Existing AI music systems are mainly designed for short, isolated clips and lack mechanisms to ensure narrative continuity. We present JenBridge, a modular and interpretable framework for adaptive long-form
video soundtracking that ensures both high-fidelity audio generation and transition naturalness. The core architecture is a
Transformer-based generative model trained with a flow-matching objective, following a two-stage paradigm: pretraining on large-scale text-audio corpora to establish robust musical priors, then adapting to the
video domain with dual text-visual conditioning for precise cross-modal
alignment. Crucially, to achieve long-form coherence across diverse scene changes, JenBridge incorporates a novel adaptive transition mechanism. This system features a versatile toolkit of transition styles, including a generative transition method, and uniquely employs a Large
Language Model (LLM)
Agent that acts as a director to select the most appropriate transition for each narrative shift intelligently. To rigorously assess this task, we propose the LVS
Benchmark, a new
benchmark that includes a curated
dataset and novel evaluation metrics focusing on holistic and transition-aware assessment. Extensive experiments on the proposed
benchmark demonstrate that JenBridge significantly outperforms existing methods in both objective and subjective metrics, particularly in terms of transition naturalness and overall narrative coherence. JenBridge represents a significant step towards fully automated, professional-quality
video soundtracking.
Read the full abstract ▾
All authors Jiashuo Yu, Yao Yao, Boyu Chen, Alex Wang
Computer Vision
Spatio-Temporal Correlation Guided Geometric Partitioning for Versatile Video Coding
Xuewei Meng, Chuanmin Jia, Xinfeng Zhang et al. (+2 more)
Geometric partitioning has attracted increasing
attention by its remarkable motion field description capability in the hybrid
video coding framework. However, the existing
geometric partitioning (GEO) scheme in Versatile
Video Coding (VVC) causes a non-negligible burden for signaling the side information. Consequently, the coding efficiency is limited. In view of this, we propose a spatio-temporal correlation guided
geometric partitioning (STGEO) scheme to efficiently describe the object information in the motion field of
video coding. The proposed method can economize the bits consumed for side information signaling, including the partitioning mode and motion information. We firstly analyze the characteristics of partitioning mode decision and motion vector selection in a statistically-sound way. Based on the observed spatio-temporal correlation, we design a mode prediction and coding method to reduce the overhead for representing the above mentioned side information. The main idea is to predict the STGEO modes and motion candidates that have higher selection possibilities, which can guide the
entropy coding, i.e., representing the predicted high-probability modes and motion candidates with fewer bits. In particular, the high-probability STGEO modes are predicted based on the edge information and history modes of adjacent STGEO-coded blocks. The corresponding motion information is represented by the index in a merge candidate list, which is adaptively inferred based on the off-line trained merge candidate selection probability. Simulation results show that the proposed approach achieves 0.95% and 1.98% bit-rate savings on average compared to VTM-8.0 without GEO for Random Access and Low-Delay B configurations, respectively.
Read the full abstract ▾
All authors Xuewei Meng, Chuanmin Jia, Xinfeng Zhang, Shanshe Wang, Siwei Ma
Computer Vision
MixerSENet: A Lightweight Framework for Efficient Hyperspectral Image Classification
Mohammed Q. Alkhatib, Swalpa Kumar Roy and Ali Jamali
In this paper, a novel framework, MixerSENet, is introduced for hyperspectral image (HSI) classification, designed to address the challenges of computational efficiency and limited labeled data. The proposed model processes hyperspectral image patches while maintaining consistent size and resolution throughout the network, effectively decoupling the mixing of spatial and channel dimensions. Notably, MixerSENet is lightweight and computationally efficient, requiring fewer parameters compared to traditional models, making it suitable for resource-constrained environments. A squeeze and excitation block is incorporated into the model to refine feature extraction, enhancing the network's ability to capture more informative features. Experimental results on two
benchmark datasets demonstrate that MixerSENet achieves superior performance, reaching an overall accuracy (OA) of 82.47% on Houston13
dataset and 96.70% on the Qingyun
dataset, outperforming state-of-the-art methods including 3D-CNN, HybridKAN, HSIFormer, SimPoolFormer, and MorphMamba. Furthermore, a detailed analysis of computational efficiency shows that MixerSENet achieves a favorable balance between accuracy and efficiency, with only 53,146 parameters and an low inference time, confirming its practicality for real-world applications. At publication, source code will be publicly available at https://github.com/mqalkhatib/MixerSENet.
Read the full abstract ▾
Computer Vision
Learning Label-Efficient Interpretable Medical Image Diagnosis via Semi-supervised Hypergraph Concept Bottleneck Model
Yijun Yang, Ruiqiang Xiao, Lijie Hu et al. (+4 more)
Deep learning has revolutionized medical image analysis, delivering exceptional diagnostic accuracy across diverse applications. Yet, the lack of interpretability in its decision-making hinders clinical adoption, particularly in high-stakes medical contexts where transparency is paramount for trustworthiness. For example, in Placenta Accreta Spectrum (PAS), subtle cues in ultrasound imaging challenge reliable diagnosis, rendering black-box models untrustworthy for accurate scoring. To address this, Concept Bottleneck Models (CBMs) offer a promising avenue by
embedding clinically meaningful intermediate concepts into the diagnosis pipeline, enabling clinicians to scrutinize and refine model outputs. However, conventional CBMs falter in capturing complex inter-concept dependencies and demand costly, expert-driven concept annotations, limiting their scalability. This study introduces a novel semi-supervised CBM framework designed for medical imaging, which leverages dual-level hypergraph learning to model high-order concept dependencies and generate domain-adaptive pseudo-labels. Our approach achieves superior interpretability and performance by integrating a concept-level hypergraph for enhanced
reasoning and an image-level hypergraph for robust pseudo-label generation. Experiments on a newly annotated PAS ultrasound
dataset and a breast ultrasound public
dataset demonstrate the effectiveness of the proposed concept label-efficient interpretable framework. Its universality is further validated on the dermoscopic image
dataset SkinCon. The code is available at https://github.com/scott-yjyang/HyperCBM.
Read the full abstract ▾
All authors Yijun Yang, Ruiqiang Xiao, Lijie Hu, Angelica I Aviles-Rivero, Yunzhu Wu, Jing Qin, Lei Zhu
Computer Vision
Understanding Identity Continuity in Thermal Video through Scene-Level Consistency
Wei-Chieh Sun, Gyungmin Ko, Heejae Kwon et al. (+2 more)
Thermal pedestrian MOT remains challenging because weak appearance cues and frequent detection interruptions cause severe trajectory fragmentation. We study whether lightweight post-processing can recover identity continuity without relying on heavy re-identification models or complex online association. Starting from a YOLOv8 and SORT baseline, we add a modular identity-repair backend consisting of online short-gap remapping and offline tracklet relinking based on temporal, spatial, motion, and border cues. Controlled ablations on a fixed validation split and evaluation on the official PBVS Thermal Pedestrian MOT
benchmark show that the main identity gains arise from conservative relinking, improving IDF1 from 82.25 to 84.93 while preserving MOTA, whereas many heuristic thresholds remain stable across broad operating ranges. These results suggest that, in low-information thermal imagery, robust identity recovery can be achieved more effectively through high-precision trajectory relinking than through increasing tracker complexity. These results provide a controlled analysis of identity recovery in thermal
video, showing that scene-level spatial-temporal consistency plays a dominant role in identity continuity compared to local frame-to-frame association.
Read the full abstract ▾
All authors Wei-Chieh Sun, Gyungmin Ko, Heejae Kwon, Hsiang-Wei Huang, Jenq-Neng Hwang
Computer Vision
RPCASSM: Robust PCA State Space Model For Infrared Small Target Detection
Pingping Liu, Aohua Li, Yubing Lu et al. (+3 more)
The detection and
segmentation of infrared small targets have important application significance in the fields of surveillance and security, maritime rescue and so on. Due to the low occupancy of these targets in long-distance imaging, the mainstream visual state space model is inefficient and difficult to accurately model the target edge. The existing infrared state space models do not deviate from the mainstream visual state space structure framework from the structural properties of infrared small targets. In order to solve this problem, this paper proposes the RPCASSM network based on the model paradigm of robust principal component analysis(RPCA), which aims to design the background state space module(BSSM) and the target state space module(TSSM) by the nature of the infrared small target in the spatial domain. The BSSM aims to use the saliency of spatial heterogeneous signals to design a spatial probe scanning mechanism(SPCM) to model background information. The TSSM designs a deformable
prompt scanning mechanism(DPCM) by using the sparsity and local highlight of the target to focus on the deformable space of the target for state space modeling. According to the above design, we effectively solve the problem that the existing mainstream
vision state space model is difficult to accurately model the edge structure of infrared small target. Experimental results on the existing
benchmark data sets prove the effectiveness of the RPCASSM design. Our code will be made public at \href{https://github.com/PepperCS/RPCASSM}{RPCASSM}.
Read the full abstract ▾
All authors Pingping Liu, Aohua Li, Yubing Lu, Jin Kuang, Tongshun Zhang, Qiuzhan Zhou
Computer Vision · Cross-listed from eess.SP
Physics-Aware Linearized ADMM and Its Unrolling
Satoshi Takabe, Shunta Arai and Tadashi Wadayama
Recently, partial differential equations (PDEs) have been used to directly model the measurement process in signal processing, although their evaluation is costly. In this paper, we propose a novel alternating direction method of multipliers (ADMM)-based
algorithm called physics-aware linearized ADMM (PA-LADMM) for inverse problems from PDE-based measurement processes. The key idea is the linearization of the subproblem with PDEs, leading to a cost-efficient update rule that calls only a PDE solver and its gradient evaluation per iteration. The
algorithm has a theoretical convergence guarantee under certain conditions. In addition, we combine it with deep unfolding to unroll the PA-LADMM and train its internal parameters using supervised data. Two distinct experiments, compressed sensing with optical fiber communication and image restoration from noisy anisotropic
diffusion, demonstrated the effectiveness of the proposed
algorithms.
Read the full abstract ▾
Computer Vision
Huayang Huang, Ruoyu Wang, Jinhui Zhao et al. (+5 more)
Generative
distillation significantly accelerates text-to-image (T2I) generation by compressing multi-step trajectories into few-step student models while preserving perceptual quality. However, existing methods primarily optimize efficiency and output fidelity, often neglecting critical properties of the original trajectory. In this work, we identify a key missing property: sensitivity to initial noise, whose degradation impairs downstream control methods relying on noise-based optimization and manipulation. We trace this issue to standard
distillation objectives that enforce pointwise output
alignment, inadvertently flattening the input-output landscape and suppressing the teacher's local
geometric structure. To address this, we propose Geometry-Aware
Distillation (GAD), a sensitivity-preserving framework that aligns the local functional behavior of teacher and student models. Specifically, GAD matches Jacobian-vector products with respect to input noise, enabling the student to reproduce the teacher's differential response to perturbations. Extensive experiments across multiple T2I paradigms and noise-driven control tasks demonstrate that GAD significantly restores sensitivity and improves diversity while maintaining high visual fidelity. Code is available at https://github.com/Hannah1102/GAD.
Read the full abstract ▾
All authors Huayang Huang, Ruoyu Wang, Jinhui Zhao, Wei Deng, Daiguo Zhou, Jian Luan, Yu Wu, Ye Zhu
Computer Vision
PhyScene3D: Physically Consistent Interactive 3D Tabletop Scene Generation
Weixing Chen, Zhuoqian Feng, Yang Liu et al. (+6 more)
Generating physically consistent 3D tabletop scenes is a fundamental yet underexplored problem for interactive and generalist robotic learning. The challenge stems from dense object hierarchies and irregular affordances. Here, an interactive scene denotes a physically valid, collision-free environment directly loadable into physics simulators. Existing methods, ranging from decoupled symbolic solvers to end-to-end regression models, often suffer from error propagation or overfitting to noisy supervision containing widespread physical violations. To address these limitations, we introduce PhyScene3D, a framework that reformulates generation as a Human-Mimetic Constructive Process. The proposed Cognitive Topological
Reasoning Chain (CTRC) factorizes scene synthesis into a sequential, anchor-conditioned process. It employs a 3D AABB-based placement scheme that imposes a strong structural inductive bias. To address imperfect supervision and physical infeasibility, we introduce Physics-Aware Denoising
Alignment (PADA). It integrates a differentiable Signed Distance Field (SDF) with Test-Time Optimization (TTO) to project generated scenes onto a physics-feasible
manifold while preserving semantic intent. Experiments demonstrate that PhyScene3D outperforms state-of-the-art approaches in both semantic accuracy and physical validity, achieving a 40% reduction in scene-wise collision rate relative to the human-annotated training data.
Read the full abstract ▾
All authors Weixing Chen, Zhuoqian Feng, Yang Liu, Yexin Zhang, Yifan Wen, Yinghong Liao, Weichao Qiu, Guanbin Li, Liang Lin
Computer Vision
Conditional Collapse in Sign Language Production: A Diagnostic and a Coverage Argument
Rui Hong and Jana Košecká
The goal of Sign
Language Production (SLP) is to translate natural
language text into corresponding sign
language movements performed by a virtual avatar. The quality of the generated motion is typically evaluated by a motion-space Frechet distance (FID) and back-translation BLEU score on
benchmarks such as How2Sign. The existing approaches can improve both metrics substantially while the underlying generator fails to represent the input sentence, a failure we call conditional collapse. In this work we propose to evaluate the generated motion at three independent levels: (tau1) initial-pose conditioning, (tau2) output diversity, and (tau3) target faithfulness. We compute these as pairwise-distance ratios using latent representations of a frozen motion autoencoder (MoAE), judged against each
dataset's own GT-vs-GT reference. We evaluate
diffusion and regression SLP models on How2Sign, including a re-implemented Neural Sign Actors, and show that tau3 faithfulness is never attained, while FID varies by nearly two orders of magnitude and is uncorrelated with faithfulness. We then apply the same models and diagnostic to Phoenix-2014T, CSL-Daily, and gloss-level ASL3DWord and show that faithfulness varies proportionally to how densely the training sentences cover the test conditions (conditioning coverage) and not corpus size. The smaller but densely covered Phoenix-2014T and ASL3DWord
datasets generate faithful sign motion, while CSL-Daily follows the input text yet falls short of sign
language faithfulness. Using the largest, open-domain How2Sign
dataset, the models collapse at every scale. We introduce two measurements of the text data that rank the corpora in the order of how densely the training sentences fill the
embedding space, and how close each test sentence lies to its nearest training sentence.
Read the full abstract ▾
Computer Vision
Edge-directed geometric partitioning for versatile video coding
Xuewei Meng, Xinfeng Zhang, Chuanmin Jia et al. (+3 more)
To improve the coding performance,
geometric partition (GEO) was proposed for the upcoming VVC standard. GEO provides 140 partition candidates. The index of optimal GEO mode needs to be signaled explicitly. Considering different structural characteristics of different CUs and the correlation between spatial adjacent blocks and temporal collocated blocks, we propose a GEO mode prediction strategy by constructing a Most Probable Mode (MPM) list to reduce the overhead of GEO index and improve coding efficiency. Based on the observation of the high correlation between the partition mode and object boundaries, an edge-directed
geometric partition scheme is proposed to construct the MPM list according to spatio-temporal edge information. The proposed method provides an objective BD-rate gain of 0.58% and 1.00% on average for RA and LDB configurations compared to VTM-6.0. Besides, it also promotes the visual quality of object boundaries.
Read the full abstract ▾
All authors Xuewei Meng, Xinfeng Zhang, Chuanmin Jia, Xia Li, Shanshe Wang, Siwei Ma
Computer Vision
CanonCGT: Reference-Based Color Grading via Canonical Pivot Representation
Jinwon Ko, Keunsoo Ko and Chang-Su Kim
Reference-based color grading aims to reproduce the tonal mood and lighting of a reference while preserving color harmony and scene structure. Existing photorealistic and filter-based methods often produce unstable tone mappings – over-shifting or inconsistently retaining colors – leading to unnatural results. We propose CanonCGT, a two-stage framework built on a canonical pivot – a style-neutral intermediate representation for stable color mapping. The first stage canonicalizes the input by removing intrinsic tonal bias, and the second color-grades it to match the reference style. A dual-phase training scheme, DP-CGT, combines supervised preset learning with self-supervised refinement on unpaired photographs. CanonCGT delivers photorealistic and tonally consistent results across diverse
datasets, surpassing state-of-the-art methods in stability and visual fidelity. Our codes are available at \href{https://github.com/Jinwon-Ko/CanonCGT}{https://github.com/Jinwon-Ko/CanonCGT}
Read the full abstract ▾
Computer Vision
Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
Pengyang Ling, Jiazi Bu, Yujie Zhou et al. (+6 more)
Group Relative Policy Optimization(GRPO) has emerged as an effective paradigm for aligning flow-based generative models with human preferences. However, the high cost of group rollouts forces existing methods to use very few denoising steps, resulting in sparse temporal supervision and leaving most intermediate stages without direct reward guidance. To address this, we propose Pave-GRPO, which reformulates the GRPO objective through principled average velocity decomposition. Rather than generating expensive high-step rollouts, we maintain efficient few-step group sampling but decompose each coarse transition into an equivalent ensemble of finer sub-trajectories spanning multiple intermediate timesteps, propagating reward feedback to a denser set of temporal stages for more comprehensive preference
alignment. Crucially, this incurs no additional
stochastic rollout generation or reward evaluation: the decomposed sub-trajectories are constructed analytically around the observed transitions, requiring only a small number of extra velocity-network evaluations during the policy update. This design offers two benefits: (i) rollout-free horizon expansion: through the direct reuse of few-step group samples and their associated rewards, Pave-GRPO significantly broadens the effective optimization scope under a fixed sampling and reward budget; and (ii) comprehensive temporal supervision: by equivalently decomposing an instantaneous velocity target into a multi-timestep ensemble, it distributes reward signals across more intermediate stages of the denoising process, enabling finer-grained and more thorough preference optimization.
Read the full abstract ▾
All authors Pengyang Ling, Jiazi Bu, Yujie Zhou, Yibin Wang, Zhenyu Hu, Zihan Zhang, Yi Jin, Huaian Chen, Yuhang Zang
Computer Vision
What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs
Abhishek Aich, Sparsh Garg, Vijay Kumar BG et al. (+2 more)
Driving
vision-
language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs), yet verification remains sparse: many slices are missing, making empirical failure rates unreliable. We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation that combines (i) an exposure-based coverage prior to prioritize rare, under-tested regions, and (ii) a neighbor-failure prior that propagates risk from similar tested conditions. SliceScorer is deliberately simple - interpretable, auditable, and conservative - properties essential for safety-critical validation. For stress testing beyond the declared ODD, we embed SliceScorer within SliceNav, an LLM-orchestrated verification pipeline where the model interprets developer queries to select relevant operators (triage, scoring, acquisition, evaluation) and vocabulary extensions, composing verification workflows while keeping all scoring deterministic and auditable. Experiments on three driving VLMs (WiseAD, DriveMM, Cosmos-Reason2-2B) show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods while maintaining diverse recommendations across the condition space. Ablations confirm both scoring components contribute, and qualitative analysis demonstrates end-to-end workflows from developer query to targeted evaluation.
Read the full abstract ▾
All authors Abhishek Aich, Sparsh Garg, Vijay Kumar BG, Turgun Yusuf Kashgari, Manmohan Chandraker
Computer Vision
Muyi Bao, Yuxin Cai, Hang Xu et al. (+7 more)
Vision-
language models (VLMs) have become a common foundation for
vision-and-
language navigation in continuous environments (VLN-CE). Yet most VLM-based methods cast
navigation as low-level action prediction, an interface that is ambiguous, tied to short-horizon motion primitives, and inefficient due to repeated VLM querying. We propose Goal2Pixel, a pure pixel-based paradigm that reformulates VLN-CE as navigable pixel grounding. Rather than predicting actions, Goal2Pixel uses the image plane as a unified spatial interface between VLM
reasoning and
robot motion: the model predicts a visible navigable pixel, which is back-projected into a 3D waypoint for forward
navigation. Non-forward actions are encoded by auxiliary left/right/bottom regions for turning and stopping. To support long-horizon
navigation, we propose ViKeyMem, a visibility-aware keyframe
memory, achieving a favorable performance-efficiency trade-off while reducing training cost by over 50%. To adapt pretrained VLMs to navigable pixel grounding, we introduce semantic
embeddings and coordinate-aware auxiliary losses. Among methods trained without external data, Goal2Pixel achieves competitive state-of-the-art performance on R2R-CE and RxR-CE. On R2R-CE Val-Unseen, it achieves 54.1% SR and 52.5% SPL with only 7.75 VLM calls per episode. Compared with direct action prediction, Goal2Pixel improves SR by 21.2 percentage points while requiring 6 times fewer VLM calls. Project Page: https://baobao0926.github.io/Goal2Pixel/.
Read the full abstract ▾
All authors Muyi Bao, Yuxin Cai, Hang Xu, Zongtai Li, Jinxi He, Jingfan Tang, Chen Lv, Ji Zhang, Yaqi Xie, Wenshan Wang
Computer Vision
Real-Time Generation of Streamable Talking Portrait Video with Reference-Guided Deep Compression VAEs
Sicheng Xu, Yu Deng, Shoukang Hu et al. (+5 more)
Video diffusion models have significantly advanced portrait
video generation, yet their high computational demands limit their use in interactive applications. This work presents a framework for streamable talking portrait
video generation conditioned on
speech audio and reference images. Designed meticulously for streaming scenarios, it features a causal
video VAE for deep latent compression and an autoregressive latent denoising model. Our causal VAE integrates a variable number of reference images as guidance, allowing the network to focus on dynamic information rather than static appearance, thereby enhancing compression efficacy and reconstruction quality. Additionally, we extend the residual auto-encoding paradigm to improve spatial-temporal causality handling in our VAE. The generator is based on a Rectified Flow
Transformer architecture and produces
video latents in a blockwise auto-regressive manner. Our method enables the real-time generation of high-quality talking portrait
videos, achieving speeds significantly faster than baseline models. Furthermore, comprehensive experiments demonstrate that it is on par with or even outperforms these large models in realism, vividness, and
video quality.
Read the full abstract ▾
All authors Sicheng Xu, Yu Deng, Shoukang Hu, Yichuan Wang, Yizhong Zhang, Zhan Chen, Jiaolong Yang, Baining Guo
Computer Vision
Turing Patterns for Multimedia: Reaction-Diffusion Multi-Modal Fusion for Language-Guided Video Moment Retrieval
Xiang Fang, Wanlong Fang, Wei Ji et al. (+1 more)
Video-
language models are pivotal for tasks such as moment
retrieval and highlight detection, yet they often struggle to capture the dynamic, non-linear interactions between temporal
video sequences and textual semantics. Existing approaches, relying on static cross-
attention or
prompt-tuning mechanisms, fail to adaptively model the evolving relationships between modalities, leading to suboptimal
alignment and limited generalization. Inspired by systems biology, we propose Reaction-
Diffusion Multimodal Fusion (RDMF), a novel framework that reimagines
video-
language alignment as a reaction-
diffusion (RD) process, drawing on the principles of pattern formation introduced by Alan Turing. In RDMF,
video features diffuse across time to capture temporal context, while text-
video interactions are modeled as non-linear reactions that amplify relevant features and suppress noise, forming emergent patterns akin to biological systems. Leveraging the Gray-Scott RD model, we design a computationally efficient fusion module that integrates
video and text representations, supported by rigorous mathematical analysis of stability and convergence using Turing instability criteria. Our framework is theoretically grounded, employing advanced mathematical tools to ensure stable pattern formation, and is practically viable, incorporating standard components like pretrained encoders and DETR-style heads for moment
retrieval and saliency prediction. RDMF represents a pioneering interdisciplinary approach, bridging systems biology and multimedia research to address the limitations of conventional
multimodal fusion. Preliminary experiments demonstrate its potential to outperform existing methods in identifying salient
video moments, offering a new paradigm for
video-
language tasks.
Read the full abstract ▾
All authors Xiang Fang, Wanlong Fang, Wei Ji, Tat-Seng Chua
Computer Vision
Self-Improving Small Object Grounding in LVLMs
Tianze Yang, Yucheng Shi, Ruitong Sun et al. (+2 more)
Can internal
attention patterns in Large
Vision Language Models (LVLMs) identify reliable small-object boxes without fine-tuning? In this work, we provide an affirmative answer.
Attention structure in LVLMs encodes grounding quality-a lightweight IoU regressor trained solely on
attention maps achieves strong IoU prediction (Pearson r > 0.67). This regressor powers the regressor-based variant of our
Attention-based Candidate Selection (ACS) framework, called ACS-Learned, which selects the best box from multiple sampled candidates to improve object grounding. By analyzing what the regressor learns, we reveal which
transformer layers and heads are most critical and derive ACS-Free: a training-free selector that ranks candidates by
attention entropy on these discriminative heads, with no learned component at inference. Experiments on COCO and Objects365 demonstrate up to 19% self-improvement on small object localization, with ACS-Free ranking best among all training-free methods, demonstrating that useful
attention structure improves both localization reliability and interpretability in LVLMs.
Read the full abstract ▾
All authors Tianze Yang, Yucheng Shi, Ruitong Sun, Ninghao Liu, Jin Sun
Computer Vision
Exploiting Semantic and Pixel Representations for Ultra-Low Bitrate Image Compression
Hao Wei, Yanhui Zhou, Chenyang Ge et al. (+2 more)
Most existing extreme compression methods fail to achieve an optimal rate-distortion-perception trade-off, as they typically prioritize perceptual fidelity and visual realism over pixel-level accuracy. Consequently, the resulting reconstructions often deviate noticeably from the originals. Ultra-low bitrate image compression is therefore crucial-not only for producing extremely compact representations but also for ensuring that reconstructed images remain semantically coherent and faithful to the source at the pixel level. To this end, we propose SPRDiff, a
diffusion-based compression method that fully leverages both semantic and pixel representations, thereby enhancing reconstruction fidelity under ultra-low bitrate constraints. Specifically, we develop a triple-encoder architecture that utilizes high-fidelity features from the pretrained distortion-oriented and semantic-oriented encoders to compensate for the limited representations extracted by the frozen VAE encoder, thereby improving latent compression and
entropy modeling. To further enhance the reconstruction fidelity of
diffusion models, we introduce a distortion-aware reconstruction module with dual feature extraction. This module not only generates a coarse reconstruction that preserves the main structures, but also provides practical and accurate semantic- and pixel-level conditional signals to guide the
diffusion model. Extensive experiments on
benchmark datasets demonstrate that our method outperforms state-of-the-art approaches in the rate-distortion-perception tradeoff at extremely low bitrates (below 0.03 bpp), effectively preserving both perceptual quality and pixel-wise fidelity in the reconstructed images. We will release the source code and trained models at https://github.com/cshw2021/SPRDiff.
Read the full abstract ▾
All authors Hao Wei, Yanhui Zhou, Chenyang Ge, Saeed Anwar, Ajmal Mian
Computer Vision
Paving the Way for Point Cloud Video Representation Learning Using A PDE Model
Zhuoxu Huang, Zhenkun Fan, Jungong Han et al. (+1 more)
Investigating spatial-temporal correlations, specifically how spatial points vary over time, is crucial for understanding point cloud
videos. Traditional methods, particularly flow-based techniques, struggle with these correlations due to the unordered spatial arrangement of sequential point cloud data. To address this challenge, we propose a novel approach that regularizes spatial-temporal correlation learning by formulating the problem as a solvable Partial Differential Equation (PDE). While PDEs have long been effective in the physical domain, their application to novel sequential data like point cloud
video remains underexplored. Inspired by fluid analysis, we construct a simplified PDE, and the process of solving PDE is guided and refined by a contrastive learning structure between the temporal
embeddings and the spatial
embeddings. With this extra supervision, our method, named MotionPDE, serves as an effective, plug-and-play enhancement module for existing backbone models, adding minimal computational overhead and parameters. Capitalizing on the contrastive learning process, we delve deeper into the self-supervised capabilities of MotionPDE, yielding promising results that underscore its utility and adaptability in point cloud
video data interpretation. The code repo with trained checkpoints will be available at https://github.com/zhh6425/motionpde.git for facilitating future research.
Read the full abstract ▾
All authors Zhuoxu Huang, Zhenkun Fan, Jungong Han, Josef Kittler
Computer Vision
EIVE: End-to-End Instance-Specific Visual Explanations for Detection Transformers
Jianlin Xiang, Yanshan Li and Linhui Dai
Visual explainability for object detection remains challenging due to the multi-instance nature of detection. Existing approaches predominantly adopt post-hoc paradigms, such as gradient-based or perturbation-based explanation methods, to interpret pretrained detectors. However, these methods require additional gradient computation or repeated model inference, resulting in limited efficiency. To address this issue, we propose an End-to-end Instance-specific Visual Explanation framework (EIVE) that directly generates instance-level saliency maps following the forward pass of Detection
Transformer (DETR)-like models. Specifically, we reformulate the cross-
attention mechanism in the decoder as an instance-level feature attribution pathway, so that the cross-
attention of each object query corresponds to the visual attribution of its predicted instance. Based on this formulation, we design a cross-layer hybrid consensus fusion (CLHCF) module to aggregate cross-
attention signals across decoder layers, producing stable and compact explanations. The explanation process of EIVE requires neither gradient computation nor input perturbation, yielding high computational efficiency, and applies to single- and multi-scale DETR-like object detectors. Finally, we present an
attention-aware joint training strategy (AAJTS) as a training-oriented application, which imposes spatial constraints on cross-
attention patterns to encourage stable and concentrated attribution representations, thereby improving both interpretability and detection performance. Experiments on MS COCO 2017, ExDark, and Cityscapes demonstrate that EIVE produces high-quality instance-level saliency maps and achieves performance comparable to, or better than, state-of-the-art post-hoc methods across standard metrics, while substantially improving explanation efficiency. Code is available at https://github.com/xjlDestiny/EIVE.git.
Read the full abstract ▾
Computer Vision
RoboTrustBench: Benchmarking the Trustworthiness of Video World Models for Robotic Manipulation
Huiqiong Li, Jiayu Wang, Zhiting Mei et al. (+3 more)
Video world models are increasingly used in robotic manipulation, yet existing
benchmarks mostly evaluate them under valid, feasible, and safe instructions. We introduce RoboTrustBench, a
benchmark for evaluating the trustworthiness of
video world models under four scenarios: Normal, Constraint-Sensitive, Counterfactual, and
Adversarial. Built from real-world DROID episodes, RoboTrustBench contains 1,207 expert-validated instruction-image pairs and a six-dimensional evaluation protocol with 13 fine-grained criteria. Evaluating seven representative
video world models with human and MLLM assessment, we find that current models often generate visually coherent
videos, but struggle with constraint
reasoning, counterfactual grounding, physical interaction, and unsafe-instruction suppression. These results show that visual quality and surface-level instruction following are insufficient for trustworthy robotic
video world modeling.
Read the full abstract ▾
All authors Huiqiong Li, Jiayu Wang, Zhiting Mei, Anirudha Majumdar, Jingjing Chen, Bin Zhu
Computer Vision
TLG: Temporal-Logic Grounding for Video Question Answering via Source-Annotation Reconstruction and Category-Targeted Reasoning
Ali Alavi
The TimeLogic Challenge evaluates formal temporal-logic
reasoning over
video - 16 operators (before, after, until, since, always, co-occur, ordering, ...) in boolean and 4-way multiple-choice form. End-to-end
video-
language models (VLMs) hover near chance on this task because they treat
video as a bag of frames and cannot localize when actions occur. We present TLG (Temporal-Logic Grounding), a three-tier system that (i) reconstructs each
video's action timeline from the public source-
dataset annotations the
benchmark was generated from, parses every question into a temporal-logic program, and executes it deterministically; (ii) falls back to a strong open VLM where no annotation exists; and (iii) routes only the question categories where the VLM is empirically weakest to a frontier
reasoning model. TLG raises test accuracy from a 46.9% VLM baseline to 71.37%, a +24.5 absolute gain, reaching within 3 points of the leaderboard top. We report extensive ablations, including three model-based timeline-reconstruction variants that all underperform a holistic VLM, isolating temporal grounding as the irreducible bottleneck and showing that real annotations - not larger models - drive accuracy.
Read the full abstract ▾
Computer Vision
Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis
Zhengfei Kuang, Adam Sun, Liyuan Zhu et al. (+7 more)
Modern vehicle platforms are equipped with a rich sensor suite, including
LiDAR, calibrated multi-camera rigs, and accurate ego-motion, that in principle offers strong signal for re-rendering a driving scene from novel viewpoints. A growing line of recent work leverages
video diffusion models for this task, using their generative priors to synthesize plausible novel views from sparse vehicle observations. In practice, however, existing methods exploit only a fragment of this signal, and their quality tends to degrade as the target trajectory departs from the recorded driving path. We argue that this is fundamentally a multi-sensor fusion problem: sparse
LiDAR reprojections supply accurate but incomplete metric geometry, surround-view reference imagery supplies dense appearance but no metric depth, and camera poses tie the two together across views. We introduce StreetNVS, a
video diffusion framework that jointly conditions on all three signals through a Reference-Enhanced Camera
Attention module based on a relative ray-level positional encoding. We develop a two-stage curriculum training strategy that gradually exposes the model to increasingly sparse
LiDAR. On the Waymo Open
Dataset, StreetNVS substantially outperforms state-of-the-art baselines under sparse
LiDAR conditioning, matches methods that rely on 10-100 times denser point clouds. We further show capabilities of synthesizing coherent
videos along extreme out-of-trajectory paths such as elevation, lane-shift, pullback, and rotation. Our website: https://streetnvs.github.io
Read the full abstract ▾
All authors Zhengfei Kuang, Adam Sun, Liyuan Zhu, Tong Wu, Shengqu Cai, Jonathan Tremblay, Iro Armeni, Ehsan Adeli, Lior Yariv, Gordon Wetzstein
Computer Vision
FLAME: Physics-Guided Neural Operators for Onboard Satellite Methane Detection in Hyperspectral Imagery
Junhyuk Heo, Junghwan Park, Sangcheol Sim et al. (+2 more)
Methane is a major driver of near-term
climate change, and rapidly identifying its emission sources is a critical
climate intervention. Spaceborne hyperspectral imagery is the primary tool for this task, but the volume of data produced by each sensor makes ground-based detection impractical and necessitates onboard detection. Classical methods incur prohibitive computational cost on onboard hardware, while deep learning models are fast but fall short on detection quality. We propose FLAME, a physics-guided neural operator that builds the physics of methane absorption directly into its architecture. On the methane detection
benchmark, FLAME achieves the highest detection accuracy among all evaluated methods, reduces the pixel-level false positive rate by nearly $3\times$ over the strongest neural baseline, uses the fewest parameters among learned baselines, and runs within the latency budget of onboard satellite hardware.
Read the full abstract ▾
All authors Junhyuk Heo, Junghwan Park, Sangcheol Sim, Beomkyu Choi, Woojin Cho