Computer Vision
SparVAR: Exploring Sparsity in Visual AutoRegressive Modeling for Training-Free Acceleration
Zekun Li, Ning Wang, Tongxin Bai et al. (+4 more)
Visual AutoRegressive (VAR) modeling has garnered significant
attention for its innovative next-scale prediction paradigm. However, mainstream VAR paradigms attend to all
tokens across historical scales at each autoregressive step. As the next scale resolution grows, the computational complexity of
attention increases quartically with resolution, causing substantial latency. Prior accelerations often skip high-resolution scales, which speeds up inference but discards high-frequency details and harms image quality. To address these problems, we present SparVAR, a training-free acceleration framework that exploits three properties of VAR
attention: (i) strong
attention sinks, (ii) cross-scale activation similarity, and (iii) pronounced locality. Specifically, we dynamically predict the sparse
attention pattern of later high-resolution scales from a sparse decision scale, and construct scale self-similar sparse
attention via an efficient index-mapping mechanism, enabling high-efficiency sparse
attention computation at large scales. Furthermore, we propose cross-scale local sparse
attention and implement an efficient block-wise sparse kernel, which achieves $\mathbf{> 5\times}$ faster forward speed than FlashAttention. Extensive experiments demonstrate that the proposed SparVAR can reduce the generation time of an 8B model producing $1024\times1024$ high-resolution images to the 1s, without skipping the last scales. Compared with the VAR baseline accelerated by FlashAttention, our method achieves a $\mathbf{1.57\times}$ speed-up while preserving almost all high-frequency details. When combined with existing scale-skipping strategies, SparVAR attains up to a $\mathbf{2.28\times}$ acceleration, while maintaining competitive visual generation quality. Code is available at \href{https://github.com/CAS-CLab/SparVAR}{SparVAR}.
Read the full abstract ▾
All authors Zekun Li, Ning Wang, Tongxin Bai, Changwang Mei, Peisong Wang, Shuang Qiu, Jian Cheng
Computer Vision
Jaehyun Kwak, Nam Cao, Boryeong Cho et al. (+3 more)
Targeted
adversarial attacks on Large
Vision-
Language Models (LVLMs) test whether small image perturbations can steer model responses toward attacker-specified content. Under the standard L-infinity constraint, targeted attacks become a regional perturbation budget allocation problem: attack success depends not only on the perturbation objective, but also on which regions receive updates and in what order. Existing localized attacks improve over global perturbations but rely on
stochastic spatial sampling, often updating weakly influential regions. We address this limitation through an
attention-based analysis showing that cross-modal
attention identifies adversarially sensitive regions and that perturbing high-
attention hotspots induces predictable redistribution toward subsequent salient regions. These findings motivate
attention-guided region sequencing, which begins from dominant hotspots and progressively moves the update support toward next-salient regions. Based on these principles, we propose Stage-wise
Attention-Guided Attack (SAGA), a black-box region-sequencing framework that uses a fixed
attention map from an open-source LVLM to guide perturbation updates without accessing target-model parameters, gradients, or
attention maps. Across ten closed-source and open-source LVLMs, SAGA achieves state-of-the-art attack success rates and the best overall imperceptibility. The source code is available at https://github.com/jaehyun-kwak/SAGA.
Read the full abstract ▾
All authors Jaehyun Kwak, Nam Cao, Boryeong Cho, Segyu Lee, Sumyeong Ahn, Se-Young Yun
Computer Vision
VecSet-Edit: Unleashing Pre-trained LRM for Mesh Editing from Single Image
Teng-Fang Hsiao, Bo-Kai Ruan, Yu-Lun Liu et al. (+1 more)
3D editing has emerged as a critical research area to provide users with flexible control over 3D assets. While current editing approaches predominantly focus on 3D Gaussian Splatting or multi-view images, the direct editing of 3D meshes remains underexplored. Prior attempts, such as VoxHammer, rely on voxel-based representations that suffer from limited resolution and necessitate labor-intensive 3D mask. To address these limitations, we propose VecSet-Edit, the first pipeline that leverages the high-fidelity VecSet Large Reconstruction Model (LRM) as a backbone for mesh editing. Our approach is grounded on a analysis of the spatial properties in VecSet
tokens, revealing that
token subsets govern distinct
geometric regions. Based on this insight, we introduce Mask-guided
Token Seeding and
Attention-aligned
Token Gating strategies to precisely localize target regions using only 2D image conditions. Also, considering the difference between VecSet
diffusion process versus voxel we design a Drift-aware
Token Pruning to reject
geometric outliers during the denoising process. Finally, our Detail-preserving Texture Baking module ensures that we not only preserve the
geometric details of original mesh but also the textural information. More details can be found in our project page: https://github.com/BlueDyee/VecSet-Edit/tree/main
Read the full abstract ▾
All authors Teng-Fang Hsiao, Bo-Kai Ruan, Yu-Lun Liu, Hong-Han Shuai
Computer Vision
Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perception
Sebastian Jung, Leonard Klüpfel, Rudolph Triebel et al. (+1 more)
We present Neural
Memory Object (NeMO), a novel object-centric representation that can be used to detect, segment and estimate the 6DoF pose of objects unseen during training using RGB images. Our method consists of an encoder that requires only a few RGB template views depicting an object to generate a sparse object-like point cloud using a learned UDF containing semantic and
geometric information. Next, a decoder takes the object encoding together with a query image to generate a variety of dense predictions. Through extensive experiments, we show that our method can be used for few-shot object perception without requiring any camera-specific parameters or retraining on target data. Our proposed concept of outsourcing object information in a NeMO and using a single network for multiple perception tasks enhances interaction with novel objects, improving scalability and efficiency by enabling quick object onboarding without retraining or extensive pre-processing. We report competitive and state-of-the-art results on various
datasets and perception tasks of the BOP
benchmark, demonstrating the versatility of our approach. https://github.com/DLR-RM/nemo
Read the full abstract ▾
All authors Sebastian Jung, Leonard Klüpfel, Rudolph Triebel, Maximilian Durner
Computer Vision
Explicit Uncertainty Modeling for Active CLIP Adaptation with Dual Prompt Tuning
Qian-Wei Wang, Yaguang Song and Shu-Tao Xia
Pre-trained
vision-
language models such as CLIP exhibit strong transferability, yet adapting them to downstream image classification tasks under limited annotation budgets remains challenging. In active learning settings, the model must select the most informative samples for annotation from a large pool of unlabeled data. Existing approaches typically estimate uncertainty via
entropy-based criteria or representation clustering, without explicitly modeling uncertainty from the model perspective. In this work, we propose a robust uncertainty modeling framework for active CLIP adaptation based on dual-
prompt tuning. We introduce two learnable
prompts in the textual branch of CLIP. The positive
prompt enhances the discriminability of task-specific textual
embeddings corresponding to light-weight tuned visual
embeddings, improving classification reliability. Meanwhile, the negative
prompt is trained in an reversed manner to explicitly model the probability that the predicted label is correct, providing a principled uncertainty signal for guiding active sample selection. Extensive experiments across different fine-tuning paradigms demonstrate that our method consistently outperforms existing active learning methods under the same annotation budget.
Read the full abstract ▾
Computer Vision
Fine-tuning Pre-trained Vision-Language Models in a Human-Annotation-Free Manner
Qian-Wei Wang, Guanghao Meng, Ren Cai et al. (+2 more)
Large-scale
vision-
language models (VLMs) such as CLIP exhibit strong zero-shot generalization, but adapting them to downstream tasks typically requires costly labeled data. Existing unsupervised self-training methods rely on pseudo-labeling, yet often suffer from unreliable confidence filtering, confirmation bias, and underutilization of low-confidence samples. We propose Collaborative Fine-Tuning (CoFT), an unsupervised adaptation framework that leverages unlabeled data through a dual-model, cross-modal collaboration mechanism. CoFT introduces a dual-
prompt learning strategy with positive and negative textual
prompts to explicitly model pseudo-label cleanliness in a sample-dependent manner, removing the need for hand-crafted thresholds or noise assumptions. The negative
prompt also regularizes lightweight visual adaptation modules, improving robustness under noisy supervision. CoFT employs a two-phase training scheme, transitioning from parameter-efficient fine-tuning on high-confidence samples to full fine-tuning guided by collaboratively filtered pseudo-labels. Building on CoFT, CoFT+ further enhances adaptation via iterative fine-tuning, momentum contrastive learning, and LLM-generated
prompts. Extensive experiments demonstrate consistent gains over existing unsupervised methods and even few-shot supervised baselines.
Read the full abstract ▾
All authors Qian-Wei Wang, Guanghao Meng, Ren Cai, Yaguang Song, Shu-Tao Xia
Computer Vision
Multiview Self-Representation Learning across Heterogeneous Views
Jie Chen, Zhu Wang, Chuanbin Liu et al. (+1 more)
Features of the same sample generated by different pretrained models often exhibit inherently distinct feature distributions because of discrepancies in the model pretraining objectives or architectures. Learning invariant representations from large-scale unlabeled visual data with various pretrained models in a fully unsupervised transfer manner remains a significant challenge. In this paper, we propose a multiview self-representation learning (MSRL) method in which invariant representations are learned by exploiting the self-representation property of features across heterogeneous views. The features are derived from large-scale unlabeled visual data through transfer learning with various pretrained models and are referred to as heterogeneous multiview data. An individual linear model is stacked on top of its corresponding frozen pretrained backbone. We introduce an information-passing mechanism that relies on self-representation learning to support feature aggregation over the outputs of the linear model. Moreover, an assignment probability distribution consistency scheme is presented to guide multiview self-representation learning by exploiting complementary information across different views. Consequently, representation invariance across different linear models is enforced through this scheme. In addition, we provide a theoretical analysis of the information-passing mechanism, the assignment probability distribution consistency and the incremental views. Extensive experiments with multiple
benchmark visual
datasets demonstrate that the proposed MSRL method consistently outperforms several state-of-the-art approaches.
Read the full abstract ▾
All authors Jie Chen, Zhu Wang, Chuanbin Liu, Xi Peng
Computer Vision
JOintGS: Joint Optimization of Cameras, Bodies and 3D Gaussians for In-the-Wild Monocular Reconstruction
Zihan Lou, Jinlong Fan, Sihan Ma et al. (+2 more)
Reconstructing high-fidelity animatable 3D human avatars from
monocular RGB
videos remains challenging, particularly in unconstrained in-the-wild scenarios where camera parameters and human poses from off-the-shelf methods (e.g., COLMAP, HMR2.0) are often inaccurate. Splatting (3DGS) advances demonstrate impressive rendering quality and real-time performance, they critically depend on precise camera calibration and pose annotations, limiting their applicability in real-world settings. We present JOintGS, a unified framework that jointly optimizes camera extrinsics, human poses, and 3D Gaussian representations from coarse initialization through a synergistic refinement mechanism. Our key insight is that explicit foreground-background disentanglement enables mutual reinforcement: static background Gaussians anchor camera estimation via multi-view consistency; refined cameras improve human body
alignment through accurate temporal correspondence; optimized human poses enhance scene reconstruction by removing dynamic artifacts from static constraints. We further introduce a temporal dynamics module to capture fine-grained pose-dependent deformations and a residual color field to model illumination variations. Extensive experiments on NeuMan and EMDB
datasets demonstrate that JOintGS achieves superior reconstruction quality, with 2.1~dB PSNR improvement over state-of-the-art methods on NeuMan
dataset, while maintaining real-time rendering. Notably, our method shows significantly enhanced robustness to noisy initialization compared to the baseline.Our source code is available at https://github.com/MiliLab/JOintGS.
Read the full abstract ▾
All authors Zihan Lou, Jinlong Fan, Sihan Ma, Yuxiang Yang, Jing Zhang
Computer Vision · Cross-listed from cs.RO
GeneralVLA: Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planning
Guoqing Ma, Siheng Wang, Zeyu Zhang et al. (+2 more)
Large foundation models have shown strong open-world generalization to complex problems in
vision and
language, but similar levels of generalization have yet to be achieved in robotics. One fundamental challenge is that the models exhibit limited zero-shot capability, which hampers their ability to generalize effectively to unseen scenarios. In this work, we propose GeneralVLA (Generalizable
Vision-
Language-Action Models with Knowledge-Guided Trajectory Planning), a hierarchical
vision-
language-action (VLA) model that can be more effective in utilizing the generalization of foundation models, enabling zero-shot manipulation and automatically generating data for robotics. In particular, we study a class of hierarchical VLA model where the high-level ASM (Affordance
Segmentation Module) is finetuned to perceive image keypoint affordances of the scene; the mid-level 3DAgent carries out task understanding, skill knowledge, and trajectory planning to produce a 3D path indicating the desired
robot end-effector trajectory. The intermediate 3D path prediction is then served as guidance to the low-level, 3D-aware control policy capable of precise manipulation. Compared to alternative approaches, our method requires no real-world robotic data collection or human demonstration, making it much more scalable to diverse tasks and viewpoints. Empirically, GeneralVLA successfully generates trajectories for 14 tasks, significantly outperforming state-of-the-art methods such as VoxPoser. The generated demonstrations can train more robust behavior cloning policies than training with human demonstrations or from data generated by VoxPoser, Scaling-up, and Code-As-Policies. We believe GeneralVLA can be the scalable method for both generating data for robotics and solving novel tasks in a zero-shot setting. Code: https://github.com/AIGeeksGroup/GeneralVLA. Website: https://aigeeksgroup.github.io/GeneralVLA.
Read the full abstract ▾
All authors Guoqing Ma, Siheng Wang, Zeyu Zhang, Shan Yu, Hao Tang
Computer Vision
Beyond Static Cropping: Layer-Adaptive Visual Localization and Decoding Enhancement
Zipeng Zhu, Zhanghao Hu, Qinglin Zhu et al. (+5 more)
Large
Vision-
Language Models (LVLMs) have advanced rapidly by aligning visual patches with the text
embedding space, but a fixed visual-
token budget forces images to be resized to a uniform pretraining resolution, often erasing fine-grained details and causing
hallucinations via over-reliance on
language priors. Recent
attention-guided enhancement (e.g., cropping or region-focused
attention allocation) alleviates this, yet it commonly hinges on a static "magic layer" empirically chosen on simple recognition
benchmarks and thus may not transfer to complex
reasoning tasks. In contrast to this static assumption, we propose a dynamic perspective on visual grounding. Through a layer-wise sensitivity analysis, we demonstrate that visual grounding is a dynamic process: while simple object recognition tasks rely on middle layers, complex visual search and
reasoning tasks require visual information to be reactivated at deeper layers. Based on this observation, we introduce Visual Activation by Query (VAQ), a metric that identifies the layer whose
attention map is most relevant to query-specific visual grounding by measuring
attention sensitivity to the input query. Building on VAQ, we further propose
LASER (Layer-adaptive
Attention-guided Selective visual and decoding Enhancement for
Reasoning), a training-free inference procedure that adaptively selects task-appropriate layers for visual localization and question answering. Experiments across diverse VQA
benchmarks show that
LASER significantly improves VQA accuracy across tasks with varying levels of complexity.
Read the full abstract ▾
All authors Zipeng Zhu, Zhanghao Hu, Qinglin Zhu, Yuxi Hong, Yijun Liu, Jingyong Su, Yulan He, Lin Gui
Computer Vision
Light Up Your Face: A Physically Consistent Dataset and Diffusion Model for Face Fill-Light Enhancement
Jue Gong, Zihan Zhou, Jingkai Wang et al. (+3 more)
Face fill-light enhancement (FFE) brightens underexposed faces by adding virtual fill light while keeping the original scene illumination and background unchanged. Most face relighting methods aim to reshape overall lighting, which can suppress the input illumination or modify the entire scene, leading to foreground-background inconsistency and mismatching practical FFE needs. To support scalable learning, we introduce LightYourFace-160K (LYF-160K), a large-scale paired
dataset built with a physically consistent renderer that injects a disk-shaped area fill light controlled by six disentangled factors, producing 160K before-and-after pairs. We first pretrain a physics-aware lighting
prompt (PALP) that embeds the 6D parameters into conditioning
tokens, using an auxiliary planar-light reconstruction objective. Building on a pretrained
diffusion backbone, we then train a fill-light
diffusion (FiLitDiff), an efficient one-step model conditioned on physically grounded lighting codes, enabling controllable and high-fidelity fill lighting at low computational cost. Experiments on held-out paired sets demonstrate strong perceptual quality and competitive full-reference metrics, while better preserving background illumination. The
dataset and model will be at https://github.com/gobunu/Light-Up-Your-Face.
Read the full abstract ▾
All authors Jue Gong, Zihan Zhou, Jingkai Wang, Xiaohong Liu, Yulun Zhang, Xiaokang Yang
Computer Vision
SkeletonGaussian: Editable 4D Generation through Gaussian Skeletonization
Lifan Wu, Ruijie Zhu, Yubo Ai et al. (+1 more)
4D generation has made remarkable progress in synthesizing dynamic 3D objects from input text, images, or
videos. However, existing methods often represent motion as an implicit deformation field, which limits direct control and editability. To address this issue, we propose SkeletonGaussian, a novel framework for generating editable dynamic 3D Gaussians from
monocular video input. Our approach introduces a hierarchical articulated representation that decomposes motion into sparse rigid motion explicitly driven by a skeleton and fine-grained non-rigid motion. Concretely, we extract a robust skeleton and drive rigid motion via linear blend skinning, followed by a hexplane-based refinement for non-rigid deformations, enhancing interpretability and editability. Experimental results demonstrate that SkeletonGaussian surpasses existing methods in generation quality while enabling intuitive motion editing, establishing a new paradigm for editable 4D generation. Project page: https://wusar.github.io/projects/skeletongaussian/
Read the full abstract ▾
All authors Lifan Wu, Ruijie Zhu, Yubo Ai, Tianzhu Zhang
Computer Vision
KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothing
Siyu Jiang, Feiyang Chen, Xiaojin Zhang et al. (+1 more)
Despite the significant progress of
Multimodal Large
Language Models (MLLMs) across diverse tasks,
hallucination – corresponding to the generation of visually inconsistent objects, attributes, or relations – remains a major obstacle to their reliable deployment. Unlike pure
language models, MLLMs must ground their generation process in visual inputs. However, existing models often suffer from semantic drift during decoding, causing outputs to diverge from visual facts as the sequence length increases. To address this issue, we propose KVSmooth, a training-free and plug-and-play method that mitigates
hallucination by performing
attention-
entropy-guided adaptive smoothing on hidden states. Specifically, KVSmooth applies an exponential moving average (EMA) to both keys and values in the KV-Cache, while dynamically quantifying the sink degree of each
token through the
entropy of its
attention distribution to adaptively adjust the smoothing strength. Unlike computationally expensive retraining or contrastive decoding methods, KVSmooth operates efficiently during inference without additional training or model modification. Extensive experiments demonstrate that KVSmooth significantly reduces
hallucination ($\mathit{CHAIR}_{S}$ from $41.8 \rightarrow 18.2$) while improving overall performance ($F_1$ score from $77.5 \rightarrow 79.2$), achieving higher precision and recall simultaneously. In contrast, prior methods often improve one at the expense of the other, validating the effectiveness and generality of our approach.
Read the full abstract ▾
All authors Siyu Jiang, Feiyang Chen, Xiaojin Zhang, Kun He
Computer Vision
Yong Li, Yuanzhi Wang, Yi Ding et al. (+3 more)
Human
multimodal emotion recognition (MER) seeks to infer human
emotions by integrating information from
language, visual, and acoustic modalities. Although existing MER approaches have achieved promising results, they still struggle with inherent
multimodal heterogeneities and varying contributions from different modalities. To address these challenges, we propose a novel framework, Decoupled Hierarchical
Multimodal Distillation (DHMD). DHMD decouples each modality's features into modality-irrelevant (homogeneous) and modality-exclusive (heterogeneous) components using a self-regression mechanism. The framework employs a two-stage knowledge
distillation (KD) strategy: (1) coarse-grained KD via a Graph
Distillation Unit (GD-Unit) in each decoupled feature space, where a dynamic graph facilitates adaptive
distillation among modalities, and (2) fine-grained KD through a cross-modal dictionary matching mechanism, which aligns semantic granularities across modalities to produce more discriminative MER representations. This hierarchical
distillation approach enables flexible knowledge transfer and effectively improves cross-modal feature
alignment. Experimental results demonstrate that DHMD consistently outperforms state-of-the-art MER methods, achieving 1.3%/2.4% (ACC$_7$), 1.3%/1.9% (ACC$_2$) and 1.9%/1.8% (F1) relative improvement on CMU-MOSI/CMU-MOSEI
dataset, respectively. Meanwhile, visualization results reveal that both the graph edges and dictionary activations in DHMD exhibit meaningful distribution patterns across modality-irrelevant/-exclusive feature spaces.
Read the full abstract ▾
All authors Yong Li, Yuanzhi Wang, Yi Ding, Shiqing Zhang, Ke Lu, Cuntai Guan
Computer Vision
Depth-Guided Metric-Aware Temporal Consistency for Monocular Video Human Mesh Recovery
Jiaxin Cen, Xudong Mao, Guanghui Yue et al. (+4 more)
Monocular video human mesh recovery faces fundamental challenges in maintaining metric consistency and temporal stability due to inherent depth ambiguities and scale uncertainties. While existing methods rely primarily on RGB features and temporal smoothing, they struggle with depth ordering, scale drift, and occlusion-induced instabilities. We propose a comprehensive depth-guided framework that achieves metric-aware temporal consistency through three synergistic components: A Depth-Guided Multi-Scale Fusion module that adaptively integrates
geometric priors with RGB features via confidence-aware gating; A Depth-guided Metric-Aware Pose and Shape (D-MAPS)
estimator that leverages depth-calibrated bone statistics for scale-consistent initialization; A Motion-Depth Aligned Refinement (MoDAR) module that enforces temporal coherence through cross-modal
attention between motion dynamics and
geometric cues. Our method achieves superior results on three challenging
benchmarks, demonstrating significant improvements in robustness against heavy occlusion and spatial accuracy while maintaining computational efficiency.
Read the full abstract ▾
All authors Jiaxin Cen, Xudong Mao, Guanghui Yue, Wei Zhou, Ruomei Wang, Fan Zhou, Baoquan Zhao
Computer Vision
ACIL: Active Class Incremental Learning for Image Classification
Aditya R. Bhattacharya, Debanjan Goswami and Shayok Chakraborty
Continual learning (or class incremental learning) is a realistic learning scenario for computer
vision systems, where deep neural networks are trained on episodic data, and the data from previous episodes are generally inaccessible to the model. Existing research in this domain has primarily focused on avoiding catastrophic forgetting, which occurs due to the continuously changing class distributions in each episode and the inaccessibility of the data from previous episodes. However, these methods assume that all the training samples in every episode are annotated; this not only incurs a huge annotation cost, but also results in a wastage of annotation effort, since most of the samples in a given episode will not be accessible to the model in subsequent episodes. Active learning
algorithms identify the salient and informative samples from large amounts of unlabeled data and are instrumental in reducing the human annotation effort in inducing a deep neural network. In this paper, we propose ACIL, a novel active learning framework for class incremental learning settings. We exploit a criterion based on uncertainty and diversity to identify the exemplar samples that need to be annotated in each episode, and will be appended to the data in the next episode. Such a framework can drastically reduce annotation cost and can also avoid catastrophic forgetting. Our extensive empirical analyses on several
vision datasets corroborate the promise and potential of our framework against relevant baselines.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Towards Next-Generation SLAM: A Survey on 3DGS-SLAM Focusing on Performance, Robustness, and Future Directions
Li Wang, Ruixuan Gong, Yumo Han et al. (+6 more)
Traditional Simultaneous Localization and Mapping (SLAM) systems often face limitations including coarse rendering quality, insufficient recovery of scene details, and poor robustness in dynamic environments. 3D Gaussian Splatting (3DGS), with its efficient explicit representation and high-quality rendering capabilities, offers a new reconstruction paradigm for SLAM. This survey comprehensively reviews key technical approaches for integrating 3DGS with SLAM. We analyze performance optimization of representative methods across four critical dimensions: rendering quality, tracking accuracy, reconstruction speed, and
memory consumption, delving into their design principles and breakthroughs. Furthermore, we examine methods for enhancing the robustness of 3DGS-SLAM in complex environments such as motion blur and dynamic environments. Finally, we discuss future challenges and development trends in this area. This survey aims to provide a technical reference for researchers and foster the development of next-generation SLAM systems characterized by high fidelity, efficiency, and robustness.
Read the full abstract ▾
All authors Li Wang, Ruixuan Gong, Yumo Han, Lei Yang, Lu Yang, Ying Li, Bin Xu, Huaping Liu, Rong Fu
Computer Vision
SPOT-Occ: Sparse Prototype-guided Transformer for Camera-based 3D Occupancy Prediction
Suzeyu Chen, Leheng Li and Ying-Cong Chen
Achieving highly accurate and real-time 3D occupancy prediction from cameras is a critical requirement for the safe and practical deployment of autonomous vehicles. While this shift to sparse 3D representations solves the encoding bottleneck, it creates a new challenge for the decoder: how to efficiently aggregate information from a sparse, non-uniformly distributed set of voxel features without resorting to computationally prohibitive dense
attention. In this paper, we propose a novel Prototype-based Sparse
Transformer Decoder that replaces this costly interaction with an efficient, two-stage process of guided feature selection and focused aggregation. Our core idea is to make the decoder's
attention prototype-guided. We achieve this through a sparse prototype selection mechanism, where each query adaptively identifies a compact set of the most salient voxel features, termed prototypes, for focused feature aggregation. To ensure this dynamic selection is stable and effective, we introduce a complementary denoising paradigm. This approach leverages ground-truth masks to provide explicit guidance, guaranteeing a consistent query-prototype association across decoder layers. Our model, dubbed SPOT-Occ, outperforms previous methods with a significant margin in speed while also improving accuracy. Source code is released at https://github.com/chensuzeyu/SpotOcc.
Read the full abstract ▾
Computer Vision · Cross-listed from math.OC
An Improved Boosted DC Algorithm for Nonsmooth Functions with Applications in Image Recovery
ZeYu Li, Te Qi and TieYong Zeng
We propose a new approach to perform the boosted difference of convex functions
algorithm (BDCA) on non-smooth and non-convex problems involving the difference of convex (DC) functions. The recently proposed BDCA uses an extrapolation step from the point computed by the classical DC
algorithm (DCA) via a line search procedure in a descent direction to get an additional decrease of the objective function and accelerate the convergence of DCA. However, when the first function in DC decomposition is non-smooth, the direction computed by BDCA can be ascent and a monotone line search cannot be performed. In this work, we proposed a monotone improved boosted difference of convex functions
algorithm (IBDCA) for certain types of non-smooth DC programs, namely those that can be formulated as the difference of a possibly non-smooth function and a smooth one. We show that any cluster point of the sequence generated by IBDCA is a critical point of the problem under consideration and that the corresponding objective value is monotonically decreasing and convergent. We also present the global convergence and the convergent rate under the Kurdyka-Lojasiewicz property. The applications of IBDCA in image recovery show the effectiveness of our proposed method. The corresponding numerical experiments demonstrate that our IBDCA outperforms DCA and other state-of-the-art DC methods in both computational time and number of iterations.
Read the full abstract ▾
Computer Vision
An Intuitionistic Fuzzy Logic Driven UNet architecture: Application to Brain Image segmentation
Hanuman Verma, Kiho Im, Pranabesh Maji et al. (+1 more)
Accurate
segmentation of MRI
brain images is essential for image analysis, diagnosis of neuro-logical disorders and medical image computing. In the deep learning approach, the convolutional neural networks (CNNs), especially UNet, are widely applied in medical image
segmentation. However, it is difficult to deal with uncertainty due to the partial volume effect in
brain images. To overcome this limitation, we propose an enhanced framework, named UNet with intuitionistic fuzzy logic (IF-UNet), which incorporates intuitionistic fuzzy logic into UNet. The model processes input data in terms of membership, nonmembership, and hesitation degrees, allowing it to better address tissue ambiguity resulting from partial volume effects and boundary uncertainties. The proposed architecture is evaluated on the Internet
Brain Segmentation Repository (IBSR)
dataset, and its performance is computed using accuracy, Dice coefficient, and intersection over union (IoU). Experimental results confirm that IF-UNet improves
segmentation quality with handling uncertainty in
brain images.
Read the full abstract ▾
All authors Hanuman Verma, Kiho Im, Pranabesh Maji, Akshansh Gupta
Computer Vision
Yao Teng, Minxuan Lin, Xian Liu et al. (+3 more)
Recent
video generation models largely rely on
video autoencoders that compress pixel-space
videos into latent representations. However, existing
video autoencoders suffer from three major limitations: (1) fixed-rate compression that wastes
tokens on simple
videos, (2) inflexible CNN architectures that prevent variable-length latent modeling, and (3) deterministic decoders that struggle to recover appropriate details from compressed latents. To address these issues, we propose One-Dimensional
Diffusion Video Autoencoder (One-DVA), a
transformer-based framework for adaptive 1D encoding and
diffusion-based decoding. The encoder employs query-based
vision transformers to extract spatiotemporal features and produce latent representations, while a variable-length dropout mechanism dynamically adjusts the latent length. The decoder is a pixel-space
diffusion transformer that reconstructs
videos with the latents as input conditions. With a two-stage training strategy, One-DVA achieves performance comparable to 3D-CNN VAEs on reconstruction metrics at identical compression ratios. More importantly, it supports adaptive compression and thus can achieve higher compression ratios. To better support downstream latent generation, we further regularize the One-DVA latent distribution for generative modeling and fine-tune its decoder to mitigate artifacts caused by the generation process.
Read the full abstract ▾
All authors Yao Teng, Minxuan Lin, Xian Liu, Shuai Wang, Xiao Yang, Xihui Liu
Computer Vision
AGMA: Adaptive Gaussian Mixture Anchors for Prior-Guided Multimodal Human Trajectory Forecasting
Chao Li, Rui Zhang, Siyuan Huang et al. (+2 more)
Human trajectory
forecasting requires capturing the
multimodal nature of pedestrian behavior. However, existing approaches suffer from prior misalignment. Their learned or fixed priors often fail to capture the full distribution of plausible futures, limiting both prediction accuracy and diversity. We theoretically establish that prediction error is lower-bounded by prior quality, making prior modeling a key performance bottleneck. Guided by this insight, we propose AGMA (Adaptive Gaussian Mixture Anchors), which constructs expressive priors through two stages: extracting diverse behavioral patterns from training data and distilling them into a scene-adaptive global prior for inference. Extensive experiments on ETH-UCY, Stanford Drone, and JRDB
datasets demonstrate that AGMA achieves state-of-the-art performance, confirming the critical role of high-quality priors in trajectory
forecasting.
Read the full abstract ▾
All authors Chao Li, Rui Zhang, Siyuan Huang, Xian Zhong, Hongbo Jiang
Computer Vision
VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latents
Feng Wang, Yichun Shi, Ceyuan Yang et al. (+4 more)
This work presents VTok, a unified
video tokenization framework that can be used for both generation and understanding tasks. Unlike the leading
vision-
language systems that tokenize
videos through a naive frame-sampling strategy, we propose to decouple the spatial and temporal representations of
videos by retaining the spatial features of a single key frame while encoding each subsequent frame into a single residual
token, achieving compact yet expressive
video tokenization. Our experiments suggest that VTok effectively reduces the complexity of
video representation from the product of frame count and per-frame
token count to their sum, while the residual
tokens sufficiently capture viewpoint and motion changes relative to the key frame. Extensive evaluations demonstrate the efficacy and efficiency of VTok: it achieves notably higher performance on a range of
video understanding and text-to-
video generation
benchmarks compared with baselines using naive tokenization, all with shorter
token sequences per
video (e.g., 3.4% higher accuracy on our TV-Align
benchmark and 1.9% higher VBench score). Remarkably, VTok produces more coherent motion and stronger guidance following in text-to-
video generation, owing to its more consistent temporal encoding. We hope VTok can serve as a standardized
video tokenization paradigm for future research in
video understanding and generation.
Read the full abstract ▾
All authors Feng Wang, Yichun Shi, Ceyuan Yang, Qiushan Guo, Jingxiang Sun, Alan Yuille, Peng Wang
Computer Vision
Continuous Degradation Modeling via Latent Flow Matching for Real-World Super-Resolution
Hyeonjae Kim, Dongjin Kim, Eugene Jin et al. (+1 more)
While deep learning-based super-resolution (SR) methods have shown impressive outcomes with synthetic degradation scenarios such as bicubic downsampling, they frequently struggle to perform well on real-world images that feature complex, nonlinear degradations like noise, blur, and compression artifacts. Recent efforts to address this issue have involved the painstaking compilation of real low-resolution (LR) and high-resolution (HR) image pairs, usually limited to several specific downscaling factors. To address these challenges, our work introduces a novel framework capable of synthesizing authentic LR images from a single HR image by leveraging the latent degradation space with flow matching. Our approach generates LR images with realistic artifacts at unseen degradation levels, which facilitates the creation of large-scale, real-world SR training
datasets. Comprehensive quantitative and qualitative assessments verify that our synthetic LR images accurately replicate real-world degradations. Furthermore, both traditional and arbitrary-scale SR models trained using our
datasets consistently yield much better HR outcomes.
Read the full abstract ▾
All authors Hyeonjae Kim, Dongjin Kim, Eugene Jin, Tae Hyun Kim
Computer Vision
DiMo: Discrete Diffusion Modeling for Motion Generation and Understanding
Ning Zhang, Zhengyu Li, Kwong Weng Loh et al. (+7 more)
Prior masked modeling motion generation methods predominantly study text-to-motion. We present DiMo, a discrete
diffusion-style framework, which extends masked modeling to bidirectional text–motion understanding and generation. Unlike GPT-style autoregressive approaches that tokenize motion and decode sequentially, DiMo performs iterative masked
token refinement, unifying Text-to-Motion (T2M), Motion-to-Text (M2T), and text-free Motion-to-Motion (M2M) within a single model. This decoding paradigm naturally enables a quality-latency trade-off at inference via the number of refinement steps. We further improve motion
token fidelity with residual vector quantization (RVQ) and enhance
alignment and controllability with Group Relative Policy Optimization (GRPO). Experiments on HumanML3D and KIT-ML show strong motion quality and competitive bidirectional understanding under a unified framework. In addition, we demonstrate model ability in text-free motion completion, text-guided motion prediction and motion caption correction without architectural change. Additional qualitative results are available on our project page: https://animotionlab.github.io/DiMo/.
Read the full abstract ▾
All authors Ning Zhang, Zhengyu Li, Kwong Weng Loh, Mingxi Xu, Qi Wang, Zhengyu Wen, Xiaoyu He, Wei Zhao, Kehong Gong, Mingyuan Zhang
Computer Vision
Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Models
Angel Martinez-Sanchez, Parthib Roy and Ross Greer
Instruction-grounded driving, where passenger
language guides trajectory planning, requires vehicles to understand intent before motion. However, most prior instruction-following planners rely on simulation or fixed command vocabularies, limiting real-world generalization. doScenes, the first real-world
dataset linking free-form instructions (with referentiality) to nuScenes ground-truth motion, enables instruction-conditioned planning. In this work, we adapt OpenEMMA, an open-source MLLM-based end-to-end driving framework that ingests front-camera views and ego-state and outputs 10-step speed-
curvature trajectories, to this setting, presenting a reproducible instruction-conditioned baseline on doScenes and investigate the effects of human instruction
prompts on predicted driving behavior. We integrate doScenes directives as passenger-style
prompts within OpenEMMA's
vision-
language interface, enabling linguistic conditioning before trajectory generation. Evaluated on 849 annotated scenes using ADE, we observe that instruction conditioning substantially improves robustness by preventing extreme baseline failures, yielding a 98.7% reduction in mean ADE. When such outliers are removed, instructions still influence trajectory
alignment, with well-phrased
prompts improving ADE by up to 5.1%. We use this analysis to discuss what makes a "good" instruction for the OpenEMMA framework. We release the evaluation
prompts and scripts to establish a reproducible baseline for instruction-aware planning. GitHub: https://github.com/Mi3-Lab/doScenes-VLM-Planning
Read the full abstract ▾
Computer Vision
HoloEv-Net: Efficient Event-based Action Recognition via Holographic Spatial Embedding and Global Spectral Gating
Weidong Hao
Event-based Action Recognition (EAR) has attracted significant
attention due to the high temporal resolution and high dynamic range of event cameras. However, existing methods typically suffer from (i) the computational redundancy of dense voxel representations, (ii) structural redundancy inherent in multi-branch architectures, and (iii) the under-utilization of spectral information in capturing global motion patterns. To address these challenges, we propose an efficient EAR framework named HoloEv-Net. First, to simultaneously tackle representation and structural redundancies, we introduce a Compact
Holographic Spatiotemporal Representation (CHSR). Departing from computationally expensive voxel grids, CHSR implicitly embeds horizontal spatial cues into the Time-Height (T-H) view, effectively preserving 3D spatiotemporal contexts within a 2D representation. Second, to exploit the neglected spectral cues, we design a Global Spectral Gating (GSG) module. By leveraging the Fast
Fourier Transform (FFT) for global
token mixing in the frequency domain, GSG enhances the representation capability with negligible parameter overhead. Extensive experiments demonstrate the scalability and effectiveness of our framework. Specifically, HoloEv-Net-Base achieves state-of-the-art performance on THU-EACT-50-CHL, HARDVS and DailyDVS-200, outperforming existing methods by 10.29%, 1.71% and 6.25%, respectively. Furthermore, our lightweight variant, HoloEv-Net-Small, delivers highly competitive accuracy while offering extreme efficiency, reducing parameters by 5.4 times, FLOPs by 300times, and latency by 2.4times compared to heavy baselines, demonstrating its potential for edge deployment.
Read the full abstract ▾
Computer Vision
Partial Ring Scan: Revisiting Scan Order in Vision State Space Models
Yi-Kuan Hsieh, Kuan-Chuan Peng, Xin li et al. (+3 more)
State Space Models (SSMs) have emerged as efficient alternatives to
attention for
vision tasks, offering lineartime sequence processing with competitive accuracy.
Vision SSMs, however, require serializing 2D images into 1D
token sequences along a predefined scan order, a factor often overlooked. We show that scan order critically affects performance by altering spatial adjacency, fracturing object continuity, and amplifying degradation under
geometric transformations such as rotation. We present Partial RIng Scan Mamba (PRISMamba), a rotation-robust traversal that partitions an image into concentric rings, performs order-agnostic aggregation within each ring, and propagates context across rings through a set of short radial SSMs. Efficiency is further improved via partial channel filtering, which routes only the most informative channels through the recurrent ring pathway while keeping the rest on a lightweight residual branch. On ImageNet-1K, PRISMamba achieves 84.5% Top-1 with 3.9G FLOPs and 3,054 img/s on A100, outperforming VMamba in both accuracy and throughput while requiring fewer FLOPs. It also maintains performance under rotation, whereas fixed-path scans drop by 1~2%. These results highlight scan-order design, together with channel filtering, as a crucial, underexplored factor for accuracy, efficiency, and rotation robustness in
Vision SSMs. Code will be released upon acceptance.
Read the full abstract ▾
All authors Yi-Kuan Hsieh, Kuan-Chuan Peng, Xin li, Ming-Ching Chang, Yu-Chee Tseng, Jun-Wei Hsieh
Computer Vision
Point2Insert: Video Object Insertion via Sparse Point Guidance
Yu Zhou, Xiaoyan Yang, Bojia Zi et al. (+6 more)
This paper introduces Point2Insert, a sparse-point-based framework for flexible and user-friendly object insertion in
videos, motivated by the growing popularity of accurate, low-effort object placement. Existing approaches face two major challenges: mask-based insertion methods require labor-intensive mask annotations, while instruction-based methods struggle to place objects at precise locations. Point2Insert addresses these issues by requiring only a small number of sparse points instead of dense masks, eliminating the need for tedious mask drawing. Specifically, it supports both positive and negative points to indicate regions that are suitable or unsuitable for insertion, enabling fine-grained spatial control over object locations. The training of Point2Insert consists of two stages. In Stage 1, we train an insertion model that generates objects in given regions conditioned on either sparse-point
prompts or a binary mask. In Stage 2, we further train the model on paired
videos synthesized by an object removal model, adapting it to
video insertion. Moreover, motivated by the higher insertion success rate of mask-guided editing, we leverage a mask-guided insertion model as a teacher to distill reliable insertion behavior into the point-guided model. Extensive experiments demonstrate that Point2Insert consistently outperforms strong baselines and even surpasses models with $\times$10 more parameters.
Read the full abstract ▾
All authors Yu Zhou, Xiaoyan Yang, Bojia Zi, Lihan Zhang, Ruijie Sun, Weishi Zheng, Haibin Huang, Chi Zhang, Xuelong Li
Computer Vision
Improving 2D Diffusion Models for 3D Medical Imaging with Inter-Slice Consistent Stochasticity
Chenhe Du, Qing Wu, Xuanyu Tian et al. (+3 more)
3D medical imaging is in high demand and essential for clinical diagnosis and scientific research. Currently,
diffusion models (DMs) have become an effective tool for medical imaging reconstruction thanks to their ability to learn rich, high-quality data priors. However, learning the 3D data distribution with DMs in medical imaging is challenging, not only due to the difficulties in data collection but also because of the significant computational burden during model training. A common compromise is to train the DMs on 2D data priors and reconstruct stacked 2D slices to address 3D medical inverse problems. However, the intrinsic randomness of
diffusion sampling causes severe inter-slice discontinuities of reconstructed 3D volumes. Existing methods often enforce continuity regularizations along the z-axis, which introduces sensitive hyper-parameters and may lead to over-smoothing results. In this work, we revisit the origin of stochasticity in
diffusion sampling and introduce Inter-Slice Consistent Stochasticity (ISCS), a simple yet effective strategy that encourages interslice consistency during
diffusion sampling. Our key idea is to control the consistency of
stochastic noise components during
diffusion sampling, thereby aligning their sampling trajectories without adding any new loss terms or optimization steps. Importantly, the proposed ISCS is plug-and-play and can be dropped into any 2D trained
diffusion based 3D reconstruction pipeline without additional computational cost. Experiments on several medical imaging problems show that our method can effectively improve the performance of medical 3D imaging problems based on 2D
diffusion models. Our findings suggest that controlling inter-slice stochasticity is a principled and practically attractive route toward high-fidelity 3D medical imaging with 2D
diffusion priors. The code is available at: https://github.com/duchenhe/ISCS
Read the full abstract ▾
All authors Chenhe Du, Qing Wu, Xuanyu Tian, Jingyi Yu, Hongjiang Wei, Yuyao Zhang