Computer Vision · Cross-listed from eess.IV
ReGenVC: End-to-End Real-Time Generative Video Coding at Ultra-Low Bitrate
Zheyuan Zhang and Johnson Wu
We present ReGenVC, an end-to-end generative
video codec that compresses talking-head
video to an ultra-low bitrate and decodes it in real time. The encoder reduces a source clip to a compact bitstream – a neurally compressed first frame, per-frame pose keypoints, and metadata – totaling about 26 kB for a 77-frame sequence. The decoder is a four-step distilled
diffusion transformer that reconstructs the
video conditioned on the transmitted pose and reference frame. Compared with x264/x265, ReGenVC reduces the bitrate to roughly one tenth of that required by traditional codecs (about 26 kB vs. 250–280 kB for essentially artifact-free reconstruction); at a matched ultra-low bitrate, conventional codecs collapse into blocking artifacts while ReGenVC stays sharp by exploiting a strong generative prior. The central obstacle to deploying such a codec is decoder latency: multi-step sampling with
transformer and VAE components is too slow for interactive use. We make the decoder real-time through four-step
distillation and three model-preserving system techniques: (i) eight-GPU unified sequence parallelism (Ulysses & Ring), (ii) a spatially-split VAE, and (iii) a three-stage overlapped pipeline; an analytical timing model characterizes the real-time feasibility region. On an 8-GPU node, the system sustains 24 fps output (972 ms per 25-frame window, within the 1000 ms budget), enabling a live browser stream without observed frame underruns. A hybrid CPU-GPU deployment further runs the encoder on the CPU at 24 fps and offloads the decoder-side one-shot conditioning encoders to the CPU, reducing the per-GPU
memory peak from 21.1 GB to about 7.7 GB. To our knowledge, ReGenVC is the first end-to-end generative
video codec to combine ultra-low-bitrate encoding with real-time decoding on an 8-GPU system.
Read the full abstract ▾
Computer Vision
Convolutional Neural Shading for High-Quality 3D Reconstruction from Multi-View Images
Juheon Hwang, Taewan Kim, Heeseok Oh et al. (+1 more)
We propose a convolutional neural shading (CNS), a novel pipeline to reconstruct high-quality 3D shapes from multi-view images. Several recent studies have used neural radiance fields and other neural differentiable rendering methods to understand 3D geometry. However, these approaches rely on single-point
geometric information, such as positions and normals of the surface, leading to a lack of detailed local geometry. Our approach addresses the inherent limitations of single-point information by leveraging a neural shader to capture variations even in dark and textureless regions with a convolutional neural shader, resulting in far more accurate geometry predictions. Additionally, our method mitigates surface irregularities at image boundaries by introducing a fine-detail displacement network, which utilizes spatial information of surface geometry and learns fine displacement details by correlating neighboring values in the rendering coordinates. Through extensive experiments, our proposed method has demonstrated significant quality improvements in the reconstructed shapes and rendered images over current state-of-the-art methods.
Read the full abstract ▾
All authors Juheon Hwang, Taewan Kim, Heeseok Oh, Jiwoo Kang
Computer Vision
Collaborative Feature Aggregation for Face Super-Resolution and Robust Re-Identification
Juheon Hwang, Taewan Kim and Jiwoo Kang
We propose a novel collaborative approach for face super-resolution (SR) and robust person re-identification from sequential or multi-view facial images. Traditional SR methods often suffer from blurring and distortion in faces recovered from poor-quality images due to low resolution. Image- and
video-based facial SR methods using facial landmarks or
segmentation also have similar challenges. To overcome these limitations, we leverage multiple correlated facial observations, across time or viewpoints, by introducing a
transformer-based collaborative feature aggregation method that unifies identity features from multi-sequence or multi-view data. This allows faces in multiple sequences of an individual to contribute to accurately estimating common facial features. Furthermore, we propose a cascade SR network to progressively restore the high-resolution image of the target's face with gradual facial feature unification. The unified identity representation is further utilized in person re-identification scenarios, enabling accurate matching even under severe image degradation. The exhaustive experimental results and comparisons show that our method outperforms other state-of-the-art methods, demonstrating consistent improvements in both face super-resolution and re-identification performance. Our work highlights the effectiveness of joint identity reconstruction and progressive image restoration from multiple facial inputs in enhancing downstream visual recognition tasks.
Read the full abstract ▾
Computer Vision
Face and Voice Cross-modal Association with Learning Convex Feature Embedding
Taewan Kim and Jiwoo Kang
Face-and-voice association learning is one of the most challenging tasks in deep learning. In this paper, we propose a simple but powerful cross-modal feature
embedding method for the association of faces and voices. Previous work has studied cross-modal association tasks to establish the correlation between voice clips and facial images. These works have addressed cross-modal discrimination but underestimate the importance of handling heterogeneity in inter-modal features between audio and
video, resulting in a lot of false positives and false negatives. To tackle the problem, the proposed method learns the
embeddings of cross-modal features by making another feature exist between cross-modal features, facilitating the voice and face features of the same person to be embedded in a convex hull. Moreover, the incorporation of cross-modal
attention mechanisms with convex
embedding techniques represents a highly effective strategy for the attenuation of false positives and false negatives, accomplished via the minimization of inter-class discrepancies. We exhaustively evaluated our method for cross-modal verification, matching, and
retrieval tasks on the large-scale VoxCeleb
dataset. Extensive experimental results demonstrate that the proposed method achieves notable improvements over existing state-of-the-art methods.
Read the full abstract ▾
Computer Vision
Towards Practical Algorithm Selection for Unsupervised Domain Adaptation in Medical Imaging
Yiheng Xiong, Luisa Gallée, Daniel Santak Wolf et al. (+2 more)
Numerous unsupervised domain adaptation (UDA)
algorithms exist, but for clinical practice, selecting the best-suited one along with proper hyperparameters often remains unclear, as the unlabeled deployment (target) domain prevents direct evaluation. We propose a label-free criterion that jointly selects the
algorithm and hyperparameters for UDA. Given a pool of candidate models from multiple
algorithms trained with different hyperparameters, our approach scores each candidate against an agreement reference, and selects the one with the highest score. The agreement reference is constructed in two levels without using target labels. First, we leverage multiple label-free selection signals, using each to nominate a model within every
algorithm. Second, the nominated models are aggregated across
algorithms to form a reference prediction for each unlabeled target sample. The candidate whose predictions agree most with this reference is then selected for deployment. Experimental results on four
brain MRI and four chest X-ray
datasets across seven clinically relevant transfer scenarios show that our method achieves better selection performance than other methods and remains effective across different
algorithm pools. Our approach takes a step towards practical, label-free
algorithm selection for clinical deployment of UDA.
Read the full abstract ▾
All authors Yiheng Xiong, Luisa Gallée, Daniel Santak Wolf, Heiko Hillenhagen, Michael Götz
Computer Vision
mmRadarTwin: A Measurement-Calibrated Signal-Level Digital Twin Platform for Indoor mmWave Radar
Jianyi Zhou, Chenghao Zhang, Yanli Li et al. (+1 more)
Indoor mmWave radar perception is difficult to reproduce because measured range-angle responses depend on scene geometry, material response, multipath, hardware conventions, and signal processing. Existing ray-tracing and digital-twin tools often expose rendering, channel, or path-level quantities, while radar sensing requires complex signal products that can be processed and compared in the same domain as real FMCW measurements. We present mmRadarTwin, a signal-level and path-attributed digital-twin platform for indoor mmWave radar. mmRadarTwin links a real radar measurement branch with an Unreal Engine scene-simulation branch through a shared receive-channel and range-angle processing interface. The simulator writes complex multi-channel receive grids and exports per-path contribution records that identify the actor, material tag, propagation event, and output-bin support of each simulated return. We evaluate mmRadarTwin in an office deployment using a commodity monostatic mmWave radar and mobile scene-capture hardware. Across 154 measured poses spanning 22 radar locations, the current physics-only path-basis simulator recalls 70.8% of measurement-active geometry-supported response regions in the central usable field of view while exposing residuals caused by weak or missing path support, shifted responses, unsupported anchors, and missing physical mechanisms. Rather than claiming complete radar-map reconstruction or cross-room generalization, mmRadarTwin establishes a practical systems workflow for constructing, comparing, and diagnosing indoor radar digital twins.
Read the full abstract ▾
All authors Jianyi Zhou, Chenghao Zhang, Yanli Li, Dong Yuan
Computer Vision · Cross-listed from cs.LG
GVR-Coder: A Visual-Feedback Framework for Structured SVG Generation in Complex Document and Meeting Scenarios
Yiming Xu, Jihua Kang, Chunsai Du et al. (+5 more)
In demanding professional environments and meeting review scenarios, lengthy text often imposes a high cognitive load. To facilitate efficient information communication, transforming verbose text into logically clear diagrams is essential. Scalable Vector Graphics (SVG) provide an effective representation for this purpose due to their editability and resolution independence. However, current research on Text-to-SVG generation remains hindered by three major challenges: (1) the scarcity of
datasets for complex, logic-rich diagrams; (2) the absence of explicit layout priors, which leads to chaotic spatial arrangements; and (3) the lack of fine-grained visual feedback to validate rendered outputs and correct aesthetic defects. To address these challenges, at the data level, we introduce DocMeetSVG-100K, a large-scale SVG
dataset tailored for document authoring and meeting review scenarios. At the model level, we propose GVR-Coder, a novel framework designed to generate high-quality logical diagrams from lengthy professional texts. Specifically, we adopt a curriculum-driven rejection sampling fine-tuning to progressively enhance the model's capability in modeling complex structures, while explicitly incorporating layout constraint knowledge during training. In addition, we introduce reinforcement learning from dual rendering feedback, a mechanism that provides implicit feedback through reward signals to jointly optimize structural complexity and visual aesthetics. Furthermore, we design a generate-verify-repair
agent loop, which improves generation quality through explicit, fine-grained feedback and targeted refinement. Extensive experiments demonstrate that GVR-Coder outperforms competitive baselines and reliably produces logically coherent and visually appealing diagrams. Code and data are available at https://github.com/CurryaNa/GVR-Coder.
Read the full abstract ▾
All authors Yiming Xu, Jihua Kang, Chunsai Du, Qifan Zhang, Wangqiu Zhou, Yiting Wu, Tianqi Li, Qi Song
Computer Vision
BladeYOLO: Wind Turbine Blade Defect Detection with Limited Annotations and Weak-Saliency Awareness
Yabin Xu, Fangtao Zhang, Fan Wang et al. (+5 more)
Wind turbine blade defect detection remains highly challenging in real-world inspection scenarios due to limited on-site data and the subtle visual characteristics of defects. In practice, blade defects are often small-scale, low-contrast, and difficult to distinguish from complex backgrounds, which significantly limits the robustness of existing detectors. To address these challenges, we propose BladeYOLO, a defect detection framework for wind turbine blades. Specifically, we integrate a
Vision Transformer (ViT) backbone initialized with DINOv3 self-supervised pre-trained weights into YOLOv12-L, enabling the transfer of large-scale generic visual priors to blade defect detection and improving feature representation under limited training annotations. To enhance the perception of subtle defects, we further develop a Mamba-guided Weak-Defect Enhancement module, which consists of a Detail-Enhanced Multi-scale Branch for preserving high-frequency structural cues and a Cross-Mamba module for progressively propagating high-level semantic guidance to shallow features. In addition, we introduce a lightweight Style-Injector module that captures environment-related style information via
Fourier decomposition and injects it into selected ViT self-
attention layers, thereby improving robustness against environment-induced appearance variations. Extensive experiments demonstrate that BladeYOLO achieves superior performance on the WTBlade-Defect
dataset, with additional annotation-budget experiments showing its favorable performance under reduced training annotations. Evaluation on the public Wind Surface Defect
dataset further provides supportive evidence for the cross-
dataset robustness of BladeYOLO. In particular, on this public
dataset, BladeYOLO outperforms the best competing method by 3.5% in mAP$_{50}$ and 2.5% in mAP$_{50-95}$.
Read the full abstract ▾
All authors Yabin Xu, Fangtao Zhang, Fan Wang, Zhan Wang, Honghua Chen, Mingqiang Wei, Haoran Xie, Sam Kwong
Computer Vision
Landmark shape spaces with induced metrics
Sarang Joshi, Peter W. Michor and Stefan Sommer
We present a unification of Kendall's landmark shape spaces, where rigid motions are factored out and scale fixed on landmark configurations equipped with Euclidean geometry, with landmark configuration spaces carrying Riemannian metrics descending from right-invariant Sobolev metrics on the diffeomorphism group. The resulting new landmark shape spaces achieve the defining properties of both approaches: The regularity of the descending metric prevents landmarks from colliding, the metric is defined in the ambient space independent of the number of landmarks, local rigid transformations are preserved, global rigid motions are removed, and scale fixed. To achieve this, we define a particular Sobolev-type operator, the screened elasticity operator, whose null-space consists exactly of the rigid motions, we show how this operator descends to achieve the desired geometry, and we present approaches to solving matching problems and computing geodesics numerically. The resulting construction allows the use of landmark configuration spaces with sufficiently regular metrics in applications while retaining the shape invariances that are a hallmark of Kendall's shape spaces.
Read the full abstract ▾
Computer Vision
Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion
Henglin Liu, Fangyuan Kong, Jing Wang et al. (+7 more)
Recent advances in preference
alignment for
diffusion-based
video generation, particularly via Direct Preference Optimization (DPO), have significantly improved visual quality. However, temporally sparse artifacts such as motion collapse, object flickering, and color oversaturation remain a major barrier to perceptual realism. Existing methods struggle with these issues due to two key limitations: (1) the preference attribution bottleneck, where offline human annotations are costly and fail to accurately capture learning dynamics, while online reward signals are rollout-aware but often unstable and biased; and (2) temporal credit misallocation, where uniformly applied supervision cannot effectively target the brief segments in which artifacts occur. To address these challenges, we propose concentrated Implicit Preference Optimization (cIPO), a post-training framework for
video diffusion models. cIPO derives implicit preference signals directly from the denoising process: given a real
video, the model adds forward noise and reconstructs it via iterative denoising, treating the original as the preferred sample and the reconstruction as the dispreferred one. This formulation captures inference-time errors without requiring human annotations or external reward models. Moreover, frame-level discrepancies between original and reconstructed
videos reveal when failures occur. cIPO leverages this by computing temporal reconstruction errors and concentrating optimization on high-error segments, enabling more precise correction of failure-prone regions. Extensive experiments demonstrate that cIPO consistently enhances
video authenticity and temporal coherence across multiple
datasets, highlighting the effectiveness and efficiency of implicit preference with temporally concentrated optimization.
Read the full abstract ▾
All authors Henglin Liu, Fangyuan Kong, Jing Wang, Yizhou Lin, Nisha Huang, Chang Liu, Xintao Wang, Pengfei Wan, Kun Gai, Xiu Li
Computer Vision · Cross-listed from eess.IV
TSOG: A Format For Temporally And Spatially Ordered Gaussians
Shady Gmira, Evangelos Alexiou, Emmanouil Potetsianakis et al. (+1 more)
We propose Temporally and Spatially Ordered Gaussians (TSOG), a format for efficient representation of 4D Gaussian Splatting (4DGS) content. TSOG extends the Spatially Ordered Gaussians (SOG) framework to the temporal domain by introducing a timeline attribute and temporal parameterization of geometry and appearance attributes. Similar to SOG, TSOG is a lossy format that assigns each Gaussian a unique index and encodes attribute values as index-aligned image data. TSOG is model-agnostic, extensible, and compatible with both discrete and continuous 4DGS representations. Evaluation using a PLYs sequence and FreeTimeGS as baselines, serving as simplistic and state-of-the-art 4DGS representations respectively, shows file size reductions exceeding 90%, with PSNR differences ranging between -0.42 and +0.85 dB. These results demonstrate substantial file size savings with minimal quality degradation, enabling efficient representation, storage, and delivery of dynamic scenes for next-generation 4D content.
Read the full abstract ▾
All authors Shady Gmira, Evangelos Alexiou, Emmanouil Potetsianakis, Emmanuel Thomas
Computer Vision
TongueReenact: Geometry-Anchored Tongue Synthesis for Face Reenactment
MD Wahiduzzaman Khan, Mingshan Jia, Xiaolin Zhang et al. (+2 more)
Modern face reenactment systems achieve impressive pose and expression transfer using geometry-driven representations. However, they largely ignore tongue dynamics, leading to anatomically inconsistent mouth interiors during
speech and expressive motions. We introduce the first framework for cross-identity tongue dynamics transfer in face reenactment. We propose a foundation-model-assisted bootstrapping pipeline that produces a dedicated tongue
segmentation model for in-the-wild reenactment without curated annotations. We further introduce a spatially constrained latent masked
diffusion model for realistic tongue synthesis, with adaptive mask dilation for seamless mouth boundary transitions. Extensive experiments demonstrate improvements of more than two times over all baselines on every tongue-specific metric. We additionally propose a VLM-based evaluation protocol that replicates expert annotation at scale, confirming perceptual superiority across all ablation variants.
Read the full abstract ▾
All authors MD Wahiduzzaman Khan, Mingshan Jia, Xiaolin Zhang, En Yu, Kaska Musial-Gabrys
Computer Vision
Split and Drive: Dual-Axis Disentanglement for Real-Time Gaussian Head Avatars
MD Wahiduzzaman Khan, Mingshan Jia, Xiaolin Zhang et al. (+2 more)
Creating photorealistic animatable head avatars from a single image remains a fundamental challenge in digital human synthesis. While recent 3D Gaussian Splatting methods have achieved promising results, they rely on external tracking pipelines whose latency is excluded from inference measurements. Furthermore, they adopt unified representations that entangle geometrically distinct facial regions, limiting both expressiveness and rendering fidelity. We propose SpiD (Split and Drive), a single-image Gaussian head avatar framework built on two disentanglement axes. The compute axis internalizes per-frame driving, eliminating external tracking dependency at inference. The feature axis decomposes the avatar into three specialized Gaussian branches, each modeling a geometrically distinct facial domain. Extensive experiments demonstrate consistently strong performance against state-of-the-art methods while achieving the fastest inference speed among all compared methods on a single GPU with the complete driving pipeline included.
Read the full abstract ▾
All authors MD Wahiduzzaman Khan, Mingshan Jia, Xiaolin Zhang, En Yu, Kaska Musial-Gabrys
Computer Vision · Cross-listed from cs.AI
MUL-T: Decoding Spatial Cellular Architecture in Multiplexed Tissue Images
Farzaneh Seyedshahi, Kai Rakovic, Adalberto Claudio Quiros et al. (+2 more)
Understanding tissue organisation in multiplexed imaging requires modelling both cellular phenotypes and their spatial context. Existing approaches typically rely on handcrafted features, such as marker intensity statistics or
cell-type proportions, which often fail to scale or generalise across cohorts with heterogeneous marker panels. We introduce MUL-T, a lightweight
transformer framework that reframes tissue architecture as a masked contextual prediction task over discrete
cell tokens. By learning contextualised [CLS]
embeddings without task-specific supervision, the model captures higher-order cellular interactions while remaining computationally efficient. We evaluate MUL-T on several clinically relevant downstream tasks, including core-level tumour pattern classification, patient-level grading, PD-L1 positivity prediction, and cross-
dataset treatment response prediction. Across tasks, MUL-T consistently outperforms classical feature-based baselines and achieves performance comparable to a foundation ViT model, despite substantially fewer parameters and lower training cost.
Read the full abstract ▾
All authors Farzaneh Seyedshahi, Kai Rakovic, Adalberto Claudio Quiros, John LeQuesne, Ke Yuan
Computer Vision
ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression
Shuhan Ye, Hongbin Yu, Chenqi Kong et al. (+4 more)
Learned
video compression relies on accurate temporal modeling to remove redundancy between adjacent frames. However, most existing codecs infer motion solely from discretely sampled RGB frames, making their estimates vulnerable to fast motion, blur, occlusion, weak texture, low illumination, and abrupt brightness changes. Event cameras asynchronously capture fine-grained intensity changes between RGB timestamps and therefore provide complementary evidence about inter-frame dynamics. We propose ENCORE, an Event-Assisted Complementary Motion Refinement framework for learned
video compression. ENCORE first employs Complementary Motion Representation (CMR) to decompose aligned RGB-event features into common and modality-specific motion representations. Spatial Energy and Redundancy-Informed Calibration (SERIC) then identifies event-specific responses that are active and novel relative to RGB, suppresses weak or redundant evidence, and predicts a candidate flow correction. Finally, Energy-Aware Routing (EAR) determines where and how strongly the correction should refine the RGB flow. Events serve solely as an auxiliary modality for motion modeling, while RGB remains the only coding and reconstruction target. Experiments on BS-ERGB, HQ-EVFI, and CED demonstrate consistent gains across
datasets and GOP lengths. On BS-ERGB, ENCORE achieves up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings, while retaining clear improvements on the other two
datasets.
Read the full abstract ▾
All authors Shuhan Ye, Hongbin Yu, Chenqi Kong, Pingchuan Ma, Chong Wang, Jun Wan, Qixin Zhang
Computer Vision
Beyond Classification: Pathology Foundation Models as Detection Encoders for Mitotic Figures
Sweta Banerjee, Alireza Teimoury, Nils Porsche et al. (+11 more)
Pathology foundation models (FMs) are models trained on vast amounts of typically unlabeled data and have been shown to yield regularized latent spaces that can be used effectively in downstream classification tasks. This is also true for the classification of mitotic figures vs. other
cells. However, it is so far unclear if the latent space of current FMs provides features that are discriminant and spatially suitably resolved to also serve as a backbone for dense object detection paradigms. In this work, we investigate this question for common current pathology FMs (UNI, UNI2-h, Virchow, Virchow2, H-optimus-0, H-optimus-1) and compare their performance against a fully end-to-end trained baseline based on a ResNet50 architecture. We combine FM backbones with representatives of single stage, dual stage and self-
attention-based detectors (RetinaNet, Faster R-CNN, Deformable DETR respectively) on the multi-domain MIDOG++
dataset, and on the TUPAC16
dataset as an out-of-domain case. We show that the H-optimus-0 and Virchow models yielded competitive performance, indicating that the latent spaces of current FMs, all trained on image-level self-supervision, are suitable for direct mitotic figure detection and may be slightly more robust on our out-of-domain test case. All code is made available publicly at https://github.com/DeepMicroscopy/FM4MFdet.
Read the full abstract ▾
All authors Sweta Banerjee, Alireza Teimoury, Nils Porsche, Alexandra K. Stoll, Viktoria Weiss, Niklas Hargarter, Jonas Ammeling, Thomas Conrad, Christoph Stroblberger, Christopher Kaltnecker, Robert Klopfleisch, Christof A. Bertram, Katharina Breininger, Marc Aubreville
Computer Vision
Deep learning-based hierarchical insect classification using camera trap imagery
Zaki Mahfoud, Juan A. Chiavassa, Simon Walther et al. (+2 more)
Declining insect populations make reliable biodiversity monitoring increasingly urgent, yet monitoring of insect biodiversity is hampered by a lack of standardised data and by costly and time-consuming manual identification by expert entomologists. Deep learning-based image classifiers, processing data from automated non-lethal camera traps, have the potential to transform and scale insect biodiversity monitoring. However, challenges remain in acquiring expert-annotated
datasets, developing model architectures that generalise well across diverse taxonomic levels and training models on highly imbalanced data. Hierarchical data also benefits from designing models that default to higher-confidence, coarser-level predictions, when uncertain about finer taxonomic levels. In this paper we address these challenges with a deep learning-based hierarchical classification model. First, we present a manually curated, long-tailed
dataset of around one million images of insects, extracted from 1,801 camera-trap
video recordings and annotated with a five-level, 34-class hierarchy. Further, we adapt a hierarchical classification model architecture to a five-level variable-depth hierarchy, with class-balanced weighting. Our model improves on non-hierarchical classifiers by leveraging biological taxonomy to extract granularity-specific visual features and makes hierarchy-consistent predictions to the deepest taxonomic level that meets a confidence threshold (T = 0.6). Our model achieved a per-level accuracy of 80-99% on test data, across five levels of hierarchy. Furthermore ...
Read the full abstract ▾
All authors Zaki Mahfoud, Juan A. Chiavassa, Simon Walther, Florian Haselbeck, Ehsan Yaghoubi
Computer Vision
ViP-Rig: Visual-Prompted Controllable Rigging
Zihan Qin, Mingze Sun, Yifan Mao et al. (+6 more)
Rigging is inherently task-dependent because the same mesh may require different skeletons and deformation behaviors across animation tasks. In practice, artists often inspect an initial rig and repeatedly edit its skeletal structure and deformation behavior to meet specific animation requirements. Existing automatic methods primarily generate a plausible rig from geometry, offering limited explicit control over the resulting skeleton and deformation behavior. In this work, we present ViP-Rig, a visual-prompted framework that supports both
prompt-first rigging and result-guided editing by injecting features extracted from user-drawn or edited 2D skeletal and
rigidity prompts into frozen pretrained backbones. Specifically, ViP-Rig consists of two stages, Skeleton Generation and Skinning Prediction. In the first stage, the skeletal sketch is processed by the Dense-to-Compact Visual
Prompt Encoding to produce compact, fixed-length conditioning
tokens. The resulting
tokens are injected into a frozen pretrained autoregressive generator through gated adapters to control joint placement and branching structure while preserving the generator's
geometric prior. In the second stage, the
rigidity map is processed using the same visual encoding design, while the pretrained skinning backbone remains frozen. The resulting
tokens are symmetrically injected into the point and joint streams to modulate point-joint compatibility and the resulting skinning weights. Experiments on Articulation-XL2.0 and zero-shot evaluation on ModelsResource show that ViP-Rig more accurately recovers target skeletons and skinning weights than geometry-conditioned baselines under
prompt-guided evaluation. Qualitative results further demonstrate explicit and localized control in both
prompt-first rigging and result-guided editing.
Read the full abstract ▾
All authors Zihan Qin, Mingze Sun, Yifan Mao, Jialei Xu, Jingfeng Guo, Changrong Hu, Wenbo Zhao, Junjun Jiang, Xianming Liu
Computer Vision
Now You Have My Healthy Attention: A U-DiT for Brain-MRI Inpainting
Danilo Danese, Angela Lombardi and Tommaso Di Noia
The ASNR-MICCAI BraTS Local Synthesis (Inpainting) task asks for the anatomically plausible completion of healthy
brain tissue within a masked region of a T1-weighted MRI, providing a tumor-free anatomical reference for downstream analysis. As the task is scored by distortion metrics (SSIM, PSNR, MSE), we build a deterministic regression model and focus on giving it inductive biases tailored to inpainting. Our network follows the U-DiT principle of performing self-
attention on a downsampled
token grid: a volumetric encoder-decoder imports long-range context through a downsampled global self-
attention block with three-dimensional rotary position
embeddings, while convolutions and skip connections preserve high-frequency detail. Two ideas drive our results. First, we constrain the
attention so that occluded ("void")
tokens attend only to known-healthy
tokens of the same volume, with a learned bias toward each query's contralateral homologue, forcing the completion to be inferred from observed anatomy rather than from other unknown regions. Second, we add a contralateral-
symmetry input that supplies the mirrored healthy hemisphere as a patient-specific prior; since the
brain is approximately bilaterally symmetric and lesions are typically unilateral, this prior improves the distortion metrics at matched structural similarity. On the official BraTS-2026 validation leaderboard our submission reaches a mean healthy-region SSIM of $0.864$, PSNR of $24.7$\,dB and MSE of $4.6{\times}10^{-3}$ over $219$ cases. We further analyse the residual smoothness inherent to distortion-optimal regression and discuss its implications for anatomical realism.
Read the full abstract ▾
Computer Vision
FootprintNet: State-Transition-Guided Dynamic Footprint Learning for Multi-temporal Remote Sensing Change Detection
Haotian Zhang, Hao Chen, Han Guo et al. (+2 more)
Despite substantial progress in remote sensing multi-temporal change detection (MTCD), most existing MTCD methods still represent the dynamic process at each spatial location over the entire observation period using a single change category associated with the final observation. This implicit single-change assumption limits their ability to characterize regions of recurrent change closely related to human activities. To address this limitation, we introduce Urban Building Dynamics Detection (UBDD), which identifies building-change dynamic footprints, i.e., the temporal intervals in which changes occur, from multi-temporal imagery and produces pixel-wise classification masks. For regions undergoing two or more changes, UBDD introduces an independent multi-change class for unified representation, thereby enabling unified modeling of single- and multi-change processes. Furthermore, we propose FootprintNet, which abstracts building-change processes as interactions between latent states and actions, and imposes state-action transition constraints to guide the learning of causally coherent change trajectories. It further exploits temporal change-boundary cues to enhance feature contrast across boundary sides, thereby improving the discrimination among different dynamic footprints and enabling accurate detection of dynamic footprints. Moreover, we introduce the Building Change Dynamics Score (BCDS) to address the inability of conventional metrics to reflect the temporal proximity between predicted footprints and labels. It evaluates predictions according to their preservation of change semantics and temporal offsets from the corresponding labels. Extensive experiments on TSCD, MUDS, and WUSU demonstrate that FootprintNet outperforms current state-of-the-art methods. The code is available at https://github.com/zmoka-zht/FootprintNet.
Read the full abstract ▾
All authors Haotian Zhang, Hao Chen, Han Guo, Zhengxia Zou, Zhenwei Shi
Computer Vision
FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval
Bohan Hou, Haoqiang Lin, Xuemeng Song et al. (+4 more)
Due to their strong generalizable
multimodal processing and
reasoning capabilities,
Multimodal Large
Language Models (MLLMs) have demonstrated significant potential as universal image retrievers, effectively addressing diverse real-world image
retrieval tasks. Nevertheless, pioneering studies, while promising, overlook the potential of fine-grained context modeling and disentangled fine-tuning objectives in enhancing MLLMs'
retrieval performance, particularly for complex tasks such as long-text-to-image
retrieval, visual dialog
retrieval, and composed image
retrieval (CIR). Therefore, in this work, we propose an automated fine-grained
multimodal quintuple
dataset construction pipeline and a novel two-stage fine-grained
multimodal fine-tuning strategy. The
dataset generation pipeline produces a comprehensive CIR
dataset with fine-grained image captions and modification text, facilitating fine-grained context modeling. Beyond the previously entangled fine-tuning paradigm, our approach separates the fine-tuning process into two distinct stages: (1) fine-grained context
reasoning-oriented fine-tuning and (2) fine-grained
retrieval-oriented fine-tuning. These stages aim to sequentially enhance the model's context understanding and query-target
alignment capabilities, thereby improving
retrieval performance. Extensive experiments across five
datasets encompassing diverse and complex image
retrieval tasks demonstrate the remarkable superiority of our method over existing approaches in zero-shot
retrieval settings, even with a more lightweight MLLM backbone compared to those methods.
Read the full abstract ▾
All authors Bohan Hou, Haoqiang Lin, Xuemeng Song, Haokun Wen, Meng Liu, Yupeng Hu, Xiangyu Zhao
Computer Vision
LAST: The Last Query Token Guides Visual Token Pruning for Edge-Cloud Collaborative MLLM Inference
Feng Yang, Xinrui Ju, Keyang Zhang et al. (+6 more)
Multimodal foundation models are reshaping edge-cloud visual intelligence from task-specific feature pipelines into
token-based interfaces, where edge devices encode visual inputs into
tokens for a general-purpose cloud MLLM. However, dense visual-
token sequences increase cloud-side inference costs. Existing
pruning methods mainly target centralized inference:
vision-driven methods can operate before cloud execution but are typically query-agnostic, whereas query-guided methods often rely on internal states of the target MLLM and cannot determine
token relevance before transmission. Compact guidance models offer an alternative, but existing designs may require costly
attention aggregation or auxiliary generation. We propose LAST, a training-free framework for query-dependent visual
token pruning in edge-cloud collaborative MLLM inference. LAST uses a compact edge-side VLM as a guidance proxy and derives a lightweight importance signal from the last query
token's
attention to visual
tokens. Under causal
attention, the last query
token can attend to the full visual sequence and the entire query context, enabling query-aware
pruning without cloud-model access, autoregressive generation, or costly aggregation over multiple query positions. LAST then retains a diverse set of query-relevant visual
tokens under a fixed
token budget. We evaluate LAST on 11
multimodal benchmarks under multiple
token budgets against
pruning methods with different guidance strategies. Experiments show that LAST consistently achieves the strongest performance, preserving 95.4% of the full-
token accuracy while retaining only 12.5% of the visual
tokens, with low edge-side selection overhead and reduced cloud-side computation.
Read the full abstract ▾
All authors Feng Yang, Xinrui Ju, Keyang Zhang, Xiandong Meng, Rongqun Lin, Howard Leung, Shiqi Wang, Haoliang Li, Chris Xing Tian
Computer Vision
ARD-REFSM: Enhancing Reflection Symmetry Detection with Asymmetric Denoising and Rotation Equivariance
Dongfu Yin, Rourou Su, Cong Zhao et al. (+1 more)
Reflection
symmetry detection remains challenging due to interference from asymmetric regions and arbitrary orientations of symmetric patterns. Asymmetric regions introduce background clutter that disrupts symmetric pattern matching, whereas conventional convolutional neural networks lack rotation equivariance, leading to inconsistent feature representations under rotational transformations. To address these issues, we propose an Asymmetric Region Denoising (ARD) module and a Rotation Equivariant Feature Similarity Matching (REFSM) module. The ARD module suppresses asymmetric interference to refine symmetric patterns, while the REFSM module enhances rotation equivariance through feature similarity matching between original and rotated images. Specifically, our dual-input REFSM framework leverages rotation loss to maximize consistency between the score maps of original and rotated images, thereby enabling precise prediction of rotation-equivariant
symmetry axes. Furthermore, we introduce GMSYM, a new
benchmark dataset that categorizes images into diverse scenarios and incorporates various interferences to address the limitations of existing reflection
symmetry detection
benchmarks. Extensive experiments on four standard
datasets (DENDI, NYU, LDRS, SDRW) and our proposed GMSYM
dataset demonstrate that our method achieves state-of-the-art performance in both accuracy and robustness.
Read the full abstract ▾
All authors Dongfu Yin, Rourou Su, Cong Zhao, Fei Yu
Computer Vision · Cross-listed from cs.LG
ODEWorld: A Continuous Predictive Architecture via Physical-Time Flow
Dongxiu Liu, Haoyi Niu, Peng Cheng et al. (+5 more)
In the physical world we inhabit, space and time are fundamentally continuous. However, existing machine learning paradigms for world modeling are largely confined to discrete-time prediction, thereby exhibiting significant inefficiency in capturing the dynamics of physical world. We introduce Physical-Time Flow (PT-Flow), a novel approach that learns a continuous latent velocity field operating in physical time. Crucially, the underlying dynamics of sequential data are parameterized by an ordinary differential equation (ODE) embedded in a well-structured representation space. Under this paradigm, the prediction of future can be recast as temporal integration via an ODE solver in the compressed latent space. Building upon PT-Flow, we construct ODEWorld, a continuous-time latent world model that is both efficient and versatile. By extracting time-variant features and enforcing ODE properties on both the dynamical representation space and the latent velocity field, ODEWorld effectively addresses the long-standing representation collapse issue in latent world model literature. This also enables high-quality image reconstruction even after long-horizon prediction. Moreover, its continuous nature allows for arbitrary temporal resolution and even backward prediction, which is impossible for most discrete-time models. Lastly, ODEWorld can provide rich planning-oriented information to facilitate downstream policy learning. Comprehensive experiments demonstrate that ODEWorld successfully reconciles planning-conducive dynamics abstraction with visual realism, excelling in both
video generation and robotic control. Project page: https://dstate.github.io/odeworld_website/.
Read the full abstract ▾
All authors Dongxiu Liu, Haoyi Niu, Peng Cheng, Yuan Gao, Xirui Kang, Sangli Teng, Koushil Sreenath, Xianyuan Zhan
Computer Vision
One Patch Is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-Based Scene Text Spotting
Rui Tang, Wentao Yang, Peirong Zhang et al. (+4 more)
Scene text spotting requires high-precision
alignment between textual recognition and spatial localization. While visual-
token grounding has emerged as a promising formulation for
Multimodal Large
Language Models (MLLMs), the previous multi-patch paradigm often introduces redundant noise and localization ambiguity, particularly for dense or small text instances. To address this, we propose Single-Patch Text Spotting (SPaTS), a
vision-centric framework that routes each text instance through a single anchor visual
token and then recovers geometry via full-image refinement. To accurately identify this anchor without oracle labels, we introduce Single-Patch Selective Optimization (SPaSO), a reinforcement learning framework that optimizes discrete visual-
token selection using patch-level rewards. To further improve representation robustness and localization precision, we introduce Directional
Embedding Alignment (DEA) to suppress unstable norm bias by decoupling feature magnitude and direction, and Patch-Enhanced Decoding (PED) to fuse the routed anchor with
language semantics and cross-attend over the full-image feature map for geometry-aware boundary regression beyond coordinate-space surrogates. Extensive experiments demonstrate that SPaTS consistently and significantly outperforms both frontier closed-source MLLMs and OCR MLLMs. Code is available at https://github.com/eeNickTang/SPaTS.
Read the full abstract ▾
All authors Rui Tang, Wentao Yang, Peirong Zhang, Yongxin Shi, Shun Zhang, Huiguo He, Lianwen Jin
Computer Vision
CoRE-UIR: Prior-guided common and residual experts for efficient all-in-one remote sensing image restoration
Zaiyan Zhang, Qiangqiang Yuan, Jie Li et al. (+5 more)
Remote sensing images acquired by unmanned aerial vehicles (
UAVs) and satellites are often degraded by adverse weather, illumination variation, and imaging artifacts, which may co-occur and jointly induce global distribution shifts and local structural corruption. Although All-in-One image restoration offers an appealing unified alternative to task-specific pipelines, existing methods still suffer from weak or implicit degradation cues and parameter redundancy caused by full-rank multi-expert designs with overlapping restoration behaviors. We propose CoRE-UIR (Common and Residual Experts for Universal Image Restoration), a prior-guided global-local framework centered on the Common-and-Residual Expert Block (CoRE). CoRE explicitly decomposes restoration capacity into a common dense expert for degradation-invariant restoration and low-rank residual experts for degradation-specific compensation, enabling adaptive specialization without redundant expert replication. Built on this design, Degradation Prior
Embedding (DPE) adapts frozen CLIP features into an explicit restoration-oriented prior, while Global Feature Modulation (GFM) aligns global feature statistics before local residual compensation. We also construct MDVD-108K (Multi-Degradation VisDrone), a large-scale
UAV restoration
dataset covering both single and compound degradations, together with a real-world test set. Extensive experiments on multiple
datasets show that CoRE-UIR improves the overall average PSNR by 1.05 dB while running 11.83$\times$ faster and reducing peak
memory by 85.3% relative to the strongest baseline, BaryIR, thereby maintaining a favorable quality-efficiency trade-off. Evaluations on downstream tasks and unseen degradation also validate the generalizability of CoRE-UIR. The code and
dataset will be released at https://github.com/zzaiyan/CoRE-UIR.
Read the full abstract ▾
All authors Zaiyan Zhang, Qiangqiang Yuan, Jie Li, Ziyang Lihe, Yu Wan, Yuzeng Chen, Xin Su, Liangpei Zhang
Computer Vision
Unifying Adversarially Robust Model Experts in Vision-Language Models
Nguyen Duc Thai, Junhao Dong, Sua Qi Rong et al. (+2 more)
Vision-
language models (VLMs), such as CLIP, are vulnerable to
adversarial attacks, posing a serious problem for real-life applications and deployment.
Adversarial fine-tuning emerges as a prominent defense method; however, different fine-tuning strategies often produce specialized models with distinct robustness characteristics. Each fine-tuned model in turn thrives in some evaluation settings but falters on others, limiting their defensive capabilities. We refer to these specialized fine-tuned models as robust model experts and propose a collaborative
adversarial fine-tuning framework: CARE - Collaborative
Adversarial Robustness fine-tuning using
Embedding alignment. CARE maintains multiple experts during training, enables knowledge exchange through
embedding-space harmonization, and consolidates the learned knowledge into a single unified robust model. Experts benefit from one another while preserving their individual specializations, enabling the final model to inherit complementary robustness properties. In this paper, we demonstrate CARE on two different
adversarial fine-tuning strategies with complementary robustness behaviors. Extensive experiments on classic image classification and downstream
vision-
language tasks display the effectiveness of our approach, with CARE being able to outperform individually learned model experts. The results suggest that collaborative learning across model experts is a promising direction for improving
adversarial robustness.
Read the full abstract ▾
All authors Nguyen Duc Thai, Junhao Dong, Sua Qi Rong, Hua Yu, Yew-Soon Ong
Computer Vision · Cross-listed from cs.AI
MMHBench: A Multi-Perspective Benchmark for Mental Health Understanding in Long-Form Videos
Jinpeng Hu, Erqiang Wang, Shan Wang et al. (+4 more)
Mental health understanding in long-form
videos requires nuanced
reasoning over observable behavior, interpersonal context, and latent psychological states. Existing
benchmarks largely reduce this task to coarse-grained classification, providing limited insight into whether models truly understand psychological phenomena or rely on superficial correlations. To address this limitation, we introduce MMHBench, a comprehensive
multimodal benchmark for multi-perspective mental health understanding, comprising 268 long-form
videos and 2,184 carefully curated questions. MMHBench organizes the evaluation into two complementary settings: (1) third-person assessment, consisting of 605 questions that focus on the interpretation of observable behaviors and
multimodal evidence, and (2) first-person perspective-taking, comprising 1,579 questions that require perspective-conditioned
reasoning to identify the interpretation of the mental state supported by the available
multimodal evidence. We propose a Multi-
Agent Question Generation (MAQG) framework that simulates diverse social roles to synthesize questions from multiple perspectives. The generated questions are refined through multi-role feedback and iterative optimization, followed by expert-guided verification to ensure quality and validity. Extensive evaluation of 22 representative
multimodal large
language models (MLLMs), spanning both open-source and leading closed-source models, demonstrates that long-form
video mental health understanding remains highly challenging.
Read the full abstract ▾
All authors Jinpeng Hu, Erqiang Wang, Shan Wang, Zhuo Li, Peipei Song, Xun Yang, Meng Wang
Computer Vision
DECODE: Tackling Representation and Decision Degradation in Continual AI-Generated Image Detection
Zihao Cai, Xinghan Li, Ruiyan Yang et al. (+3 more)
As generative models continue to evolve, AI-generated image detectors must incrementally adapt to emerging generative domains while preserving knowledge acquired from previous ones. This continual learning setting is particularly challenging because forensic traces are often subtle and generator-specific, making detectors highly vulnerable to catastrophic forgetting. Existing methods primarily address this problem by stabilizing feature representations, implicitly treating forgetting as a representation-level issue. In this paper, we show that this perspective is incomplete. We demonstrate that even when feature representations remain discriminative, the decision boundary can progressively drift as the classification head is continually optimized on new domains. These two effects jointly give rise to a compound failure mode, termed Dual Degradation. To overcome this challenge, we propose DECODE, a decoupled continual detection framework that jointly mitigates representation- and decision-level forgetting. Specifically, we introduce Subspace Diversity Regularization (SDR) to preserve diverse forensic representations and Closed-Form Decision
Alignment (CDA) to recalibrate the shared classification head after each adapter merge without manual hyperparameter tuning. Extensive experiments on 19 generative domains show that DECODE achieves an average accuracy of 99.36% with only 0.39% forgetting, while further generalizing to 11 unseen generators with 95.36% accuracy.
Read the full abstract ▾
All authors Zihao Cai, Xinghan Li, Ruiyan Yang, Xue Song, Haijun Shan, Jingjing Chen
Computer Vision
Learning to Understand Body Language from Flight through Robust 3D Avatar Placing
Dragos Costea, Alina Marcu, Cristina Lazar et al. (+1 more)
Perceiving human motion and intent at long range is a prerequisite for socially intelligent aerial
robots, yet the data to learn it barely exists. We introduce Drones2BodyLanguage, a
dataset grounding human motion in real
UAV footage: avatars manifesting ten communicative intents are placed into unmodified 4K drone scenes with metrically correct position, scale and orientation, maintained over hundreds of frames of camera motion. Enabling it is a lightweight
geometric world model of the local scene - semantically selected anchors lifted to 3D through streaming
monocular depth - in which a placement point is predicted as an affine anchor combination with provably rigid-invariant weights, and re-rendered under an SVD-fitted ground rotation. Across twelve architectures on scene- and motion-disjoint splits, training on placed data lifts mean intent accuracy by a wide margin for real, retargeted and generated motion alike, with gains confirmed on two in-the-wild scenes.
Read the full abstract ▾
All authors Dragos Costea, Alina Marcu, Cristina Lazar, Marius Leordeanu