Computer Vision
Jiazi Bu, Pengyang Ling, Yujie Zhou et al. (+10 more)
Text-Image-to-
Video (TI2V) models are an emerging unified architecture, where a single model simultaneously supports text-to-
video (T2V) and image-to-
video (I2V) generation. Given a high-quality first frame or a detailed textual
prompt, TI2V models unlock substantially better visual quality than their T2V mode, raising a natural question: can the capability elicited by such privileged conditions be internalized into the model's own base generation ability? A common approach toward this goal is model self-
distillation. However, the most straightforward solution, supervised fine-tuning, follows an off-policy strategy: its supervision is confined to teacher-generated endpoints from a fixed offline distribution rather than student-visited states, lacking precise correction tailored to the evolving policy. Recent on-policy
distillation methods instead suffer from condition-state mismatch, where supervision is steered toward the given first frame instead of the student's actual content, misleading the correction. To achieve self-
distillation that absorbs the teacher's privileged prior while retaining precise policy correction, in this work, we propose Hybrid-Policy Self-
Distillation (HPSD), a novel self-
distillation framework where a single TI2V model acts as both teacher and student under different conditions: the teacher operates in TI2V mode with a high-quality first frame and an enhanced
prompt, while the student runs in the base T2V mode with only the vanilla
prompt. Specifically, the student inherits off-policy teacher trajectory points as anchors, locally refines them toward its own policy, and finally receives velocity-level supervision on these self-generated roll-outs. Extensive experiments demonstrate that HPSD significantly improves T2V performance while also delivering notable TI2V gains, effectively strengthening the model's base generation ability.
Read the full abstract ▾
All authors Jiazi Bu, Pengyang Ling, Yujie Zhou, Yibin Wang, Yuhang Zang, Xuanlang Dai, Shengyuan Ding, Tianyi Wei, Xiaohang Zhan, Jiaqi Wang, Tong Wu, Dahua Lin, Xingang Pan
Computer Vision
Fidelity-Constrained Anchoring for Black-Box Denoisers
Masaki Satoh
We propose a fidelity-constrained framework that anchors the output of a black-box denoiser to its input without retraining and with little additional computation. The method linearly blends the denoised image with the input and selects the maximum blending factor that satisfies a prescribed local fidelity constraint using Peak Signal-to-Noise Ratio (PSNR) or Structural Similarity Index (SSIM). For PSNR control, a closed-form solution is obtained under a local constant-blending assumption. For SSIM control, we derive a tractable formulation based on inverse SSIM under the same assumption and solve it efficiently using iterative root finding. Experiments on DIV2K images with synthetic Gaussian noise and outputs from Real-ESRGAN and a non-local means denoiser show that the proposed anchoring strategy provides effective fidelity control while balancing denoising performance and statistical naturalness, as measured by the excess kurtosis of residual noise. In particular, SSIM-based anchoring yields more consistent behavior across noise levels than PSNR-based anchoring.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
ProME: Prototype-Margin Environments with Repair-Aware Selection for Group-Robust Learning
Qianqian Wang, Yunshan Li, Dawei Huang et al. (+2 more)
Group-robust learning is crucial for maintaining accuracy on rare subpopulations when training-group labels are unavailable. However, existing methods often infer environments from a separate reference model and select representations before fitting the classifier used at deployment, leaving both decisions misaligned with the deployed predictor. In this work, we formulate group robustness without training-group labels as the endogenous environments with repair-aware selection (ERAS) problem, and propose ProME (Prototype-Margin Environments) to align both decisions with the deployed predictor. ProME splits prototype margins at their median to construct approximately balanced environments along the training trajectory, and fits a group-balanced linear head on group-annotated validation data to rank the resulting predictors by validation worst-group accuracy. We theoretically bound the worst risk across the inferred environments for a fixed predictor and partition, showing that this bound transfers to the oracle groups under an explicit
alignment condition. Extensive experiments show that prototype margins enrich shortcut-conflicting examples, classifier repair reshapes candidate evaluation, and ProME achieves the highest average worst-group accuracy among the compared methods with the same group-label access.
Read the full abstract ▾
All authors Qianqian Wang, Yunshan Li, Dawei Huang, Wenwu Gong, Lili Yang
Computer Vision
SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpainting
Zian Yang
Sketch-guided image inpainting provides intuitive structural control, yet real sketches often mix reliable global intent with locally crowded, displaced, incomplete, or deliberately unconventional strokes. Existing approaches typically either retain the input sketch as a fixed condition throughout denoising or refine it into a clean structure before RGB synthesis. The former assumes uniformly reliable strokes and can propagate local errors throughout generation; the latter must resolve ambiguous structure before emerging appearance and semantic context become available. We propose SketchSense, a framework that interprets imperfect sketch guidance by synchronously denoising interacting RGB and structure streams. Bidirectional
Attention Fusion couples appearance generation with structural recovery, producing a refined structure that exposes the model's evolving sketch interpretation. A phrase-level objective aligns the semantic grounding of the two streams. Sketch-Aware Spatial Regulation further adapts sketch use to local generation states by modulating
attention and the fusion process, while an optional signed prior injects preserve-versus-correct intent into feature representations and
attention behavior. Experiments on natural and structurally complex imagery show substantial gains over existing methods in both restoration quality and structural fidelity.
Read the full abstract ▾
Computer Vision
A Controlled Study of Self-Supervised Image and Video Pretraining under Limited Resources
Brunó B. Englert and Gijs Dubbelman
Visual foundation models are a cornerstone of image and
video understanding but typically require large amounts of data and computation. The current scale required for pretraining visual foundation models may be unsustainable or unnecessary, and significant benefits arise when effective models can be obtained with fewer resources. To better understand how self-supervised learning (SSL) objectives behave under resource constraints, we conduct a controlled study of image and
video SSL objectives under matched data, architecture, and compute budgets. We compare contrastive, reconstruction, feature-prediction, and
diffusion objectives and evaluate both standalone and jointly trained image-
video SSL formulations across a diverse set of image and
video understanding tasks. Our results show that DINOv2-style pretraining consistently provides the strongest overall performance under limited resources. Furthermore, combining DINOv2 with
video SSL objectives such as VideoMAE substantially improves image classification and
segmentation performance, but degrades
video tracking and camera-pose estimation performance, revealing an important tradeoff between semantic and
geometric representation learning. These findings suggest that combining image and
video SSL objectives can be beneficial in resource-limited settings, while highlighting the need for improved methods that better balance semantic, temporal, and
geometric supervision.
Read the full abstract ▾
Computer Vision
TRAPSBench: Vision-Language Models Encode but Fail to Express Epistemic Restraint
Fnu Pramono, John Cai and Sourabh Kulkarni
When visual evidence is occluded or chaotic, models should abstain. In this paper, we show that
Vision-
Language Models (VLMs) can internally distinguish when abstention is required, but fail to express it anyway. We introduce TRAPSBench, a procedurally generated
video benchmark of 1,404 matched physics pairs in which a single targeted change renders the outcome undeterminable from the visual evidence. Furthermore, we introduce Penalized Epistemic Calibration Score (PECS), a new robust metric that requires models to both answer correctly when the outcome is knowable, and abstain when the outcome is not. Across 16 VLMs spanning five families, spontaneous restraint is poor: the best PECS is 0.292. The bottleneck is expression, not perception: linear probes decode answerability from hidden states at up to 0.91 AUROC across physics domains; steering a single-layer void direction causally induces or suppresses abstention. Our results replicate across three open-weight families (Qwen, Gemma, LLaVA). The failure is also more pronounced in visual than textual uncertainty: models detect textual impossibility about 4x more readily than missing visual evidence. Closing this representation–output gap likely requires output-stage interventions.
Read the full abstract ▾
Computer Vision
Splat-based Metal Artifact Reduction in Cone-Beam CT via Polychromatic Modeling
Kiseok Choi, Inchul Kim, Jaemin Cho et al. (+2 more)
Cone-beam computed
tomography (CBCT) enables volumetric reconstruction from X-ray projections, but suffers from severe artifacts–especially beam hardening–when imaging materials with high attenuation such as metals. These artifacts arise from the polychromatic nature of X-rays and are not properly addressed by conventional monochromatic reconstruction
algorithms. While recent neural representation-based methods offer improved reconstruction quality, they are computationally expensive and often impractical for deployment. We propose a novel physics-inspired, self-calibrating metal artifact reduction method that efficiently reconstructs 3D CBCT volumes while correcting beam hardening artifacts. Our method integrates a polychromatic X-ray projection model, material-dependent attenuation profiles, and system response modeling into a Gaussian Splatting framework. Unlike prior work, we eliminate the need for manual metal masks or strong prior assumptions, and we optimize both reconstruction parameters and X-ray spectral characteristics jointly during training. We further introduce a high-fidelity synthetic CBCT
dataset generation pipeline validated on Monte-Carlo x-ray simulation toolbox and release new
datasets with severe metal-induced artifacts to support the community. This is the first splat-based method for reducing beam hardening in CBCT. Extensive experiments on both synthetic and real-world
datasets demonstrate that our method outperforms state-of-the-art approaches in artifact suppression and reconstruction accuracy.
Read the full abstract ▾
All authors Kiseok Choi, Inchul Kim, Jaemin Cho, Hyeongjun Cho, Min H. Kim
Computer Vision
Less Annotation, More Interpretation: Prior-Guided Concept Bottleneck Models for Interpretable Cancer Imaging Diagnosis
Baoqiang Ma and Kenneth Gilhuijs
Concept bottleneck models (CBMs) can improve the transparency of cancer image diagnostic prediction by expressing predictions through radiological concepts. However, their dependence on instance-level concept annotations limits practical applicability. We propose a prior-guided hybrid CBM that integrates limited concept annotations, class-conditional concept distribution matching on unannotated patients, and prior initialization of the concept-to-diagnosis head. We evaluate the method on CBIS-DDSM mammographic masses and calcifications and LIDC-IDRI pulmonary nodules across 0-100% concept annotation. In the clinically relevant 0-20% annotation regime, the hybrid CBM consistently improves mean concept AUC over a matched standard CBM, while maintaining diagnostic performance close to black-box models. At 10% annotation specifically, concept AUC increases from 0.619 to 0.741 for masses, from 0.650 to 0.787 for calcifications, and from 0.597 to 0.642 for pulmonary nodules. Ablation experiments identify prior initialization as the main component contributing to improved concept detection, likely by stabilizing the concept-to-diagnosis head. Zero-shot VLMs remain insufficient for reliable fine-grained tumor-level concept prediction. These findings suggest that structured priors can substantially reduce the annotation burden of interpretable cancer imaging models.
Read the full abstract ▾
Computer Vision
Geometry-Grounded Unified 3D Perception for Autonomous Driving
Longfei Xu, Xiaohui Wang, Zehao Huang et al. (+4 more)
Camera-based autonomous driving perception requires a shared representation that preserves metric 3D structure across synchronized multi-camera streams. However, existing image-based frameworks often rely on backbones pretrained for semantic recognition, and introduce 3D geometry through downstream task-specific modules. As a result, their shared representations may fail to preserve explicit metric geometry and consistent 3D scene structure. In this paper, we present a Geometry-grounded Unified 3D Perception (GeoUP) framework that adapts the reconstruction-oriented latent of VGGT to calibrated, streaming multi-camera driving scenes. GeoUP factorizes cross-image interaction into self, temporal, and view
attention to capture structurally distinct temporal and cross-view correspondences. It further injects calibration-aware raymap encodings to provide metric scale and camera geometry. The resulting geometry-grounded latent is decoded for metric depth estimation, 3D object detection, and semantic occupancy prediction, corresponding to surface-, instance-, and volume-level readouts of the same 3D scene. Through joint multi-task and multi-
dataset training, GeoUP effectively leverages heterogeneous annotations and generalizes across diverse sensor configurations and perception ranges. Extensive experiments on nuScenes, Argoverse 2, Waymo, KITTI, and DDAD demonstrate that GeoUP achieves SOTA performance across detection, occupancy, and depth estimation. These results validate the effectiveness of geometry-grounded representations for unified 3D driving perception.
Read the full abstract ▾
All authors Longfei Xu, Xiaohui Wang, Zehao Huang, Han Li, Ya Yang, Naiyan Wang, Si Liu
Computer Vision
MergeOver: Post-Training Token Merging for Recursive Vision Transformers
Junseo Kim, Uraz Odyurt and Amirreza Yousefzadeh
Vision Transformers (ViTs) demonstrate exceptional performance in computer
vision but suffer from large parameter counts and quadratic computational complexity, severely limiting their deployment on resource-constrained edge hardware. While recursive weight-sharing reduces parameter counts and
token merging mitigates computational and
memory bottlenecks, integrating these two paradigms without costly retraining is non-trivial, leaving this intersection largely unexplored. We propose MergeOver, a post-training approach that integrates
Token Merging (ToMe) into the recursively weight-shared Sliced Recursive
Transformer (SReT). Through an Unmerge tracking stack, constraint-safe merge-rate adjustment, and synchronised
token-mass tracking across spatial permutations, MergeOver resolves the spatial and merging constraints of this integration. We further employ a stage-wise single-shot schedule that performs
token reduction at the first block of each stage and maintains a fixed sequence length throughout its subsequent recursive iterations. Benchmarked on ImageNet-1K, our selected configuration reduces top-1 accuracy by 1.47 percentage points. On the GPU, it reduces peak activation
memory by 37.3% and 38.4% at batch sizes 1 and 16, while throughput decreases by 21.7% at batch size 1 but increases by 21.7% at batch size 16. On a Raspberry Pi 5 (ARM CPU), it reduces latency by 2.4% and 17.6% at batch sizes 1 and 16. These results show that MergeOver can recover a meaningful part of the throughput and
memory cost that recursive weight-sharing introduces, without retraining, and provides a baseline for combining
token merging with hierarchical recursive
transformers.
Read the full abstract ▾
Computer Vision
Predicting Signed Distance Functions for Visual Instance Segmentation
Emil Brissman, Joakim Johnander and Michael Felsberg
Visual instance
segmentation is a challenging problem and becomes even more difficult if objects of interest varies unconstrained in shape. Some objects are well described by a rectangle, however, this is hardly always the case. Consider for instance long, slender objects such as ropes. Anchor-based approaches classify predefined bounding boxes as either negative or positive and thus provide a limited set of shapes that can be handled. Defining anchor-boxes that fit well to all possible shapes leads to an infeasible number of prior boxes. We explore a different approach and propose to train a neural network to compute distance maps along different directions. The network is trained at each pixel to predict the distance to the closest object contour in a given direction. By pooling the distance maps we obtain an approximation to the signed distance function (SDF). The SDF may then be thresholded in order to obtain a foreground-background
segmentation. We compare this
segmentation to foreground
segmentations obtained from the state-of-the-art instance
segmentation method YOLACT. On the COCO
dataset, our
segmentation yields a higher performance in terms of foreground intersection over union (IoU). However, while the distance maps contain information on the individual instances, it is not straightforward to map them to the full instance
segmentation. We still believe that this idea is a promising research direction for instance
segmentation, as it better captures the different shapes found in the real world.
Read the full abstract ▾
Computer Vision
QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing
Shubin Lu, Jiaqi Yin and Yihao Huang
Typographic attacks pose a critical threat to
vision-
language models (VLMs) by injecting misleading text into images and causing models to rely on
adversarial textual cues rather than visual evidence. Existing defenses often require model-specific modifications, additional training, or access to internal model components, limiting their applicability to modern closed-source VLMs. In this paper, we propose QuISE, a model-agnostic, training-free black-box defense based on query-irrelevant semantic editing. QuISE first identifies text regions likely to affect the current query through influence-aware text localization. QuISE then replaces these regions with two semantically distinct replacement texts that are irrelevant to both the query and the image. The final answer is determined by answer consistency across the edited images. Extensive experiments on three typographic-attack
benchmarks, four attack settings, and four VLMs show that QuISE consistently improves defended accuracy. QuISE achieves a recovery rate of 67.9-75.0% with a harm rate of 0.5-1.1%.
Read the full abstract ▾
Computer Vision
Fast Iterative Five point Relative Pose Estimation
Johan Hedborg and Michael Felsberg
Robust estimation of the relative pose between two cameras is a fundamental part of Structure and Motion methods. For calibrated cameras, the five point method together with a robust
estimator such as RANSAC gives the best result in most cases. The current state-of-the-art method for solving the relative pose problem from five points is due to Nister [9], because it is faster than other methods and in the RANSAC scheme one can improve precision by increasing the number of iterations. In this paper, we propose a new iterative method, which is based on Powell's Dog Leg
algorithm. The new method has the same precision and is approximately twice as fast as Nister's
algorithm. The proposed method is easily extended to more than five points while retaining a efficient error metrics. This makes it also very suitable as an refinement step. The proposed
algorithm is systematically evaluated on three types of
datasets with known ground truth.
Read the full abstract ▾
Computer Vision
Weitao Chen, Hu Jiaxin, Xie Tianyidan et al. (+15 more)
Recent advances in
Multimodal Large
Language Models (MLLMs) have led to substantial progress in
video understanding, accompanied by a growing number of long
video benchmarks. However, existing
benchmarks rely predominantly on web-sourced
videos that lack inter-clip spatiotemporal continuity, making it difficult to assess whether models can maintain consistent
memory across days or weeks of real-world experience. We introduce EgoMonth, the first month-level
egocentric video understanding
benchmark. EgoMonth comprises over 300 hours of first-person daily-life recordings from 20 participants spanning 20 to 120 days, paired with 1,443 human-crafted multiple-choice question-answer pairs. We design a cognitively grounded 14-task evaluation framework organized into three hierarchical cognitive levels: Schema Consolidation, Episodic Indexing, and Cascading
Reasoning. Evaluation of state-of-the-art open-source and closed-source MLLMs reveals that even the best-performing model, Gemini 2.5 Pro, achieves only 71.8% macro-average accuracy, remaining 22.4 percentage points below the corrected human baseline of 94.2%. Several models perform near or below the 25% chance level on tasks such as Route
Reasoning, Cross-view Spatial
Reasoning, and Direction Judgement, while even the strongest closed-source model remains substantially below human performance. These results indicate that current MLLMs function as lossy summarizers rather than faithful memorizers, highlighting the need for architectures with genuine long-term spatiotemporal
memory.
Read the full abstract ▾
All authors Weitao Chen, Hu Jiaxin, Xie Tianyidan, Yang Li, Yuyi Qian, Banghao Xu, Ziheng Tang, Shenyi Wang, Mingyue Yu, Duo Li, Jiacheng Shi, Gao Wang, Zhan Xu, Zhicheng Qiu, Xuanfu Li, Jian Yang, Lanjun Wang, Zili Yi
Computer Vision
Towards Physics-Faithful Generation of Scientific Diagrams
Minghui Zhang, Jinxin Shi, Yifan Chang et al. (+12 more)
Text-to-image generation has reached photorealistic quality, yet state-of-the-art systems remain unreliable at producing scientific diagrams, whose value depends not on appearance but on physical faithfulness: correct force directions, valid coordinate systems, consistent thermodynamic states, and equations matching the depicted scenario. Trained on web imagery with physically shallow captions, generic models produce diagrams that look plausible but are physically wrong, harmful in education and scientific communication. We present Princigram, a physics-faithful scientific-diagram generator, and its data pipeline. Our central advance is Structured Physical Chain-of-Thought (SP-CoT): a per-subdiscipline schema that decomposes a physics diagram into an explicit multi-step
reasoning chain across six subdisciplines, from scene identification through force or process analysis to governing laws and synthesis. Unlike free-form chain-of-thought, SP-CoT follows a fixed schema with strict fidelity rules that separate visually grounded facts from physically inferred
reasoning and type all mathematics symbolically; it serves both as dense training supervision and, at inference, as a structured "thinking"
prompt. With it we curate and structurally annotate 4.3 million physics images, of which 115,037 carry expert-level annotation, and adapt a unified
multimodal backbone. We further introduce VeriphyT2IBench, whose questions are derived from each held-out diagram's own structured annotation: each diagram becomes an item-specific bank of binary questions about its objects, forces, and states, so a judge model's score decomposes into named physical facts rather than one holistic number. On the physics subset of GenExam and on VeriphyT2IBench, Princigram shows that explicit physics-structured supervision improves the physical faithfulness of generated scientific diagrams.
Read the full abstract ▾
All authors Minghui Zhang, Jinxin Shi, Yifan Chang, Liangliang Zhao, Yuandong Pu, Qian Yu, Ming Hu, Hanxiao Zhang, Yun Gu, Yirong Chen, Yu Qiao, Bo Zhang, Xiangchao Yan, Bin Fu, Yihao Liu
Computer Vision
Online Learning of Correspondences between Images
Michael Felsberg, Fredrik Larsson, Johan Wiklund et al. (+2 more)
We propose a novel method for iterative learning of point correspondences between image sequences. Points moving on surfaces in 3D space are projected into two images. Given a point in either view, the considered problem is to determine the corresponding location in the other view. The geometry and distortions of the projections are unknown as is the shape of the surface. Given several pairs of point-sets but no access to the 3D scene, correspondence mappings can be found by excessive global optimization or by the fundamental
matrix if a perspective projective model is assumed. However, an iterative solution on sequences of point-set pairs with general imaging geometry is preferable. We derive such a method that optimizes the mapping based on Neyman's chi-square divergence between the densities representing the uncertainties of the estimated and the actual locations. The densities are represented as channel vectors computed with a basis function approach. The mapping between these vectors is updated with each new pair of images such that fast convergence and high accuracy are achieved. The resulting
algorithm runs in real-time and is superior to state-of-the-art methods in terms of convergence and accuracy in a number of experiments.
Read the full abstract ▾
All authors Michael Felsberg, Fredrik Larsson, Johan Wiklund, Niclas Wadströmer, Jörgen Ahlberg
Computer Vision
RbFT-Net: Rectify-Before-Fuse Temporal Radar Anchors for 4D Radar-Camera Depth Completion
Wentao Zhao, Shouxuan Wu, Yongtao Cen et al. (+3 more)
Dense metric depth prediction from cameras and millimeter-wave radar offers a cost-effective sensing solution for autonomous systems. However, radar measurements are inherently sparse and susceptible to clutter, multipath reflections, and projection errors. While aggregating multiple radar frames provides denser metric cues, it also introduces temporal misalignment and dynamic-object interference. Directly propagating such unreliable measurements can therefore corrupt large regions of the predicted depth map. To address this issue, we propose RbFT-Net, an end-to-end rectify-before-fuse framework for multi-frame 4D radar-camera depth completion. Rather than assuming accumulated radar returns to be accurate, RbFT-Net treats them as noisy temporal anchor candidates. An image-conditioned rectification module jointly corrects their image-plane locations and metric depths while estimating pointwise reliability. The rectified anchors are then selectively propagated before high-level multi-modal fusion, suppressing the influence of unreliable measurements. Experiments on ZJU-4DRadarCam and a newly collected 4D radar-camera-
LiDAR dataset show that RbFT-Net consistently outperforms the evaluated independent radar-camera methods and remains competitive with plug-in pipelines using auxiliary
monocular depth models. Cross-platform evaluation and component analyses further support the effectiveness of the proposed rectification and reliability-aware propagation strategy.
Read the full abstract ▾
All authors Wentao Zhao, Shouxuan Wu, Yongtao Cen, Tianchen Deng, Yuyang Zhang, Jingchuan Wang
Computer Vision · Cross-listed from cs.RO
Semantic Radiance Fields as Simulators for Spatial Reasoning in Real-World Scenes
Nico Heider, Michał Jan Włodarczyk, Katarzyna Wasielewska-Michniewska et al. (+5 more)
Training and evaluating spatial
reasoning in
embodied agents requires diverse environments that are both geometrically faithful and semantically queryable. Synthetic simulators offer ground truth semantics but sacrifice realism; simulators based on reconstructions of real-world environments have realistic appearance but lack ground truth semantics by default. We propose using Semantic Radiance Fields (SRF) as simulators for spatial
reasoning agents. SRFs are a representation that unifies these requirements by lifting 2D semantic
segmentations from pretrained
vision models into a 3D radiance field that jointly encodes geometry, appearance, and per-class semantic identity. The resulting fields are reconstructed from posed RGB captures of real scenes and support novel-view synthesis, semantic and free-space queries within a single grounded representation. This enables the efficient generation of diverse real-world environments to train and evaluate spatial
reasoning models. As an example application, we outline an SRF-driven simulator for an orchard apple-reaching task, in which the radiance field supplies camera rendering, semantic ground truth, and occupancy queries to a physics engine.
Read the full abstract ▾
All authors Nico Heider, Michał Jan Włodarczyk, Katarzyna Wasielewska-Michniewska, Przemysław Hołda, Martin Schieck, Marcin Paprzycki, Maria Ganzha, Bogdan Franczyk
Computer Vision
Paths: Prompt-aware Spatio-temporal Transformer with Hierarchical Multi-modal Fusion for RGB-Event Video Person Re-Identification
Yakun Huo, Yingquan Wang, Yangyang Liu et al. (+4 more)
RGB-Event
Video Person Re-Identification (RE-VReID) aims to retrieve specific person across non-overlapping cameras with complementary RGB
videos and event streams. However, existing methods often decouple spatial and temporal modeling, which limits their interaction. In addition, global-level RGB-Event fusion fails to fully exploit fine-grained discriminative cues. To address these issues, we propose Paths, a unified framework with spatio-temporal modeling and hierarchical multi-modal fusion for RE-VReID. Specifically, we first design a
Memory-Augmented Backbone (MAB) to maintain modality-specific identity prototypes for stable intra-modal representation learning. Then, we propose a
Prompt-aware Spatio-temporal
Transformer (PST) to jointly model spatial and temporal cues within a unified
Transformer. Finally, we introduce a Hierarchical Multi-modal Fusion (HMF) to integrate RGB and event features at global and local levels. With these modules, our framework can learn robust and discriminative representations for RE-VReID. Extensive experiments on three public RE-VReID
benchmarks including EvReID, MARS and iLIDS-VID, demonstrate the effectiveness of our proposed method. The code is available at https://github.com/Reflection0427/Paths.
Read the full abstract ▾
All authors Yakun Huo, Yingquan Wang, Yangyang Liu, Tianyu Yan, Yunzhi Zhuge, Pingping Zhang, Huchuan Lu
Computer Vision
Learning Unified Video and Image Representation for Video Face Forgery Detection
Haotian Liu, Yang Liu, Guoying Zhao et al. (+1 more)
Face forgery detection is crucial for preserving the security and integrity of facial data given the rapid developments in face manipulation techniques and deep generative models. Existing methods for
video face forgery detection typically assume that all frames in a forged
video are manipulated, while detecting partially forged
videos that contain only a subset of altered frames remains challenging. To address this issue, we propose a novel framework, UVIF, that utilizes additional annotated images to provide fine-grained supervision for detecting partial forgeries in
videos. UVIF employs a unified encoder and a multi-task learning paradigm to jointly model facial
videos and images for boosted
video face forgery detection. A 2D backbone with temporal fusion modules is employed as the unified encoder. A pseudo labeling process is designed for
video frames to bridge their representations with those of static images. A
video-oriented feature
alignment strategy is further introduced to reduce the distribution gap between
videos and images. Extensive experiments on
benchmark datasets demonstrate the effectiveness of our framework, which outperforms state-of-theart methods in detecting partially forged
videos while introducing no additional computational overhead. Our code is available at https://github.com/haotianll/UVIF.
Read the full abstract ▾
All authors Haotian Liu, Yang Liu, Guoying Zhao, Xiaobai Li
Computer Vision · Cross-listed from cs.RO
H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models
Dingyi Rong, Yue Shi, Chaofan Ma et al. (+6 more)
Large-scale manipulation data is essential for
robot learning, yet collecting
robot demonstrations remains expensive and difficult to scale. Meanwhile, abundant
egocentric human manipulation
videos provide rich behavioral experiences, but transferring them across embodiments remains challenging due to differences between human hands and robotic end-effectors. Recent advances in
video world models offer a promising pathway to synthesize
robot-centric manipulation
videos from human observations, while their cross-embodiment transfer capability remains largely unexplored. Therefore, we introduce H2R-Bench, a
benchmark for evaluating cross-embodiment human-to-
robot manipulation
video generation, where models transform
egocentric human demonstrations into
robot manipulation
videos under specified embodiments. Each
benchmark instance contains a human demonstration
video, target embodiment constraints, and source-grounded annotations covering task goals, action events, functional contacts, and object responses. H2R-Bench evaluates generated
videos through five dimensions, including goal-state completion, action-event completion, functional contact transfer, embodiment correctness, and general
video quality. We
benchmark eleven state-of-the-art
video generation models across six manipulation families and two
robot embodiments. Our evaluation reveals that current
video world models remain limited in human-to-
robot manipulation transfer: even leading models often fail in embodiment consistency, functional interaction, and task execution. H2R-Bench provides a systematic diagnostic framework for evaluating whether
video world models can bridge the human-to-
robot embodiment gap and convert human manipulation observations into
robot-centric training resources.
Read the full abstract ▾
All authors Dingyi Rong, Yue Shi, Chaofan Ma, Jiezhang Cao, Zongrui Wang, Zeyu Zhang, Yao Mu, Guangtao Zhai, Ning Liu
Computer Vision
Topology-Unified 2D Pose Estimation across Intact, Residual and Prosthetic Limbs
Tianye Qi, Tengyue Zhang, Jiaying Ying et al. (+2 more)
Driven by the availability of large-scale
datasets, Human Pose Estimation (HPE) plays a critical role in numerous downstream tasks. However, mainstream
benchmarks exhibit severe representation bias, predominantly featuring able-bodied individuals. While a few pioneering
datasets have attempted to address limb differences, their annotation protocols fail to generalize, struggling to represent specialized mechanical structures like running blades or unprosthetized residual limbs. To bridge this gap, we introduce ProPose, a large-scale
benchmark featuring a novel annotation protocol that unifies the topological representation of biological limbs, diverse prostheses, and physical absences within a single framework. Because real-world prosthetic images are inherently scarce and exhibit extreme long-tail distributions, we design a Real-to-Synthetic data expansion pipeline to explicitly synthesize and expand the underrepresented cases. However, simply training existing models on this enriched
dataset often leads to suboptimal solutions, as they estimate each keypoint independently and might hallucinate non-existent joints on mechanical structures. To resolve this, we propose ProLoss, a structure-aware objective that enforces keypoint dependencies within a single limb to prevent unrealistic limb predictions. Extensive experiments demonstrate that our approach improves the classification accuracy of long-tail prosthetic joints by 2% to 6% without compromising spatial coordinate localization performance. This work sets a foundation for inclusive pose estimation, unlocking new possibilities for understanding the interactions between human bodies and assistive devices.
Read the full abstract ▾
All authors Tianye Qi, Tengyue Zhang, Jiaying Ying, Tianqing Zhu, Xin Yu
Computer Vision
P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation
Yi Shi, Huichao Xie, Yuqing Wang et al. (+7 more)
Infrared-visible image fusion (IVIF) is pivotal for
multimodal perception, yet reconciling the inherent information disparity between thermal and textural features remains a fundamental challenge. Existing prior-guided methods often rely on static constraints that induce optimization conflicts or utilize extrinsic semantic priors from large-scale foundation models (e.g., CLIP/DINO), which frequently fail to exploit the intrinsic modality characteristics essential for high-fidelity fusion. To address these issues, we propose P2Fusion, a prior-guided
distillation-based framework that reformulates IVIF via dual intrinsic
prompts. Instead of imposing hard-coded penalties, we distill image-intrinsic priors, thermal saliency and spatial quality, into learnable dynamic regulators. Specifically, a Teach-to-Fuse mechanism provides dual-granularity progressive guidance, coupled with a Gated Dynamic Expert Recalibration (GDER) module for decoupled feature refinement. This design enables the network to adaptively mediate modal competition through expert specialization. Extensive experiments demonstrate that P2Fusion achieves state-of-the-art performance across five mainstream
datasets. Notably, our framework demonstrates consistent performance advantages in fusion quality, achieving state-of-the-art results in 14 out of 20 key evaluation metrics across 5
benchmarks. Furthermore, it effectively contributes to the robustness of downstream perception, such as +3.2% mAP on MSRS, +0.5% mAP on M3FD and +0.9% mAP on DroneVehicle for object detection. Our code will be available at https://github.com/YiShi99/P2Fusion
Read the full abstract ▾
All authors Yi Shi, Huichao Xie, Yuqing Wang, Mingyu Wang, Kaihui Yang, Yu Liu, Ruitao Lu, Lizhe Li, Junwei Han, Dingwen Zhang
Computer Vision · Cross-listed from cs.AI
From Local Mismatch to Global Impact: Optimizing Cache Reuse Policy for Efficient Diffusion
Xichen Ye, Yifan Wu, Zhikang Xie et al. (+3 more)
Diffusion models have achieved dominant performance in visual generation but suffer from substantial inference overhead. While cache-based acceleration has emerged as a promising solution, existing policies rely on local similarity heuristics, which we identify as being significantly misaligned with final generation quality. This discrepancy stems from the non-uniform propagation and accumulation of errors along the denoising trajectory. To address this, we propose Global-Impact Cache (GCache). We first establish a rigorous theoretical characterization of the error propagation upper bound. Recognizing that this bound can be overly conservative for complex, highly non-convex
diffusion models, we further reparameterize the propagation exponent with a Bernstein form and reformulate cache policy search as a bilevel optimization problem. In detail, GCache identifies an optimal reuse policy in the inner objective while aligning the error-weighting function with generation quality loss in the outer objective. This framework effectively reconciles theoretical rigor with empirical performance, learning to prioritize computation where it most impacts visual fidelity. Extensive experiments demonstrate that GCache consistently outperforms prior caching strategies on both
video and image generation. Notably, on the state-of-the-art Wan2.1
video diffusion model, GCache maintains a 2.17x speedup while significantly enhancing generation quality, reducing LPIPS from 0.1095 to 0.0316.
Read the full abstract ▾
All authors Xichen Ye, Yifan Wu, Zhikang Xie, Xiangyu Yue, Cheng Jin, Weizhong Zhang
Computer Vision
Spatially-Grounded Text-to-Video Generation via Inference-Time Gradient-Free Optimization
Guillaume Jeanneret, Mathis Koroglu, Hugo Caselles-Dupré et al. (+2 more)
Diffusion Transformer Text-to-
Video models have achieved remarkable synthesis quality, yet fine-grained spatial controllability remains a significant challenge. While existing training-free methods produce solid overall results in spatially grounded generation, \ie, placing a specific object in a designated location, they rely on gradient-based optimization techniques that incur prohibitive computational overhead, a bottleneck amplified in modern large-scale architectures. To address this limitation, we present Gradient-free Analytical Trajectory Optimization
Video Generation (GATO-Vid), a novel training-free and gradient-free approach for precise spatial guidance. Rather than relying on costly backward passes, we introduce an alternative cross-
attention score and solve it analytically to obtain an exact, closed-form solution. To use our analytical solution, we propose an on-the-fly injection mechanism tailored to the topological
manifold of the
transformer's latent space. Our experiments demonstrate that GATO-Vid significantly outperforms existing baselines in localization accuracy while introducing minimal computational overhead.
Read the full abstract ▾
All authors Guillaume Jeanneret, Mathis Koroglu, Hugo Caselles-Dupré, Arnaud Dapogny, Matthieu Cord
Computer Vision
UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations
Peng Li, Qianqian Xu, Shilong Bao et al. (+2 more)
Traffic
video understanding has become an important problem in intelligent transportation, as road
videos provide direct evidence for accidents, violations, and interactions between vehicles and vulnerable road users. A useful system should explain how a traffic event develops, why it happens, and when the relevant interaction occurs, yet this remains difficult for
multimodal large
language models (MLLMs) because traffic
videos contain sparse events and varied viewpoints. We introduce UniTraffic-
Agent, the MR-CAS solution for Track~3 of the 10th AI City Challenge, which includes Traffic Anomaly
Reasoning (TAR) and two out-of-domain evaluations: FETV for fisheye traffic events and PSI-VQA for pedestrian intention
reasoning. UniTraffic-
Agent follows an observe–reason–act–verify workflow that samples timestamped visual evidence, reasons over all questions from the same clip in one request, and converts responses through task-specific action adapters. On the official Public leaderboards, MR-CAS ranks 16th on TAR with a score of 0.5780, 2nd on FETV with 0.4884, and 4th on PSI-VQA with 64.4161. The code is available at https://github.com/Roclp/UniTraffic-
Agent.
Read the full abstract ▾
All authors Peng Li, Qianqian Xu, Shilong Bao, Yangbangyan Jiang, Qingming Huang
Computer Vision
RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction
Tianyu Sun, Zhoujie Fu, Zihui Gao et al. (+2 more)
Human-to-
robot (H2R) object handover is a fundamental capability for human-
robot collaboration, yet progress is hindered by the scarcity of large-scale, human-centric
datasets and the significant sim-to-real gap. To address these challenges, we introduce Hand2Bot, an RGB-D
video dataset that provides rich contextual information such as body posture and facial expressions, specifically collected for handover scenarios with real-world noise patterns. We further propose PassGen, a generative pipeline that leverages stable
video diffusion and an Intention-Aware Temporal Face Encoder to synthesize realistic handover sequences while ensuring hand-object consistency. To bridge the sim-to-real gap, we implement a morphology-based depth editing strategy that replicates realistic sensor noise found in physical depth maps. Experimental evaluations demonstrate that our framework achieves high intention identification accuracy and low false trigger rates in both ablation studies and real-world deployment on a physical
robot platform. Our results confirm that training on PassGen allows for robust zero-shot transfer and earlier intention anticipation compared to traditional hand-centric baselines, effectively enabling socially aware robotic behavior in shared workspaces.
Read the full abstract ▾
All authors Tianyu Sun, Zhoujie Fu, Zihui Gao, Bang Zhang, Guosheng Lin
Computer Vision
EgoPHI: Estimating 3D Hand-Object Contact and Force from Egocentric Vision
Andela Ilic, Rachel Schuchert, Yijing Jiang et al. (+1 more)
Understanding hand-object interaction from
egocentric vision is essential for modeling how people physically engage with the surrounding world. Yet
reasoning about physically grounded interaction requires estimating the forces acting on hands and objects, beyond localizing contact. We present EgoPHI, the first method that jointly estimates dense contact maps and 3D force distributions on hand and object meshes from a single
monocular RGB image and object geometry. To address the lack of scalable ground-truth force annotations, we introduce a physics-based simulation pipeline that augments existing hand-object
datasets with dense per-vertex force supervision. EgoPHI then learns dense 3D contact and force on interacting hand and articulated object meshes, extending
vision-based force estimation beyond image-space or planar settings. Our evaluation on in-distribution and out-of-distribution
benchmarks shows that EgoPHI improves force estimation over existing approaches while generalizing to unseen
datasets. To evaluate sim-to-real transfer, we constructed two physical objects that capture dense object contact and force magnitude and used them to record a
dataset of interactions from eight participants across diverse touch and grasp types. Our results demonstrate that EgoPHI recovers meaningful 3D contact and force distributions in simulated, out-of-distribution, and real-world settings, advancing
egocentric hand-object understanding from contact localization toward physically grounded interaction
reasoning.
Read the full abstract ▾
All authors Andela Ilic, Rachel Schuchert, Yijing Jiang, Christian Holz
Computer Vision
Structure-aware Riemannian Growth Fields for 4D Plant Modeling
Meng-Yu Jennifer Kuo and Ryo Kawahara
In this paper, we introduce a novel framework for 4D plant growth modeling that reconstructs the continuous
geometric and topological
evolution of plants from sparse temporal observations. Existing methods mainly rely on dense registration, yet reliable dense sequences are hard to obtain due to scanning constraints and self-occlusions, leaving these approaches struggling under large temporal gaps where rapid organ emergence violates local
rigidity. To overcome this, we bridge these gaps by formulating plant morphogenesis as a continuous procedural process on a structure-aware Riemannian growth field; this jointly models
topology evolution and
geometric deformation, preserving botanical hierarchies and stable spatio-temporal correspondences across distant timepoints. Our key idea is to ground symbolic growth rules within a continuous geodesic flow, where organ development follows biologically modulated trajectories that preserve structural coherence under topological changes. We further contribute a 10-day dual-species
dataset with dense
geometric and semantic annotations. Experiments demonstrate that our method accurately tracks individual organ growth over time and significantly outperforms state-of-the-art baselines in both
geometric accuracy and correspondence consistency.
Read the full abstract ▾
Computer Vision
PixSDS: Why Latent SDS Makes Noisy Pixels
Vsevolod Skorokhodov
Score
Distillation Sampling (SDS) enables text-to-3D generation by optimizing rendered images with a pretrained
diffusion prior, but latent SDS often produces structured color artifacts and high-frequency texture noise. We identify a failure mode of latent SDS caused by VAE-induced pixel drift: the optimized image can move along pixel-space directions that are weakly constrained by the VAE encoder, so its latent representation remains clean and semantically meaningful while the image itself accumulates visible artifacts. We support this diagnosis with controlled 2D SDS experiments, VAE-only optimization, and a simplified analysis showing that encoder-like latent objectives can amplify image-space noise when the inverse mapping to pixels is underconstrained. Motivated by this observation, we propose PixSDS, a lightweight VAE-consistent gradient repair method. PixSDS decodes a latent SDS lookahead step and uses the decoded image as a clean direction for pixel-space optimization, reducing motion in VAE-inconsistent directions without retraining the
diffusion model, changing the renderer, or replacing the SDS objective. Experiments in 2D optimization and text-to-3D generation show that PixSDS substantially reduces structured artifacts while preserving semantic content. Code is publicly available at https://sevashasla.github.io/pixsds-webpage/.
Read the full abstract ▾