Computer Vision · Cross-listed from cs.LG
P-Flow: Proxy-gradient Flows for Linear Inverse Problems
Zehua Jiang, Fenghao Zhu, Xinquan Wang et al. (+2 more)
Generative models based on flow matching have emerged as a powerful paradigm for inverse problems, offering straighter trajectories and faster sampling compared to
diffusion models. However, existing approaches often necessitate differentiating through unrolled paths, leading to numerical instability and prohibitive computational overhead. To address this, we propose P-Flow, a framework that stabilizes the reconstruction process by leveraging a proxy gradient to update the source point. This approach effectively circumvents the numerical instability and
memory overhead of long-chain differentiation. To ensure consistency with the prior distribution, we employ a Gaussian spherical projection motivated by the concentration of measure phenomenon in high-dimensional spaces. We further provide a theoretical analysis for P-Flow based on
Bayesian theory and Lipschitz continuity. Experiments across diverse restoration tasks demonstrate that P-Flow delivers competitive performance, especially under extreme degradations such as severely ill-posed conditions and high measurement noise.
Read the full abstract ▾
All authors Zehua Jiang, Fenghao Zhu, Xinquan Wang, Chongwen Huang, Zhaoyang Zhang
Computer Vision
Improved monocular depth prediction using distance transform over pre-semantic contours with self-supervised neural networks
Marwane Hariat, Antoine Manzanera and David Filliat
Monocular depth estimation (MDE) with self-supervised training approaches struggles in low-texture areas, where photometric losses may lead to ambiguous depth predictions. To address this, we propose a novel technique that enhances spatial information by applying a distance transform over pre-semantic contours, augmenting discriminative power in low texture regions. Our approach jointly estimates pre-semantic contours, depth and ego-motion. The pre-semantic contours are leveraged to produce new input images, with variance augmented by the distance transform in uniform areas. This approach results in more effective loss functions, enhancing the training process for depth and ego-motion. We demonstrate theoretically that the distance transform is the optimal variance-augmenting technique in this context. Through extensive experiments on KITTI, Cityscapes, Waymo, NYUv2 and ScanNet our model demonstrates robust performance, surpassing competing self-supervised methods in MDE.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning
Xi Wang and Cheng Deng
Model merging provides a compelling paradigm for integrating specialized expertise into a unified multi-task model, a goal that aligns naturally with the sequential knowledge acquisition in continual learning (CL). However, the requirement for preserving diverse forms of previous knowledge conflicts with the storage limitations inherent to CL. In this paper, we systematically analyze existing model merging methods under the constraints of CL. We find that current methods prioritize global
alignment, which often leads to the accumulation and amplification of task-specific errors within the continuous data stream; and the vanishing gradients at the onset of subsequent tasks frequently cause optimization to stagnate. These leave the merged model in a suboptimal state at the beginning of the next training phase. To address these challenges, we propose Trajectory Regularized Merging (TRM), a framework that reformulates the merging phase as an optimization process within an augmented trajectory subspace. Our framework integrates three synergistic objectives including task
alignment, prediction consistency, and gradient responsiveness to concurrently preserve merged model's historical stability and re-activate optimization dynamics. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple
benchmarks.
Read the full abstract ▾
Computer Vision
BenchHAR: Benchmarking Self-Supervised Learning for Generalizable Sensor-based Activity Recognition
Yize Cai, Rui Feng, Anlan Yu et al. (+2 more)
Human Activity Recognition (HAR) from wearable sensors supports broad healthcare and behavior science applications. However, data heterogeneity and the scarcity of labeled data limit its real-world generalization. Recent advances in self-supervised learning (SSL) in
vision and
language domains have shown strong capability for learning generalizable representations from unlabeled data. Yet, few studies have systematically compared the generalization performance of SSL methods or explored how to adapt them for generalizable HAR. To address these gaps, we present BenchHAR, a unified framework for evaluating the generalization capability of SSL methods for sensor-based HAR on unseen target distributions. BenchHAR curates a large-scale
dataset (~258K samples) and evaluates eight representative SSL methods across 12 encoder-classifier architectures. Our results reveal that existing SSL methods struggle to achieve satisfactory generalization performance. We find that: (1) For HAR models, the hybrid paradigm (combining reconstruction and contrastive pretraining) achieves the best overall performance. The CNN encoder exhibits the strongest ability to learn generalizable representations, while more expressive classifier architectures further improve generalization. (2) For data scale, increasing the amount of pretraining data from downstream activity classes consistently improves generalization, while adding more labeled data yields limited gains. Interestingly, incorporating unlabeled data from non-downstream activity classes does not improve generalization. (3) Sensor data collected from custom-grade devices generalizes better than that from research-grade devices, and data from limb transfers more effectively to trunk positions. BenchHAR provides a unified
benchmark and actionable insights for generalizable sensor-based HAR systems. Our code is available at https://github.com/saiketa/HAR-Bench.
Read the full abstract ▾
All authors Yize Cai, Rui Feng, Anlan Yu, Baoshen Guo, Zhiqing Hong
Computer Vision
Distill, Diffuse, Segment: Unsupervised 3D Semantic Segmentation for Autonomous Driving Based on Multi-Level Distillation and Graph Diffusion
Yijing Wang, Ruonan Li, Qilin Wang et al. (+2 more)
LiDAR-based semantic
segmentation is essential for autonomous-driving perception, yet dense point-wise annotations are costly, and long-tailed outdoor scenes make small safety-critical objects difficult to discover without supervision. Existing unsupervised methods face three key challenges: they struggle to preserve small and sparsely observed objects under substantial scale variation, have difficulty enforcing intra-region consistency and inter-region discrimination during cross-modal transfer, and lack an efficient feature-preserving mechanism for contextual propagation over superpoint graphs. We therefore propose DDS, an unsupervised 3D semantic
segmentation framework. First, a coarse-to-fine multi-granularity mask cascade provides complementary 3D region cues for objects across different scales, improving the preservation of small and sparsely observed objects. Second, region-guided multi-level
distillation transfers self-supervised visual knowledge through point-level
alignment, mask-level prototype
alignment, and prototype-level contrastive learning, enhancing intra-region consistency and inter-region discrimination. Third, restart-based graph
diffusion efficiently propagates contextual information among superpoints while anchoring the refined representation to the initial distilled features and avoiding explicit graph eigendecomposition. Experiments on real-world driving
datasets show that DDS outperforms representative unsupervised baselines, improving oAcc, mAcc, and mIoU by up to 2.9%, 9.7%, and 4.1%, respectively. These results demonstrate the effectiveness and transferability of DDS for unsupervised 3D scene understanding in autonomous-driving scenarios.
Read the full abstract ▾
All authors Yijing Wang, Ruonan Li, Qilin Wang, Rongqiang Zhao, Jie Liu
Computer Vision · Cross-listed from eess.IV
A Paired Point-of-Care Ultrasound Dataset for Image Quality Enhancement and Benchmarking via a cGAN Baseline
Lennard M. van Karnenbeek, Hilde G.A. van der Pol, Mark Wijkhuizen et al. (+5 more)
Purpose: We aim to enhance the image quality of point-of-care ultrasound (POCUS) devices using deep learning and a novel paired
dataset of POCUS and high-end ultrasound images. Approach: We collected the first accurately paired
dataset using a custom-built automated gantry system of low-end POCUS and high-end ultrasound images. A conditional generative
adversarial network (cGAN) was utilized based on the pix2pix architecture, with a U-Net generator that incorporates both L1 and structural similarity index (SSIM) losses to improve perceptual quality. Pretraining on a simulation
dataset further boosts performance. Evaluation was performed on 1064 paired ex vivo tissue and phantom ultrasound image sets. Results: Our approach improves the SSIM from 0.29 to 0.54 and PSNR from 19.16 dB to 22.41 dB. No-reference metrics also indicate substantial enhancement, with the Natural Image Quality Evaluator (NIQE) and Perception-based Image Quality Evaluator (PIQE) scores dropping from 7.95 to 4.44 and 31.12 to 19.99, respectively. Conclusions: This work presents the first publicly available accurately paired
dataset of low-end POCUS to high end ultrasound images. Additionally, our results demonstrate the potential of the proposed framework to overcome hardware limitations of handheld POCUS, enhancing its diagnostic value in low-resource and point-of-care settings. The POCUS-IQ
Dataset is publicly available at https://github.com/NKI-MedTech-AI/POCUS-IQ.
Read the full abstract ▾
All authors Lennard M. van Karnenbeek, Hilde G.A. van der Pol, Mark Wijkhuizen, Eva Poelman, Caroline A. Drukker, Theo Ruers, Freija Geldof, Behdad Dashtbozorg
Computer Vision
Is Class Signal Clustered or Routed in Task-Induced Implicit Neural Representation Weight Spaces?
Xinyi Guo, Mingyi He, Haobin Ding et al. (+7 more)
Implicit neural representations (INRs) encode images as neural-network weights, making image classification a problem of weight-space classifiability. A natural
geometric hypothesis is that classifier feedback should make image-specific weights cluster by class in the shared-anchor coordinate. We test this hypothesis in the SIREN-based Meta Weight
Transformer (MWT) regime, where end-to-end training meta-learns a shared initialization and inner-loop update schedule for fitting image-specific SIRENs. We find that this prediction fails. Exposed weight-space geometry and supervised clustering pressure do not reliably track trained-reader accuracy; clustering can even make local neighborhoods more class-consistent while making the trained reader worse. Crucially, the reader constructs rather than inherits class-aligned geometry:
token-flow diagnostics show that class-aligned neighborhoods become strongly predictive of trained-reader accuracy only after late reader interactions, not in the input coordinate. We further identify the native SIREN bias column in the augmented weight
token as a low-dimensional, sample-dependent causal readout route for the trained reader; targeted controls rule out generic scalar-column and marginal-distribution artifacts. The diagnosis motivates interventions that strengthen reader routing, add an explicit bias route, or use denser inner-loop fitting; under the lane-specific training conventions used here, route-directed variants often outperform the shared-anchor baseline but interact non-additively. Task-induced INR weights are classifiable not because they form raw
geometric clusters, but because their class signal is routed through the reader.
Read the full abstract ▾
All authors Xinyi Guo, Mingyi He, Haobin Ding, Weiming Chen, Xinrui Chen, Jiawen Li, Di Zhang, Minxi Ouyang, Yizhi Wang, Xitong Ling
Computer Vision
Beyond ViT Tokens: Masked-Diffusion Pretrained Convolutional Pathology Foundation Model for Cell-Level Dense Prediction
Weiming Chen, Xitong Ling, Zhenyang Cai et al. (+5 more)
Cell-level dense prediction is central to computational pathology, but remains challenging due to fine-grained histological structures, strong domain shifts, and costly dense annotations. Existing ViT-based pathology foundation models rely on patch tokenization, which can disrupt spatial continuity and weaken local morphological details needed for
cell-level prediction. To address this, we propose Masked-
Diffusion Convolutional Foundation Models, termed ConvNeXt Masked-
Diffusion (CMD), a self-supervised convolutional generative pretraining framework for dense pathology representation learning. CMD uses a fully convolutional ConvNeXt-UNet backbone, performs masked-
diffusion pretraining in pixel space, and incorporates frozen pathology foundation model features through adaptive normalization. Experimental results demonstrate that CMD consistently outperforms existing ViT-based pathology foundation models and even surpasses state-of-the-art end-to-end
segmentation methods while fine-tuning only a small number of task-specific parameters across multiple pathology dense prediction tasks. The advantage is particularly pronounced under limited annotation settings, where CMD exhibits stronger robustness and generalization ability. Our findings suggest that purely convolutional architectures can also serve as competitive pathology foundation models for
cell-level dense prediction, achieving leading performance within the current ViT-dominated paradigm and providing a scalable, high-performance solution that better preserves histological structural priors for fine-grained pathology understanding.
Read the full abstract ▾
All authors Weiming Chen, Xitong Ling, Zhenyang Cai, Xidong Wang, Jiawen Li, Tian Guan, Benyou Wang, Yonghong He
Computer Vision · Cross-listed from eess.IV
Model-based Dynamic 3D MRI Reconstructions using Neural Fields and Tensor Product Expansions
Ray Sheombarsing, Max van Riel, David Heesterbeek et al. (+2 more)
Conventional MRI reconstruction methods treat images and coil sensitivities as discrete objects, leading to high
memory demands and limited structural awareness that hamper effective regularization. These limitations hinder accurate reconstruction in highly undersampled scenarios, such as dynamic 3D cardiac
magnetic resonance (CMR). We introduce a discretization-free,
memory-efficient, model-based framework for dynamic 2D and 3D MRI reconstruction from highly undersampled data. We represent magnetization and coil sensitivities as continuous objects – differentiable functions – using tensor products of univariate neural fields. This tensor product structure enables scalable optimization in high-dimensional spatiotemporal settings. Our method outperforms state-of-the-art model-based reconstructions in dynamic 2D and 3D MR settings, preserving structure and motion even under aggressive undersampling (e.g., acceleration factor 16).
Read the full abstract ▾
All authors Ray Sheombarsing, Max van Riel, David Heesterbeek, Nico van den Berg, Alessandro Sbrizzi
Computer Vision
Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoning
Jiazheng Li, Chi-Hao Wu, Yunze Liu et al. (+3 more)
Understanding ultra-long
videos such as
egocentric recordings, live streams, or surveillance footage spanning days to weeks, remains a challenge. For current
multimodal LLMs: even with million-
token context windows, frame budgets cover only tens of minutes of densely sampled
video, and most evidence is discarded before inference begins.
Memory-augmented and agentic approaches help with scale, but their
retrieval remains fragmented across modalities and lacks long-range narrative summaries that span days or weeks. We propose MAGIC-
Video, a training-free framework built around a
multimodal memory graph with interleaved narrative chain: the graph unifies episodic, semantic, and visual content through six typed edges and supports cross-modal
retrieval, while the chain distils long-horizon entity biographies and recurring activity events. At inference time, an agentic loop interleaves graph
retrieval with narrative fact injection, covering both the modality and time dimensions of ultra-long
video in a single
retrieval pipeline. On EgoLifeQA, Ego-R1 and MM-Lifelong, MAGIC-
Video consistently outperforms strong general-purpose, long-
video, and agentic baselines, with gains of 10.1, 7.4, and 5.9 points over the prior best agentic system on each
benchmark. Code is available at https://github.com/lijiazheng0917/MAGIC-
video.
Read the full abstract ▾
All authors Jiazheng Li, Chi-Hao Wu, Yunze Liu, Kaize Ding, Jundong Li, Chuxu Zhang
Computer Vision
SAFformer:Improving Spiking Transformer via Active Predictive Filtering
Zequan Xie, Weiming Zeng, Yunhua Chen et al. (+3 more)
Spiking Neural Networks (SNNs) offer notable advantages in biological plausibility and energy efficiency, making them promising candidates for building low-power
Transformers. However, existing Spiking
Transformers largely adhere to a passive reactive paradigm, which struggles to focus on task-relevant information and incurs substantial computational overhead when processing redundant visual data. To overcome this fundamental yet underexplored limitation, we propose SAFformer, a novel Spiking
Transformer architecture based on an active predictive filtering paradigm. Inspired by the
brain's predictive coding mechanism, SAFformer actively suppresses predictable signals and focuses on salient visual features. Extensive experiments show that SAFformer establishes new state-of-the-art performance on CIFAR-10/100 and CIFAR10-DVS. Remarkably, on ImageNet-1K, it achieves 80.44% Top-1 accuracy with only 26.58M parameters and an energy consumption of 5.88 mJ, demonstrating an exceptional balance between accuracy and efficiency.
Read the full abstract ▾
All authors Zequan Xie, Weiming Zeng, Yunhua Chen, Sichang Ling, Tongyang Chen, Jinsheng Xiao
Computer Vision · Cross-listed from eess.IV
Coarse-to-Fine: Progressive Image Compression for Semantically Hierarchical Classification
Jungwoo Kim, Jun-Hyuk Kim and Jong-Seok Lee
Recent advances in learned image compression (LIC) have enabled practical deployments, spurring active research into image compression for machines and progressive coding schemes. However, their integration remains under-explored: prior works on progressive machine codec predominantly target sample-level difficulty adaptation (i.e., easy-to-hard), without considering semantic-level scalability. In this work, we introduce a semantic hierarchy-aware progressive codec that enables semantic scalability (i.e., coarse-to-fine) from a single bitstream. We first systematically categorize ImageNet-1K classes into CLIP
embedding-based semantic hierarchies. Based on a channel-wise autoregressive framework, we decompose latent representations into hierarchically ordered channel blocks, each explicitly optimized for a corresponding semantic hierarchy. Extensive experiments demonstrate that our approach substantially improves coarse-level recognition at low bitrates while maintaining fine-grained accuracy at higher bitrates. By reframing progressive transmission through the lens of semantic scalability, our work provides an efficient and interpretable solution for task-adaptive image coding, outperforming existing progressive codecs under hierarchical evaluation.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
123D: Unifying Multi-Modal Autonomous Driving Data at Scale
Daniel Dauner, Valentin Charraut, Bastian Berle et al. (+10 more)
The pursuit of autonomous driving has produced one of the richest sensor data collections in all of robotics. However, its scale and diversity remain largely untapped. Each
dataset adopts different 2D and 3D modalities, such as cameras,
lidar, ego states, annotations, traffic lights, and HD maps, with different rates and synchronization schemes. They come in fragmented formats requiring complex dependencies that cannot natively coexist in the same development environment. Further, major inconsistencies in annotation conventions prevent training or measuring generalization across multiple
datasets. We present 123D, an open-source framework that unifies such multi-modal driving data through a single API. To handle synchronization, we store each modality as an independent timestamped event stream with no prescribed rate, enabling synchronous or asynchronous access across arbitrary
datasets. Using 123D, we consolidate eight real-world driving
datasets spanning 3,300 hours and 90,000 kilometers, together with a synthetic
dataset with configurable collection scripts, and provide tools for data analysis and visualization. We conduct a systematic study comparing annotation statistics and assessing each
dataset's pose and calibration accuracy. Further, we showcase two applications 123D enables: cross-
dataset 3D object detection transfer and reinforcement learning for planning, and offer recommendations for future directions. Code and documentation are available at https://github.com/kesai-labs/py123d.
Read the full abstract ▾
All authors Daniel Dauner, Valentin Charraut, Bastian Berle, Tianyu Li, Long Nguyen, Jiabao Wang, Changhui Jing, Maximilian Igl, Holger Caesar, Boris Ivanovic, Yiyi Liao, Andreas Geiger, Kashyap Chitta
Computer Vision
Normalizing Trajectory Models
Jiatao Gu, Tianrong Chen, Ying Shen et al. (+3 more)
Diffusion-based models decompose sampling into many small Gaussian denoising steps – an assumption that breaks down when generation is compressed to a few coarse transitions. Existing few-step methods address this through
distillation, consistency training, or
adversarial objectives, but sacrifice the likelihood framework in the process. We introduce Normalizing Trajectory Models (NTM), which models each reverse step as an expressive conditional normalizing flow with exact likelihood training. Architecturally, NTM combines shallow invertible blocks within each step with a deep parallel predictor across the trajectory, forming an end-to-end network trainable from scratch or initializable from pretrained flow-matching models. Its exact trajectory likelihood further enables self-
distillation: a lightweight denoiser trained on the model's own score produces high-quality samples in four steps. On text-to-image
benchmarks, NTM matches or outperforms strong image generation baselines in just four sampling steps while uniquely retaining exact likelihood over the generative trajectory.
Read the full abstract ▾
All authors Jiatao Gu, Tianrong Chen, Ying Shen, David Berthelot, Shuangfei Zhai, Josh Susskind
Computer Vision
EmambaIR: Efficient Visual State Space Model for Event-guided Image Reconstruction
Wei Yu and Yunhang Qian
Recent event-based image reconstruction methods predominantly rely on Convolutional Neural Networks (CNNs) and
Vision Transformers (ViTs) to process complementary event information. However, these architectures face fundamental limitations: CNNs often fail to capture global feature correlations, whereas ViTs incur quadratic computational complexity (e.g., $O(n^2)$), hindering their application in high-resolution scenarios. To address these bottlenecks, we introduce EmambaIR, an Efficient visual State Space Model designed for image reconstruction using spatially sparse and temporally continuous event streams. Our framework introduces two key components: the cross-modal Top-k Sparse
Attention Module (TSAM) and the Gated State-Space Module (GSSM). TSAM efficiently performs pixel-level top-k sparse
attention to guide cross-modal interactions, yielding rich yet sparse fusion features. Subsequently, GSSM utilizes a nonlinear gated unit to enhance the temporal representation of vanilla linear-complexity ($O(n)$) SSMs, effectively capturing global contextual dependencies without the typical computational overhead. Extensive experiments on six
datasets across three diverse image reconstruction tasks - motion deblurring, deraining, and High Dynamic Range (HDR) enhancement - demonstrate that EmambaIR significantly outperforms state-of-the-art methods while offering substantial reductions in
memory consumption and computational cost. The source code and data are publicly available at: https://github.com/YunhangWickert/EmambaIR
Read the full abstract ▾
Computer Vision
Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment
Jerry Jiang, Haowen Sun, Denis Gudovskiy et al. (+4 more)
Spatial intelligence in
vision-
language models (VLMs) attracts research interest with the practical demand to reason in the 3D world.Despite promising results, most existing methods follow the conventional 2D pipeline in VLMs and use pixel-aligned representations for the
vision modality. However, correspondence-based models with implicit 3D scene understanding often fail to achieve spatial consistency, and representation-based models with 3D
geometric priors lack efficiency in
vision sequence serialization. To address this, we propose a Proxy3D method with compact yet comprehensive 3D proxy representations for the
vision modality. Given only
video frames as input, we employ semantic and
geometric encoders to extract scene features and then perform their semantic-aware clustering to obtain a set of proxies in the 3D space. For representation
alignment, we further curate the SpaceSpan
dataset and apply multi-stage training to adopt the proposed 3D proxy representations with the VLM. When using shorter sequences for
vision information, our method achieves competitive or state-of-the-art performance in 3D visual question answering, visual grounding and general spatial intelligence
benchmarks.
Read the full abstract ▾
All authors Jerry Jiang, Haowen Sun, Denis Gudovskiy, Yohei Nakata, Tomoyuki Okuno, Kurt Keutzer, Wenzhao Zheng
Computer Vision
Flow-OPD: On-Policy Distillation for Flow Matching Models
Zhen Fang, Wenxuan Huang, Yu Zeng et al. (+8 more)
Existing Flow Matching (FM) text-to-image models suffer from two critical bottlenecks under multi-task
alignment: the reward sparsity induced by scalar-valued rewards, and the gradient interference arising from jointly optimizing heterogeneous objectives, which together give rise to a 'seesaw effect' of competing metrics and pervasive reward hacking. Inspired by the success of On-Policy
Distillation (OPD) in the large
language model community, we propose Flow-OPD, the first unified post-training framework that integrates on-policy
distillation into Flow Matching models. Flow-OPD adopts a two-stage
alignment strategy: it first cultivates domain-specialized teacher models via single-reward GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; it then establishes a robust initial policy through a Flow-based Cold-Start scheme and seamlessly consolidates heterogeneous expertise into a single student via a three-step orchestration of on-policy sampling, task-routing labeling, and dense trajectory-level supervision. We further introduce
Manifold Anchor Regularization (MAR), which leverages a task-agnostic teacher to provide full-data supervision that anchors generation to a high-quality
manifold, effectively mitigating the aesthetic degradation commonly observed in purely RL-driven
alignment. Built upon Stable
Diffusion 3.5 Medium, Flow-OPD raises the GenEval score from 63 to 92 and the OCR accuracy from 59 to 94, yielding an overall improvement of roughly 10 points over vanilla GRPO, while preserving image fidelity and human-preference
alignment and exhibiting an emergent 'teacher-surpassing' effect. These results establish Flow-OPD as a scalable
alignment paradigm for building generalist text-to-image models. The codes and weights will be released in: https://github.com/CostaliyA/Flow-OPD .
Read the full abstract ▾
All authors Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao, Shuang Chen, Kaituo Feng, Yunlong Lin, Lin Chen, Zehui Chen, Shaosheng Cao, Feng Zhao
Computer Vision
6D Pose Estimation via Keypoint Heatmap Regression with RGB-D Residual Neural Networks
Ismail Aljosevic, Amir Masoud Almasi, Ana Parovic et al. (+1 more)
In this paper, we propose a modular framework for 6D pose estimation based on keypoint heatmap regression. Our approach combines YOLOv10m for object detection with a ResNet18-based network that predicts 2D heatmaps from RGB images. Keypoints extracted from these heatmaps are used to estimate the 6D object pose via the PnP RANSAC
algorithm. We compare different keypoint selection strategies to assess their impact on pose accuracy. Additionally, we extend the baseline by incorporating depth data using a cross-fusion architecture, which enables interaction between RGB and depth features at multiple stages. We further explore general training improvements, such as experimenting with activation functions and learning rate scheduling strategies to improve model performance. Our best RGB-only model achieved a mean ADD-based accuracy of 84.50%, while the RGB-D fusion model reached 92.41% on the LINEMOD
dataset. The code is available at https://github.com/ameermasood/HeatNet.
Read the full abstract ▾
All authors Ismail Aljosevic, Amir Masoud Almasi, Ana Parovic, Ashkan Shafiei
Computer Vision
Towards Highly-Constrained Human Motion Generation with Retrieval-Guided Diffusion Noise Optimization
Hanchao Liu, Fang-Lue Zhang, Shining Zhang et al. (+2 more)
Generating human motion that satisfies customized zero-shot goal functions, enabling applications such as controllable character animation and behavior synthesis for virtual
agents, is a critical capability. While current approaches handle many unseen constraints, they fail on tasks with very challenging spatiotemporal restrictions, such as severe spatial obstacles or specified numbers of walking steps. To equip motion generators for these highly constrained tasks, we present a
retrieval-guided method built on the training-free
diffusion noise optimization framework. The key idea is to search within large motion
datasets for guidance that can potentially satisfy difficult constraints. We introduce relational task parsing to group target constraints and identify the difficult ones to be handled by retrieved reference. A better initialization for
diffusion noise is then obtained via a reward-guided mask that combines random noise with retrieved noise. By optimizing
diffusion noise from this improved initialization, we successfully solve highly constrained generation tasks. By leveraging LLM for relational task parsing, the whole framework is further enabled to automatically reason for what to retrieve, improving the intelligence of moving
agents under a training-free optimization scheme.
Read the full abstract ▾
All authors Hanchao Liu, Fang-Lue Zhang, Shining Zhang, Tai-Jiang Mu, Shi-Min Hu
Computer Vision
MoCoTalk: Multi-Conditional Diffusion with Adaptive Router for Controllable Talking Head Generation
Xinyan Ye, Jiankang Deng and Abbas Edalat
Talking-head generation requires joint modeling of identity, head pose, facial expression, and mouth dynamics. Existing methods typically address only a subset of these factors, and rely on fixed-weight or heuristic fusion when multiple conditions are involved. We present MoCoTalk, a multi-conditional
video diffusion framework that unifies four complementary control signals: a reference image, facial keypoints, 3DMM-rendered shading meshes, and the corresponding
speech audio. To resolve destructive interference among heterogeneous conditions, we introduce an Adaptive Multi-Condition Router that computes channel-wise, timestep-aware gating over the four condition streams, allowing the fusion strategy to vary with both feature subspace and noise level. To better capture
speech-related facial dynamics, we design a Mouth-Augmented Shading Mesh, a 3DMM-based representation that decouples head motion, mouth motion, expression, and lighting. This design provides a temporally consistent
geometric prior and allows flexible recombination of these attributes at inference. We further introduce a lip consistency loss to tighten audio-visual
alignment. Extensive experiments show that MoCoTalk achieves state-of-the-art performance on the majority of structural, motion, and perceptual metrics, while offering attribute-level controllability that single-condition methods do not provide.
Read the full abstract ▾
Computer Vision
SCOPE: Structured Decomposition and Conditional Skill Orchestration for Complex Image Generation
Tianfei Ren, Zhipeng Yan, Yiming Zhao et al. (+13 more)
While text-to-image models have made strong progress in visual fidelity, faithfully realizing complex visual intents remains challenging because many requirements must be tracked across grounding, generation, and verification. We refer to these requirements as semantic commitments and formalize their lifecycle discontinuity as the Conceptual Rift, where commitments may be locally resolved or checked but fail to remain identifiable as the same operational units throughout the generation lifecycle. To address this, we propose SCOPE, a specification-guided skill orchestration framework that maintains semantic commitments in an evolving structured specification and conditionally invokes
retrieval,
reasoning, and repair skills around unresolved or violated commitments. To evaluate commitment-level intent realization, we introduce Gen-Arena, a human-annotated
benchmark with entity- and constraint-level specifications, together with Entity-Gated Intent Pass Rate (EGIP), a strict entity-first pass criterion. SCOPE substantially outperforms all evaluated baselines on Gen-Arena, achieving 0.60 EGIP, and further achieves strong results on WISE-V (0.907) and MindBench (0.61), demonstrating the effectiveness of persistent commitment tracking for complex image generation.
Read the full abstract ▾
All authors Tianfei Ren, Zhipeng Yan, Yiming Zhao, Zhen Fang, Yu Zeng, Guohui Zhang, Hang Xu, Xiaoxiao Ma, Shiting Huang, Ke Xu, Wenxuan Huang, Lionel Z. Wang, Lin Chen, Zehui Chen, Jie Huang, Feng Zhao
Computer Vision
Kaidi Jia, Yujie Lin, Chengyi Yang et al. (+2 more)
Vision-
language models (VLMs) raise growing concerns about
privacy, copyright, and bias, motivating machine unlearning to remove sensitive knowledge. However, existing methods primarily fine-tune the
language decoder, leading to superficial forgetting that fails to erase underlying visual representations and often introduces object
hallucination. We propose HFRU, a reinforcement unlearning framework that operates on the
vision encoder for deep semantic removal. Our two-stage approach combines
alignment disruption with GRPO-based optimization using a composite reward, including an abstraction reward that encourages semantically valid substitutions and mitigates
hallucinations. Experiments on object recognition and face identity tasks show that HFRU achieves over 98% forgetting and retention performance, while introducing negligible object
hallucination, significantly outperforming prior methods.Our code and implementation details are available at https://github.com/XMUDeepLIT/HFRU.
Read the full abstract ▾
All authors Kaidi Jia, Yujie Lin, Chengyi Yang, Jiayao Ma, Jinsong Su
Computer Vision
PET-Adapter: Test-Time Domain Adaptation for Full and Limited-Angle PET Image Reconstruction
Rüveyda Yilmaz, Yuli Wu, Johannes Stegmaier et al. (+1 more)
Positron Emission
Tomography (PET) image reconstruction is inherently challenged by Poisson noise and physical degradation factors, which are further exacerbated in limited-angle acquisitions. While deep learning methods demonstrate promising performance, their generalization to unseen clinical data distributions remains limited without extensive retraining. We propose PET-Adapter, a test-time domain adaptation framework for generative PET reconstruction models pretrained solely on phantom data. Our method enables adaptation to clinical
datasets with varying anatomies, tracers, and scanner configurations without requiring paired ground truth. PET-Adapter introduces layer-wise low-rank anatomical conditioning during adaptation and Ordered Subset Expectation Maximization-based warm-starting that initializes the generation from physics-informed reconstructions, reducing
diffusion steps from 50 to 2 without compromising quality. Experiments across multiple clinical
datasets demonstrate superior 3D reconstruction performance in both full-angle and limited-angle settings, highlighting the clinical feasibility and computational efficiency of the proposed approach.
Read the full abstract ▾
All authors Rüveyda Yilmaz, Yuli Wu, Johannes Stegmaier, Volkmar Schulz
Computer Vision
STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation
Ying Shen, Tianrong Chen, Yuan Gao et al. (+6 more)
Deep generative models have advanced rapidly across text and
vision, motivating unified
multimodal systems that can understand, reason over, and generate interleaved text-image sequences. Most existing approaches combine autoregressive
language modeling with
diffusion-based image generators, inheriting a structural mismatch between causal text generation and iterative visual denoising. We observe that autoregressive normalizing flows are autoregressive
Transformers–sharing the same causal mask, KV-cache mechanism, and left-to-right structure as LLMs–making them the most natural paradigm for true unified
multimodal generation. We present STARFlow2, built on the Pretzel architecture that vertically interleaves a pretrained VLM stream with a TarFlow stream via residual skip connections, both operating under the same causal mask. Combined with a deep-shallow flow design and a unified FAE latent space, STARFlow2 enables cache-friendly interleaved generation where both text and visual outputs directly enter the KV-cache without re-encoding. Experiments demonstrate strong performance across image generation and
multimodal understanding
benchmarks, validating autoregressive flows as a viable foundation for unified
multimodal modeling.
Read the full abstract ▾
All authors Ying Shen, Tianrong Chen, Yuan Gao, Yizhe Zhang, Yuyang Wang, Miguel Ángel Bautista, Shuangfei Zhai, Joshua M. Susskind, Jiatao Gu
Computer Vision
TRAS: An Interactive Software for Tracing Tree Ring Cross Sections
Henry Marichal, Diego Passarella and Gregory Randall
Tree ring marking remains a key step in dendrometry and dendrochronology, but it is often performed manually, making the process time-consuming, subjective, and difficult to scale to large image
datasets. We present the Tree Ring Analyzer Suite (TRAS), an open-source graphical software for automatic delineation, manual correction, and measurement of tree rings in wood cross-sectional images. TRAS integrates three complementary detection
algorithms: the classical image-processing method CS-TRD and two deep-learning approaches, DeepCS-TRD and INBD. The interface allows users to refine automatic detections, remove false positives, and manually add missing rings. It also computes dendrochronological metrics such as earlywood and latewood areas, ring perimeter, equivalent ring width, and custom path-based ring-width measurements. TRAS was evaluated on 18 expertly annotated Pinus taeda L. cross-section images. DeepCS-TRD achieved the best automatic detection performance, with an F-score of 81.0% and precision of 86.4%. Automatic detection reduced the required manual correction effort to approximately 20% of ring boundaries. For one-dimensional ring-width measurements, TRAS showed excellent agreement with CooRecorder ($r > 0.99$). Common detection errors, such as jump propagation or false positives near
knots, were easily corrected through the postprocessing interface. TRAS provides a flexible and reproducible solution for tree-ring analysis on Windows, macOS, and Linux. Code is available at the https://hmarichal93.github.io/tras.
Read the full abstract ▾
Computer Vision
SphereVAD: Training-Free Video Anomaly Detection via Geodesic Inference on the Unit Hypersphere
Chao Huang, Pengfei Wei, Wei Wang et al. (+5 more)
Video anomaly detection (VAD) aims to automatically identify events that deviate from normal patterns in untrimmed surveillance
videos. Existing methods universally depend on large-scale annotations or task-specific training procedures, severely limiting their rapid deployment to novel scenes. We observe that intermediate-layer features of pre-trained
multimodal large
language models (MLLMs) already encode rich anomaly semantics, yet existing approaches rely on the
language output pathway and fail to exploit the
geometric discriminability latent in these representations. Based on this finding, we propose SphereVAD, a fully training-free, zero-shot VAD framework that recasts anomaly discrimination as von Mises-Fisher (vMF) likelihood-ratio geodesic inference on the unit hypersphere, unleashing latent discriminability through principled
geometric reasoning rather than learning new representations. Specifically, SphereVAD first applies Frechet mean centering to unfold feature distributions and eliminate domain biases, then employs Holistic Scene
Attention (HSA) to reinforce feature consistency using cross-
video priors, and finally performs vMF-guided Spherical Geodesic Pulling (SGP) to align ambiguous segments with directional prototypes on the spherical
manifold. This training-free pipeline requires only minimal synthetic images for calibration. SphereVAD establishes new state-of-the-art results among training-free approaches on three major
benchmarks and remains competitive with fully supervised baselines. Code will be available upon acceptance.
Read the full abstract ▾
All authors Chao Huang, Pengfei Wei, Wei Wang, Jie Wen, Zhihua Wang, Li Shen, Wenqi Ren, Xiaochun Cao
Computer Vision
Rethinking Dense Optical Flow without Test-Time Scaling
Praroop Chanda and Suryansh Kumar
Recent progress in dense optical flow has been driven by increasingly complex architectures and multi-step refinement for test-time scaling. While these approaches achieve strong
benchmark performance, they also require substantial computation during inference. This raises a fundamental question: Is scaling test-time computation the only way to improve dense optical flow accuracy? We argue that it is not. Instead, powerful visual semantic and
geometric priors encoded in modern foundation models can reduce, if not overcome, the need for computationally expensive iterative refinement at test-time. In this paper, we present a framework that estimates dense optical flow in a single forward pass, leveraging pretrained foundation representations, while avoiding iterative refinement and additional inference-time computation, thus offering an alternative to test-time scaling. Our method extracts visual semantic features from a frozen DINO-v2 backbone and combines them with
geometric cues from a
monocular depth foundation model. We fuse these complementary priors into a unified representation and apply a global matching formulation to estimate dense correspondences without recurrent updates or test-time optimization. Despite avoiding iterative refinement, our approach achieves strong cross-
dataset generalization across challenging
benchmarks. On Sintel Final, we obtain 2.81 EPE without refinement, significantly improving over state-of-the-art (SOTA) SEA-RAFT under comparable training conditions and outperforming RAFT, GMFlow (without refinement), and recent FlowSeek in the same setting. These results suggest that strong foundation priors can substitute for test-time scaling, offering a computationally efficient alternative to refinement-heavy pipelines.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Uncertainty Quantification for Cardiac Shape Reconstruction with Deep Signed Distance Functions via MCMC methods
Jan Verhülsdonk, Thomas Grandits, Francisco Sahli Costabal et al. (+3 more)
Atlas-based approaches allow high-quality, patient-specific shape reconstructions of cardiac anatomy from sparse and/or noisy data such as point clouds. However, these methods are mainly prior-driven, so the impact of uncertainty can be large, limiting their clinical reliability. We propose a probabilistic framework for uncertainty-aware cardiac shape reconstruction that combines Deep Signed Distance Functions (DeepSDFs) with Markov Chain Monte Carlo (MCMC) sampling. Cardiac geometries are modeled implicitly as zero-level sets of a neural network conditioned on learned latent codes, enabling multi-surface reconstruction of the left and right ventricles. By interpreting the reconstruction loss as a log-likelihood, we perform
Bayesian inference in the latent space to obtain both maximum a posteriori (MAP) and posterior-sampled reconstructions. Experiments on a public cardiac
dataset show that our approach produces accurate reconstructions and well-calibrated uncertainty estimates.
Read the full abstract ▾
All authors Jan Verhülsdonk, Thomas Grandits, Francisco Sahli Costabal, Thomas Beiert, Simone Pezzuto, Alexander Effland
Computer Vision
Seeing Across Skies and Streets: Feedforward 3D Reconstruction from Satellite, Drone, and Ground Images
Qiwei Wang, Zhongyao Tuo, Xianghui Ze et al. (+1 more)
Cross-view localization classically asks: where does this ground image lie on the satellite tile? Existing methods are typically limited to 3-DoF estimates – an $(x,y)$ position and a yaw angle – because nadir satellite imagery provides no direct cues for roll, pitch, or altitude, forcing a reliance on planar-motion and zero-tilt assumptions. These assumptions break on real terrain with slopes, ramps, and tilted camera mounts. To overcome this, we introduce a single
UAV image as an intermediate viewpoint: it reveals the 3D structure invisible from nadir, supplies the cues for roll, pitch, and altitude that the satellite alone cannot provide, and needs only spatial overlap with the ground camera – no known relative pose is required. Building on this insight, we propose **Cross3R**, a flexible feed-forward model that ingests a satellite tile together with a
UAV image, a ground image, or both, and, in a single forward pass, recovers a cross-view 3D point cloud, the 6-DoF poses of every input camera, and the on-tile $(x,y)$ position and yaw of each perspective camera. For training and evaluation, we also construct **CrossGeo**, a 278K-image tri-view
dataset spanning 85 scenes across every continent except Antarctica. On CrossGeo, Cross3R consistently outperforms feed-forward 3D baselines in point-cloud reconstruction, 6-DoF camera-pose estimation, and cross-view localization. On KITTI, it outperforms dedicated cross-view methods trained on KITTI on most metrics, despite having no KITTI training itself.
Read the full abstract ▾
All authors Qiwei Wang, Zhongyao Tuo, Xianghui Ze, Yujiao Shi
Computer Vision
HEART: Hyperspherical Embedding Alignment via Kent-Representation Traversal in Diffusion Models
Arani Roy, Shristi Das Biswas and Kaushik Roy
Text-to-image
diffusion models can generate visually stunning images, yet, controlling what appears and how it appears, remains surprisingly difficult, especially when operating solely within the constraints of the text-conditioning space. For example, changing a subject or adjusting an attribute often leads to unintended side effects, such as altered backgrounds or distorted details. This is because most existing text-based control methods treat the
embedding space as Euclidean and apply simple linear transformations, which do not reflect how semantic concepts are actually organized. In this work, we take a step back and ask: what is the true geometry of these
embeddings? We find that text encoder representations lie on a hypersphere, where concepts are not linear directions but structured, anisotropic distributions better captured by Kent distributions. Building on this insight, we propose HEART, a training-free framework that performs Kent-aware geodesic transformations directly on the hypersphere. By respecting the underlying geometry, HEART enables intuitive and precise edits, such as consistent subject replacement and fine-grained attribute control, while preserving the original scene. Importantly, HEART requires no finetuning, inversion, or optimization, and generalizes across
diffusion model architectures. Our results show that a simple shift in perspective, from linear to spherical, can unlock fast, and controllable image generation.
Read the full abstract ▾