Computer Vision
GR3EN: Generative Relighting for 3D Environments
Xiaoyan Xing, Philipp Henzler, Junhwa Hur et al. (+4 more)
We present a method for relighting 3D reconstructions of large room-scale environments. Existing solutions for 3D scene relighting often require solving under-determined or ill-conditioned inverse rendering problems, and are as such unable to produce high-quality results on complex real-world scenes. Though recent progress in using generative image and
video diffusion models for relighting has been promising, these techniques are either limited to 2D image and
video relighting or 3D relighting of individual objects. Our approach enables controllable 3D relighting of room-scale scenes by distilling the outputs of a
video-to-
video relighting
diffusion model into a 3D reconstruction. This side-steps the need to solve a difficult inverse rendering problem, and results in a flexible system that can relight 3D reconstructions of complex real-world scenes. We validate our approach on both synthetic and real-world
datasets to show that it can faithfully render novel views of scenes under new lighting conditions.
Read the full abstract ▾
All authors Xiaoyan Xing, Philipp Henzler, Junhwa Hur, Runze Li, Jonathan T. Barron, Pratul P. Srinivasan, Dor Verbin
Computer Vision
CamPilot: Improving Camera Control in Video Diffusion Model with Efficient Camera Reward Feedback
Wenhang Ge, Guibao Shen, Jiawei Feng et al. (+5 more)
Recent advances in camera-controlled
video diffusion models have significantly improved
video-camera
alignment. However, the camera controllability still remains limited. In this work, we build upon Reward Feedback Learning and aim to further improve camera controllability. However, directly borrowing existing ReFL approaches faces several challenges. First, current reward models lack the capacity to assess
video-camera
alignment. Second, decoding latent into RGB
videos for reward computation introduces substantial computational overhead. Third, 3D
geometric information is typically neglected during
video decoding. To address these limitations, we introduce an efficient camera-aware 3D decoder that decodes
video latent into 3D representations for reward quantization. Specifically,
video latent along with the camera pose are decoded into 3D Gaussians. In this process, the camera pose not only acts as input, but also serves as a projection parameter. Misalignment between the
video latent and camera pose will cause
geometric distortions in the 3D structure, resulting in blurry renderings. Based on this property, we explicitly optimize pixel-level consistency between the rendered novel views and ground-truth ones as reward. To accommodate the
stochastic nature, we further introduce a visibility term that selectively supervises only deterministic regions derived via
geometric warping. Extensive experiments conducted on RealEstate10K and WorldScore
benchmarks demonstrate the effectiveness of our proposed method. Project page: \href{https://a-bigbao.github.io/CamPilot/}{CamPilot Page}.
Read the full abstract ▾
All authors Wenhang Ge, Guibao Shen, Jiawei Feng, Luozhou Wang, Hao Lu, Xingye Tian, Xin Tao, Ying-Cong Chen
Computer Vision
Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition
Geo Ahn, Inwoong Lee, Taeoh Kim et al. (+3 more)
Zero-Shot Compositional Action Recognition (ZS-CAR) requires recognizing novel verb-object combinations composed of previously observed primitives. In this work, we tackle a key failure mode: models predict verbs via object-driven shortcuts (i.e., relying on the labeled object class) rather than temporal evidence. We argue that sparse compositional supervision and verb-object learning asymmetry can promote object-driven shortcut learning. Our analysis with proposed diagnostic metrics shows that existing methods overfit to training co-occurrence patterns and underuse temporal verb cues, resulting in weak generalization to unseen compositions. To address object-driven shortcuts, we propose Robust COmpositional REpresentations (RCORE) with two components. Co-occurrence Prior Regularization (CPR) adds explicit supervision for unseen compositions and regularizes the model against frequent co-occurrence priors by treating them as hard negatives. Temporal Order Regularization for Composition (TORC) enforces temporal-order sensitivity to learn temporally grounded verb representations. Across Sth-com and EK100-com, RCORE reduces shortcut diagnostics and consequently improves compositional generalization.
Read the full abstract ▾
All authors Geo Ahn, Inwoong Lee, Taeoh Kim, Minho Shim, Dongyoon Wee, Jinwoo Choi
Computer Vision
PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation
Onkar Susladkar, Tushar Prakash, Adheesh Juvekar et al. (+4 more)
Discrete
video VAEs underpin modern text-to-
video generation and
video understanding systems, yet existing tokenizers typically learn visual codebooks at a single scale with limited vocabularies and shallow
language supervision, leading to poor cross-modal
alignment and zero-shot transfer. We introduce PyraTok, a
language-aligned pyramidal tokenizer that learns semantically structured discrete latents across multiple spatiotemporal resolutions. PyraTok builds on a pretrained
video VAE and a novel
Language aligned Pyramidal Quantization (LaPQ) module that discretizes encoder features at several depths using a shared large binary codebook, yielding compact yet expressive
video token sequences. To tightly couple visual
tokens with
language, PyraTok jointly optimizes multi-scale text-guided quantization and a global autoregressive objective over the
token hierarchy. Across ten
benchmarks, PyraTok delivers state-of-the-art (SOTA)
video reconstruction, consistently improves text-to-
video quality, and sets new SOTA zero-shot performance on
video segmentation, temporal action localization, and
video understanding, scaling robustly to up to 4K/8K resolutions.
Read the full abstract ▾
All authors Onkar Susladkar, Tushar Prakash, Adheesh Juvekar, Kiet A. Nguyen, Dong-Hwan Jang, Inderjit S Dhillon, Ismini Lourentzou
Computer Vision
Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
Shengbang Tong, Boyang Zheng, Ziteng Wang et al. (+7 more)
Representation Autoencoders (RAEs) have shown distinct advantages in
diffusion modeling on ImageNet by training in high-dimensional semantic latent spaces. In this work, we investigate whether this framework can scale to large-scale, freeform text-to-image (T2I) generation. We first scale RAE decoders on the frozen representation encoder (SigLIP-2) beyond ImageNet by training on web, synthetic, and text-rendering data, finding that while scale improves general fidelity, targeted data composition is essential for specific domains like text. We then rigorously stress-test the RAE design choices originally proposed for ImageNet. Our analysis reveals that scaling simplifies the framework: while dimension-dependent noise scheduling remains critical, architectural complexities such as wide
diffusion heads and noise-augmented decoding offer negligible benefits at scale Building on this simplified framework, we conduct a controlled comparison of RAE against the state-of-the-art FLUX VAE across
diffusion transformer scales from 0.5B to 9.8B parameters. RAEs consistently outperform VAEs during pretraining across all model scales. Further, during finetuning on high-quality
datasets, VAE-based models catastrophically overfit after 64 epochs, while RAE models remain stable through 256 epochs and achieve consistently better performance. Across all experiments, RAE-based
diffusion models demonstrate faster convergence and better generation quality, establishing RAEs as a simpler and stronger foundation than VAEs for large-scale T2I generation. Additionally, because both visual understanding and generation can operate in a shared representation space, the
multimodal model can directly reason over generated latents, opening new possibilities for unified models.
Read the full abstract ▾
All authors Shengbang Tong, Boyang Zheng, Ziteng Wang, Bingda Tang, Nanye Ma, Ellis Brown, Jihan Yang, Rob Fergus, Yann LeCun, Saining Xie
Computer Vision · Cross-listed from cs.LG
Feature-Space Smoothing: Certified Robustness of Deep Representations
Song Xia, Meiwen Ding, Chenqi Kong et al. (+2 more)
Modern deep learning models exhibit strong capabilities across diverse applications, yet remain vulnerable to malicious inputs that induce erroneous predictions via feature-space distortion. To address that, we propose Feature-Space Smoothing (FS), a representation-level certification framework built upon Gaussian randomized smoothing. We show that FS converts a given feature encoder into a smoothed variant that is guaranteed to maintain a certified lower bound on the cosine similarity between clean and
adversarial features under l_2-bounded perturbations. We then establish that this Feature Cosine Similarity Bound (FCSB) can be extended to the prediction-wise certification under the cosine similarity measure, and the value of FCSB is determined by the encoder intrinsic Gaussian robustness score. Building on those insights, we introduce the Gaussian Smoothness Booster (GSB), a plug-and-play module to improve the encoder Gaussian robustness score. Specifically, the GSB module is plugged to enhance the feature-space consistency and maintain the feature utility for downstream tasks under Gaussian perturbations. This design enables seamless integration of FS on the protected model, e.g.,
Multimodal Large
Language Models (MLLMs), without additional model retraining or
alignment, improving its robustness while preserving the performance for downstream task-oriented decoding. Extensive experiments demonstrate that integrating FS consistently provides non-trivial certified robustness and significantly improves task-oriented performance under strong white-box
adversarial attacks across diverse models and applications.
Read the full abstract ▾
All authors Song Xia, Meiwen Ding, Chenqi Kong, Wenhan Yang, Xudong Jiang
Computer Vision
360Anything: Geometry-Free Lifting of Images and Videos to 360{\deg}
Ziyi Wu, Daniel Watson, Andrea Tagliasacchi et al. (+3 more)
Lifting perspective images and
videos to 360{\deg} panoramas enables immersive 3D world generation. Existing approaches often rely on explicit
geometric alignment between the perspective and the equirectangular projection (ERP) space. Yet, this requires known camera metadata, obscuring the application to in-the-wild data where such calibration is typically absent or noisy. We propose 360Anything, a geometry-free framework built upon pre-trained
diffusion transformers. By treating the perspective input and the panorama target simply as
token sequences, 360Anything learns the perspective-to-equirectangular mapping in a purely data-driven way, eliminating the need for camera information. Our approach achieves state-of-the-art performance on both image and
video perspective-to-360{\deg} generation, outperforming prior works that use ground-truth camera information. We also trace the root cause of the seam artifacts at ERP boundaries to zero-padding in the VAE encoder, and introduce Circular Latent Encoding to facilitate seamless generation. Finally, we show competitive results in zero-shot camera FoV and orientation estimation
benchmarks, demonstrating 360Anything's deep
geometric understanding and broader utility in computer
vision tasks. Additional results are available at https://360anything.github.io/.
Read the full abstract ▾
All authors Ziyi Wu, Daniel Watson, Andrea Tagliasacchi, David J. Fleet, Marcus A. Brubaker, Saurabh Saxena
Computer Vision
HVD: Human Vision-Driven Video Representation Learning for Text-Video Retrieval
Zequn Xie, Xin Liu, Boyun Zhang et al. (+3 more)
The success of CLIP has driven substantial progress in text-
video retrieval. However, current methods often suffer from "blind" feature interaction, where the model struggles to discern key visual information from background noise due to the sparsity of textual queries. To bridge this gap, we draw inspiration from human cognitive behavior and propose the Human
Vision-Driven (HVD) model. Our framework establishes a coarse-to-fine
alignment mechanism comprising two key components: the Frame Features Selection Module (FFSM) and the Patch Features Compression Module (PFCM). FFSM mimics the human macro-perception ability by selecting key frames to eliminate temporal redundancy. Subsequently, PFCM simulates micro-perception by aggregating patch features into salient visual entities through an advanced
attention mechanism, enabling precise entity-level matching. Extensive experiments on five
benchmarks demonstrate that HVD not only captures human-like visual focus but also achieves state-of-the-art performance.
Read the full abstract ▾
All authors Zequn Xie, Xin Liu, Boyun Zhang, Yuxiao Lin, Sihang Cai, Tao Jin
Computer Vision
ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion
Remy Sabathier, David Novotny, Niloy J. Mitra et al. (+1 more)
Generating animated 3D objects is at the heart of many applications, yet most advanced works are typically difficult to apply in practice because of their limited setup, their long runtime, or their limited quality. We introduce ActionMesh, a generative model that predicts production-ready 3D meshes "in action" in a feed-forward manner. Drawing inspiration from early
video models, our key insight is to modify existing 3D
diffusion models to include a temporal axis, resulting in a framework we dubbed "temporal 3D
diffusion". Specifically, we first adapt the 3D
diffusion stage to generate a sequence of synchronized latents representing time-varying and independent 3D shapes. Second, we design a temporal 3D autoencoder that translates a sequence of independent shapes into the corresponding deformations of a pre-defined reference shape, allowing us to build an animation. Combining these two components, ActionMesh generates animated 3D meshes from different inputs like a
monocular video, a text description, or even a 3D mesh with a text
prompt describing its animation. Besides, compared to previous approaches, our method is fast and produces results that are rig-free and
topology consistent, hence enabling rapid iteration and seamless applications like texturing and retargeting. We evaluate our model on standard
video-to-4D
benchmarks (Consistent4D, Objaverse) and report state-of-the-art performances on both
geometric accuracy and temporal consistency, demonstrating that our model can deliver animated 3D meshes with unprecedented speed and quality.
Read the full abstract ▾
All authors Remy Sabathier, David Novotny, Niloy J. Mitra, Tom Monnier
Computer Vision
Learning to Watermark in the Latent Space of Generative Models
Sylvestre-Alvise Rebuffi, Tuan Tran, Valeriu Lacatusu et al. (+6 more)
Existing approaches for watermarking AI-generated images often rely on post-hoc methods applied in pixel space, introducing computational overhead and potential visual artifacts. In this work, we explore latent space watermarking and introduce DistSeal, a unified approach for latent watermarking that works across both
diffusion and autoregressive models. Our approach works by training post-hoc watermarking models in the latent space of generative models. We demonstrate that these latent watermarkers can be effectively distilled either into the generative model itself or into the latent decoder, enabling in-model watermarking. The resulting latent watermarks achieve competitive robustness while offering similar imperceptibility and up to 20x speedup compared to pixel-space baselines. Our experiments further reveal that distilling latent watermarkers outperforms distilling pixel-space ones, providing a solution that is both more efficient and more robust.
Read the full abstract ▾
All authors Sylvestre-Alvise Rebuffi, Tuan Tran, Valeriu Lacatusu, Pierre Fernandez, Tomáš Souček, Nikola Jovanović, Tom Sander, Hady Elsahar, Alexandre Mourachko
Computer Vision
Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing
Tingyu Song, Yanzhao Zhang, Mingxin Li et al. (+6 more)
Composed Image
Retrieval (CIR) is a pivotal and complex task in
multimodal understanding. Current CIR
benchmarks typically feature limited query categories and fail to capture the diverse requirements of real-world scenarios. To bridge this evaluation gap, we leverage image editing to achieve precise control over modification types and content, enabling a pipeline for synthesizing queries across a broad spectrum of categories. Using this pipeline, we construct EDIR, a novel fine-grained CIR
benchmark. EDIR encompasses 5,000 high-quality queries structured across five main categories and fifteen subcategories. Our comprehensive evaluation of 13
multimodal embedding models reveals a significant capability gap; even state-of-the-art models (e.g., RzenEmbed and GME) struggle to perform consistently across all subcategories, highlighting the rigorous nature of our
benchmark. Through comparative analysis, we further uncover inherent limitations in existing
benchmarks, such as modality biases and insufficient categorical coverage. Furthermore, an in-domain training experiment demonstrates the feasibility of our
benchmark. This experiment clarifies the task challenges by distinguishing between categories that are solvable with targeted data and those that expose intrinsic limitations of current model architectures.
Read the full abstract ▾
All authors Tingyu Song, Yanzhao Zhang, Mingxin Li, Zhuoning Guo, Dingkun Long, Pengjun Xie, Siyue Zhang, Yilun Zhao, Shu Wu
Computer Vision · Cross-listed from cs.SD
Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networks
Abdul Hannan, Daniele Falavigna, Shah Nawaz et al. (+3 more)
Edge devices operate in constrained and varying resource settings, requiring dynamic architectures that can adapt to limitations of the available resources. To meet such demands, layer dropping ($\mathcal{LD}$) approach is typically used to transform static models into dynamic ones by skipping parts of the network along with reducing overall computational complexity. However, existing $\mathcal{LD}$ methods greatly impact the dynamic model's performance for low and high dropping cases, deteriorating the performance-computation trade-off. To this end, we propose a
distillation-based layer dropping (DLD) framework that effectively combines the capabilities of knowledge
distillation and $\mathcal{LD}$ in an end-to-end fashion, thereby achieving state-of-the-art performance for dynamic
speech networks. Comprehensive experimentation utilizing well-known
speech recognition methods, including conformer and WavLM, on three public
benchmarks demonstrates the effectiveness of our framework, reducing the word error rate by $9.32\%$ and $2.25\%$ for high and no dropping cases with $33.3\%$ reduction in training time.
Read the full abstract ▾
All authors Abdul Hannan, Daniele Falavigna, Shah Nawaz, Mubashir Noman, Markus Schedl, Alessio Brutti
Computer Vision · Cross-listed from cs.CL
synthocr-gen: A synthetic ocr dataset generator for low-resource languages- breaking the data barrier
Haq Nawaz Malik, Kh Mohmad Shafi and Tanveer Ahmad Reshi
Optical Character Recognition (OCR) for low-resource
languages remains a significant challenge due to the scarcity of large-scale annotated training
datasets.
Languages such as Kashmiri, with approximately 7 million speakers and a complex Perso-Arabic script featuring unique diacritical marks, currently lack support in major OCR systems including Tesseract, TrOCR, and PaddleOCR. Manual
dataset creation for such
languages is prohibitively expensive, time-consuming, and error-prone, often requiring word by word transcription of printed or handwritten text. We present SynthOCR-Gen, an open-source synthetic OCR
dataset generator specifically designed for low-resource
languages. Our tool addresses the fundamental bottleneck in OCR development by transforming digital Unicode text corpora into ready-to-use training
datasets. The system implements a comprehensive pipeline encompassing text
segmentation (character, word, n-gram, sentence, and line levels), Unicode normalization with script purity enforcement, multi-font rendering with configurable distribution, and 25+ data augmentation techniques simulating real-world document degradations including rotation, blur, noise, and scanner artifacts. We demonstrate the efficacy of our approach by generating a 600,000-sample word-segmented Kashmiri OCR
dataset, which we release publicly on HuggingFace. This work provides a practical pathway for bringing low-resource
languages into the era of
vision-
language AI models, and the tool is openly available for researchers and practitioners working with underserved writing systems worldwide.
Read the full abstract ▾
Computer Vision
Clustering-Guided Spatial-Spectral Mamba for Hyperspectral Image Classification
Zack Dewis, Yimin Zhu, Zhengsen Xu et al. (+4 more)
Although Mamba models greatly improve Hyperspectral Image (HSI) classification, they have critical challenges in terms defining efficient and adaptive
token sequences for improve performance. This paper therefore presents CSSMamba (Clustering-guided Spatial-Spectral Mamba) framework to better address the challenges, with the following contributions. First, to achieve efficient and adaptive
token sequences for improved Mamba performance, we integrate the clustering mechanism into a spatial Mamba architecture, leading to a cluster-guided spatial Mamba module (CSpaMamba) that reduces the Mamba sequence length and improves Mamba feature learning capability. Second, to improve the learning of both spatial and spectral information, we integrate the CSpaMamba module with a spectral mamba module (SpeMamba), leading to a complete clustering-guided spatial-spectral Mamba framework. Third, to further improve feature learning capability, we introduce an
Attention-Driven
Token Selection mechanism to optimize Mamba
token sequencing. Last, to seamlessly integrate clustering into the Mamba model in a coherent manner, we design a Learnable Clustering Module that learns the cluster memberships in an adaptive manner. Experiments on the Pavia University, Indian Pines, and Liao-Ning 01
datasets demonstrate that CSSMamba achieves higher accuracy and better boundary preservation compared to state-of-the-art CNN,
Transformer, and Mamba-based methods.
Read the full abstract ▾
All authors Zack Dewis, Yimin Zhu, Zhengsen Xu, Mabel Heffring, Saeid Taleghanidoozdoozan, Quinn Ledingham, Lincoln Linlin Xu
Computer Vision · Cross-listed from cs.NE
Neural Particle Automata: Learning Self-Organizing Particle Dynamics
Hyunsoo Kim, Ehsan Pajouheshgar, Sabine Süsstrunk et al. (+2 more)
We introduce Neural Particle Automata (NPA), a Lagrangian generalization of Neural Cellular Automata (NCA) from static
lattices to dynamic particle systems. Unlike classical Eulerian NCA where
cells are pinned to pixels or voxels, NPA model each
cell as a particle with a continuous position and internal state, both updated by a shared, learnable neural rule. This particle-based formulation yields clear individuation of
cells, allows heterogeneous dynamics, and concentrates computation only on regions where activity is present. At the same time, particle systems pose challenges: neighborhoods are dynamic, and a naive implementation of local interactions scale quadratically with the number of particles. We address these challenges by replacing grid-based neighborhood perception with differentiable Smoothed Particle Hydrodynamics (SPH) operators backed by
memory-efficient, CUDA-accelerated kernels, enabling scalable end-to-end training. Across tasks including morphogenesis, point-cloud classification, and particle-based texture synthesis, we show that NPA retain key NCA behaviors such as robustness and self-regeneration, while enabling new behaviors specific to particle systems. Together, these results position NPA as a compact neural model for learning self-organizing particle dynamics.
Read the full abstract ▾
All authors Hyunsoo Kim, Ehsan Pajouheshgar, Sabine Süsstrunk, Wenzel Jakob, Jinah Park
Computer Vision
SAMTok: Representing Any Mask with Two Words
Yikang Zhou, Tao Zhang, Dengxian Gong et al. (+13 more)
Pixel-wise capabilities are essential for building interactive intelligent systems. However, pixel-wise multi-modal LLMs (MLLMs) remain difficult to scale due to complex region-level encoders, specialized
segmentation decoders, and incompatible training objectives. To address these challenges, we present SAMTok, a discrete mask tokenizer that converts any region mask into two special
tokens and reconstructs the mask using these
tokens with high fidelity. By treating masks as new
language tokens, SAMTok enables base MLLMs (such as the QwenVL series) to learn pixel-wise capabilities through standard next-
token prediction and simple reinforcement learning, without architectural modifications and specialized loss design. SAMTok builds on SAM2 and is trained on 209M diverse masks using a mask encoder and residual vector quantizer to produce discrete, compact, and information-rich
tokens. With 5M SAMTok-formatted mask understanding and generation data samples, QwenVL-SAMTok attains state-of-the-art or comparable results on region captioning, region VQA, grounded conversation, referring
segmentation, scene graph parsing, and multi-round interactive
segmentation. We further introduce a textual answer-matching reward that enables efficient reinforcement learning for mask generation, delivering substantial improvements on GRES and GCG
benchmarks. Our results demonstrate a scalable and straightforward paradigm for equipping MLLMs with strong pixel-wise capabilities. Our code and models are available.
Read the full abstract ▾
All authors Yikang Zhou, Tao Zhang, Dengxian Gong, Yuanzheng Wu, Ye Tian, Haochen Wang, Haobo Yuan, Jiacong Wang, Lu Qi, Hao Fei, Anran Wang, Zhuochen Wang, Yujing Wang, Cheng Chen, Shunping Ji, Xiangtai Li
Computer Vision
Masked Modeling for Human Motion Recovery Under Occlusions
Zhiyin Qian, Siwei Zhang, Bharat Lal Bhatnagar et al. (+2 more)
Human motion reconstruction from
monocular videos is a fundamental challenge in computer
vision, with broad applications in AR/VR, robotics, and digital content creation, but remains challenging under frequent occlusions in real-world settings. Existing regression-based methods are efficient but fragile to missing observations, while optimization- and
diffusion-based approaches improve robustness at the cost of slow inference speed and heavy preprocessing steps. To address these limitations, we leverage recent advances in generative masked modeling and present MoRo: Masked Modeling for human motion Recovery under Occlusions. MoRo is an occlusion-robust, end-to-end generative framework that formulates motion reconstruction as a
video-conditioned task, and efficiently recover human motion in a consistent global coordinate system from RGB
videos. By masked modeling, MoRo naturally handles occlusions while enabling efficient, end-to-end inference. To overcome the scarcity of paired
video-motion data, we design a cross-modality learning scheme that learns multi-modal priors from a set of heterogeneous
datasets: (i) a trajectory-aware motion prior trained on MoCap
datasets, (ii) an image-conditioned pose prior trained on image-pose
datasets, capturing diverse per-frame poses, and (iii) a
video-conditioned masked
transformer that fuses motion and pose priors, finetuned on
video-motion
datasets to integrate visual cues with motion dynamics for robust inference. Extensive experiments on EgoBody and RICH demonstrate that MoRo substantially outperforms state-of-the-art methods in accuracy and motion realism under occlusions, while performing on-par in non-occluded scenarios. MoRo achieves real-time inference at 70 FPS on a single H200 GPU.
Read the full abstract ▾
All authors Zhiyin Qian, Siwei Zhang, Bharat Lal Bhatnagar, Federica Bogo, Siyu Tang
Computer Vision
DSFedMed: Dual-Scale Federated Medical Image Segmentation via Mutual Distillation Between Foundation and Lightweight Models
Hanwen Zhang, Qiaojin Shen, Yuxi Liu et al. (+2 more)
Foundation Models (FMs) have demonstrated strong generalization across diverse
vision tasks. However, their deployment in
federated settings is hindered by high computational demands, substantial communication overhead, and significant inference costs. We propose DSFedMed, a dual-scale
federated framework that enables mutual knowledge
distillation between a centralized foundation model and lightweight client models for medical image
segmentation. To support knowledge
distillation, a set of high-quality medical images is generated to replace real public
datasets, and a learnability-guided sample selection strategy is proposed to enhance efficiency and effectiveness in dual-scale
distillation. This mutual
distillation enables the foundation model to transfer general knowledge to lightweight clients, while also incorporating client-specific insights to refine the foundation model. Evaluations on five medical imaging
segmentation datasets show that DSFedMed achieves an average 2 percent improvement in Dice score while reducing communication costs and inference time by nearly 90 percent compared to existing
federated foundation model baselines. These results demonstrate significant efficiency gains and scalability for resource-limited
federated deployments.
Read the full abstract ▾
All authors Hanwen Zhang, Qiaojin Shen, Yuxi Liu, Yuesheng Zhu, Guibo Luo
Computer Vision
DTP: A Simple yet Effective Distracting Token Pruning Framework for Vision-Language Action Models
Chenyang Li, Jieyuan Liu, Bin Li et al. (+5 more)
Vision-
Language Action (VLA) models have shown remarkable progress in robotic manipulation by leveraging the powerful perception abilities of
Vision-
Language Models (VLMs) to understand environments and directly output actions. However, by default, VLA models may overly attend to image
tokens in the task-irrelevant region, which we describe as 'distracting
tokens'. This behavior can disturb the model from the generation of the desired action
tokens in each step, affecting the success rate of tasks. In this paper, we introduce a simple yet effective plug-and-play Distracting
Token Pruning (DTP) framework, which dynamically detects and prunes these distracting image
tokens. By correcting the model's visual
attention patterns, we aim to improve the task success rate, as well as exploring the performance upper boundaries of the model without altering its original architecture or adding additional inputs. Experiments on the SIMPLER
Benchmark (Li et al., 2024) show that our method consistently achieving relative improvements in task success rates across different types of novel VLA models, demonstrating generalizability to
transformer-based VLAs. Further analysis reveals a negative correlation between the task success rate and the amount of attentions in the task-irrelevant region for all models tested, highlighting a common phenomenon of VLA models that could guide future research. We also publish our code at: https://anonymous.4open.science/r/CBD3.
Read the full abstract ▾
All authors Chenyang Li, Jieyuan Liu, Bin Li, Bo Gao, Yilin Yuan, Yangfan He, Yuchen Li, Jingqun Tang
Computer Vision · Cross-listed from eess.IV
Phi-SegNet: Phase-Integrated Supervision for Medical Image Segmentation
Shams Nafisa Ali and Taufiq Hasan
Deep learning has substantially advanced medical image
segmentation, yet achieving robust generalization across diverse imaging modalities and anatomical structures remains a major challenge. A key contributor to this limitation lies in how existing architectures, ranging from CNNs to
Transformers and their hybrids, primarily encode spatial information while overlooking frequency-domain representations that capture rich structural and textural cues. Although recent studies have begun exploring spectral information at the feature level, supervision-level integration of frequency cues-crucial for fine-grained object localization-remains largely untapped. To this end, we propose Phi-SegNet, a CNN-based architecture that incorporates phase-aware information at both architectural and optimization levels. The network integrates Bi-Feature Mask Former (BFMF) modules that blend neighboring encoder features to reduce semantic gaps, and Reverse
Fourier Attention (RFA) blocks that refine decoder outputs using phase-regularized features. A dedicated phase-aware loss aligns these features with structural priors, forming a closed feedback loop that emphasizes boundary precision. Evaluated on five public
datasets spanning X-ray, US, histopathology, MRI, and colonoscopy, Phi-SegNet consistently achieved state-of-the-art performance, with an average relative improvement of 1.54+/-1.26% in IoU and 0.98+/-0.71% in F1-score over the next best-performing model. In cross-
dataset generalization scenarios involving unseen
datasets from the known domain, it also exhibits robust and superior performance, highlighting its adaptability and modality-agnostic design. These findings demonstrate the potential of leveraging spectral priors in both feature representation and supervision, paving the way for generalized
segmentation frameworks that excel in fine-grained object localization. Our code is available on GitHub.
Read the full abstract ▾
Computer Vision
Yuan Lin, Murong Xu, Marc Hölle et al. (+5 more)
Widely adopted medical image
segmentation methods, although efficient, are primarily deterministic and remain poorly amenable to natural
language prompts. Thus, they lack the capability to estimate multiple proposals, human interaction, and cross-modality adaptation. Recently, text-to-image
diffusion models have shown potential to bridge the gap. However, training them from scratch requires a large
dataset-a limitation for medical image
segmentation. Furthermore, they are often limited to binary
segmentation and cannot be conditioned on a natural
language prompt. To this end, we propose a novel framework called ProGiDiff that leverages existing image generation models for medical image
segmentation purposes. Specifically, we propose a ControlNet-style conditioning mechanism with a custom encoder, suitable for image conditioning, to steer a pre-trained
diffusion model to output
segmentation masks. It naturally extends to a multi-class setting simply by prompting the target organ. Our experiment on organ
segmentation from CT images demonstrates strong performance compared to previous methods and could greatly benefit from an expert-in-the-loop setting to leverage multiple proposals. Importantly, we demonstrate that the learned conditioning mechanism can be easily transferred through low-rank, few-shot adaptation to segment MR images.
Read the full abstract ▾
All authors Yuan Lin, Murong Xu, Marc Hölle, Chinmay Prabhakar, Andreas Maier, Vasileios Belagiannis, Bjoern Menze, Suprosanna Shit
Computer Vision · Cross-listed from cs.RO
Junha Lee, Eunha Park and Minsu Cho
Language-driven dexterous grasp generation requires the models to understand task semantics, 3D geometry, and complex hand-object interactions. While
vision-
language models have been applied to this problem, existing approaches directly map observations to grasp parameters without intermediate
reasoning about physical interactions. We present DextER, Dexterous Grasp Generation with
Embodied Reasoning, which introduces contact-based
embodied reasoning for multi-finger manipulation. Our key insight is that predicting which hand links contact where on the object surface provides an embodiment-aware intermediate representation, bridging task semantics with physical constraints. DextER autoregressively generates
embodied contact
tokens specifying which finger links contact where on the object surface, followed by grasp
tokens encoding the hand configuration. On DexGYS, DextER achieves 67.14% success rate, outperforming state-of-the-art by 3.83 p.p. with 96.4% improvement in intention
alignment. We also demonstrate steerable generation through partial contact specification, providing fine-grained control over grasp synthesis.
Read the full abstract ▾
Computer Vision
PAINT: Pathology-Aware Integrated Next-Scale Transformation for Virtual Immunohistochemistry
Rongze Ma, Mengkang Lu, Zhenyu Xiang et al. (+4 more)
Virtual immunohistochemistry (IHC) aims to computationally synthesize molecular staining patterns from routine Hematoxylin and Eosin (H&E) images, offering a cost-effective and tissue-efficient alternative to traditional physical staining. However, this task is particularly challenging: H&E morphology provides ambiguous cues about
protein expression, and similar tissue structures may correspond to distinct molecular states. Most existing methods focus on direct appearance synthesis to implicitly achieve cross-modal generation, often resulting in semantic inconsistencies due to insufficient structural priors. In this paper, we propose Pathology-Aware Integrated Next-Scale Transformation (PAINT), a visual autoregressive framework that reformulates the synthesis process as a structure-first conditional generation task. Unlike direct image translation, PAINT enforces a causal order by resolving molecular details conditioned on a global structural layout. Central to this approach is the introduction of a Spatial Structural Start Map (3S-Map), which grounds the autoregressive initialization in observed morphology, ensuring deterministic, spatially aligned synthesis. Experiments on the IHC4BC and MIST
datasets demonstrate that PAINT outperforms state-of-the-art methods in structural fidelity and clinical downstream tasks, validating the potential of structure-guided autoregressive modeling.
Read the full abstract ▾
All authors Rongze Ma, Mengkang Lu, Zhenyu Xiang, Yongsheng Pan, Yicheng Wu, Qingjie Zeng, Yong Xia
Computer Vision
Keyframe-Based Feed-Forward Visual Odometry
Weichen Dai, Wenhan Su, Da Kong et al. (+2 more)
The emergence of visual foundation models has revolutionized visual odometry~(VO) and SLAM, enabling pose estimation and dense reconstruction within a single feed-forward network. However, unlike traditional pipelines that leverage keyframe methods to enhance efficiency and accuracy, current foundation model based methods, such as VGGT-Long, typically process raw image sequences indiscriminately. This leads to computational redundancy and degraded performance caused by low inter-frame parallax, which provides limited contextual stereo information. Integrating traditional
geometric heuristics into these methods is non-trivial, as their performance depends on high-dimensional latent representations rather than explicit
geometric metrics. To bridge this gap, we propose a novel keyframe-based feed-forward VO. Instead of relying on hand-crafted rules, our approach employs reinforcement learning to derive an adaptive keyframe policy in a data-driven manner, aligning selection with the intrinsic characteristics of the underlying foundation model. We train our
agent on TartanAir
dataset and conduct extensive evaluations across several real-world
datasets. Experimental results demonstrate that the proposed method achieves consistent and substantial improvements over state-of-the-art feed-forward VO methods.
Read the full abstract ▾
All authors Weichen Dai, Wenhan Su, Da Kong, Yuhang Ming, Wanzeng Kong
Computer Vision
PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models
Chak-Wing Mak, Guanyu Zhu, Boyi Zhang et al. (+16 more)
Modern foundational
Multimodal Large
Language Models (MLLMs) and
video world models have advanced significantly in mathematical, common-sense, and visual
reasoning, but their grasp of the underlying physics remains underexplored. Existing
benchmarks attempting to measure this matter rely on synthetic, Visual Question Answer templates or focus on perceptual
video quality that is tangential to measuring how well the
video abides by physical laws. To address this fragmentation, we introduce PhysicsMind, a unified
benchmark with both real and simulation environments that evaluates law-consistent
reasoning and generation over three canonical principles: Center of Mass, Lever
Equilibrium, and Newton's First Law. PhysicsMind comprises two main tasks: i) VQA tasks, testing whether models can reason and determine physical quantities and values from images or short
videos, and ii)
Video Generation(VG) tasks, evaluating if predicted motion trajectories obey the same center-of-mass, torque, and inertial constraints as the ground truth. A broad range of recent models and
video generation models is evaluated on PhysicsMind and found to rely on appearance heuristics while often violating basic mechanics. These gaps indicate that current scaling and training are still insufficient for robust physical understanding, underscoring PhysicsMind as a focused testbed for physics-aware
multimodal models. Our data will be released upon acceptance.
Read the full abstract ▾
All authors Chak-Wing Mak, Guanyu Zhu, Boyi Zhang, Hongji Li, Xiaowei Chi, Kevin Zhang, Yichen Wu, Yangfan He, Chun-Kai Fan, Wentao Lu, Kuangzhi Ge, Xinyu Fang, Hongyang He, Kuan Lu, Tianxiang Xu, Li Zhang, Yongxin Ni, Youhua Li, Shanghang Zhang
Computer Vision
HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Models
Xin Xie, Jiaxian Guo and Dong Gong
Diffusion model
alignment aims to bridge the gap between generated outputs and human preferences by enhancing both semantic consistency with textual
prompts and overall visual quality. Existing
alignment methods face a challenging trade-off: test-time approaches enable input-specific adaptability but introduce significant computational overhead and tend to under-optimize, while fine-tuning approaches risk reward over-optimization and loss of generation diversity. To bridge this gap, we propose HyperAlign, a framework that trains a hypernetwork for efficient and effective test-time
alignment. Instead of modifying latent states directly, HyperAlign dynamically generates input-and-state-conditioned low-rank adaptation weights to modulate the denoising trajectory toward target rewards. We introduce multiple HyperAlign variants of varying granularity to balance
alignment quality and computational efficiency. The hypernetwork is optimized with a reward objective regularized by preference data to mitigate reward hacking. We evaluate HyperAlign across multiple generative paradigms, including Stable
Diffusion and FLUX, where it significantly outperforms existing
alignment methods in semantic consistency and visual quality.
Read the full abstract ▾
Computer Vision
EVolSplat4D: Efficient Volume-based Gaussian Splatting for 4D Urban Scene Synthesis
Sheng Miao, Sijin Li, Pan Wang et al. (+5 more)
Novel view synthesis (NVS) of static and dynamic urban scenes is essential for autonomous driving simulation, yet existing methods often struggle to balance reconstruction time with quality. While state-of-the-art neural radiance fields and 3D Gaussian Splatting approaches achieve photorealism, they often rely on time-consuming per-scene optimization. Conversely, emerging feed-forward methods frequently adopt per-pixel Gaussian representations, which lead to 3D inconsistencies when aggregating multi-view predictions in complex, dynamic environments. We propose EvolSplat4D, a feed-forward framework that moves beyond existing per-pixel paradigms by unifying volume-based and pixel-based Gaussian prediction across three specialized branches. For close-range static regions, we predict consistent geometry of 3D Gaussians over multiple frames directly from a 3D feature volume, complemented by a semantically-enhanced image-based rendering module for predicting their appearance. For dynamic actors, we utilize object-centric canonical spaces and a motion-adjusted rendering module to aggregate temporal features, ensuring stable 4D reconstruction despite noisy motion priors. Far-Field scenery is handled by an efficient per-pixel Gaussian branch to ensure full-scene coverage. Experimental results on the KITTI-360, KITTI, Waymo, and PandaSet
datasets show that EvolSplat4D reconstructs both static and dynamic environments with superior accuracy and consistency, outperforming both per-scene optimization and state-of-the-art feed-forward baselines.
Read the full abstract ▾
All authors Sheng Miao, Sijin Li, Pan Wang, Dongfeng Bai, Bingbing Liu, Yue Wang, Andreas Geiger, Yiyi Liao
Computer Vision
NeuroMamba: Multi-Perspective Feature Interaction with Visual Mamba for Neuron Segmentation
Liuyun Jiang, Yizhuo Lu, Yanchao Zhang et al. (+2 more)
Neuron segmentation is the cornerstone of reconstructing comprehensive neuronal connectomes, which is essential for deciphering the functional organization of the
brain. The irregular morphology and densely intertwined structures of
neurons make this task particularly challenging. Prevailing CNN-based methods often fail to resolve ambiguous boundaries due to the lack of long-range context, whereas
Transformer-based methods suffer from boundary imprecision caused by the loss of voxel-level details during patch partitioning. To address these limitations, we propose NeuroMamba, a multi-perspective framework that exploits the linear complexity of Mamba to enable patch-free global modeling and synergizes this with complementary local feature modeling, thereby efficiently capturing long-range dependencies while meticulously preserving fine-grained voxel details. Specifically, we design a channel-gated Boundary Discriminative Feature Extractor (BDFE) to enhance local morphological cues. Complementing this, we introduce the Spatial Continuous Feature Extractor (SCFE), which integrates a resolution-aware scanning mechanism into the Visual Mamba architecture to adaptively model global dependencies across varying data resolutions. Finally, a cross-modulation mechanism synergistically fuses these multi-perspective features. Our method demonstrates state-of-the-art performance across four public EM
datasets, validating its exceptional adaptability to both anisotropic and isotropic resolutions. The source code will be made publicly available.
Read the full abstract ▾
All authors Liuyun Jiang, Yizhuo Lu, Yanchao Zhang, Jiazheng Liu, Hua Han
Computer Vision
Class Confidence Aware Reweighting for Long Tailed Learning
Brainard Philemon Jagati, Jitendra Tembhurne, Harsh Goud et al. (+2 more)
Deep neural network models degrade significantly in the long-tailed data distribution, with the overall training data dominated by a small set of classes in the head, and the tail classes obtaining less training examples. Addressing the imbalance in the classes,
attention in the related literature was given mainly to the adjustments carried out in the decision space in terms of either corrections performed at the logit level in order to compensate class-prior bias, with the least
attention to the optimization process resulting from the adjustments introduced through the differences in the confidences among the samples. In the current study, we present the design of a class and confidence-aware re-weighting scheme for long-tailed learning. This scheme is purely based upon the loss level and has a complementary nature to the existing methods performing the adjustment of the logits. In the practical implementation stage of the proposed scheme, we use an {\Omega}(p_t, f_c) function. This function enables the modulation of the contribution towards the training task based upon the confidence value of the prediction, as well as the relative frequency of the corresponding class. Our observations in the experiments are corroborated by significant experimental results performed on the CIFAR-100-LT, ImageNet-LT, and iNaturalist2018
datasets under various values of imbalance factors that clearly authenticate the theoretical discussions above.
Read the full abstract ▾
All authors Brainard Philemon Jagati, Jitendra Tembhurne, Harsh Goud, Rudra Pratap Singh, Chandrashekhar Meshram
Computer Vision
A Multi-View Pipeline and Benchmark Dataset for 3D Hand Pose Estimation in Surgery
Valery Fischer, Alan Magdaleno, Anna-Katharina Calek et al. (+8 more)
Purpose: Accurate 3D hand pose estimation supports
surgical applications such as skill assessment,
robot-assisted interventions, and geometry-aware workflow analysis. However,
surgical environments pose severe challenges, including intense and localized lighting, frequent occlusions by instruments or staff, and uniform hand appearance due to gloves, combined with a scarcity of annotated
datasets for reliable model training. Method: We propose a robust multi-view pipeline for 3D hand pose estimation in
surgical contexts that requires no domain-specific fine-tuning and relies solely on off-the-shelf pretrained models. The pipeline integrates reliable person detection, whole-body pose estimation, and state-of-the-art 2D hand keypoint prediction on tracked hand crops, followed by a constrained 3D optimization. In addition, we introduce a novel
surgical benchmark dataset comprising over 68,000 frames and 3,000 manually annotated 2D hand poses with triangulated 3D ground truth, recorded in a replica operating room under varying levels of scene complexity. Results: Quantitative experiments demonstrate that our method consistently outperforms baselines, achieving a 31% reduction in 2D mean joint error and a 76% reduction in 3D mean per-joint position error. Conclusion: Our work establishes a strong baseline for 3D hand pose estimation in surgery, providing both a training-free pipeline and a comprehensive annotated
dataset to facilitate future research in
surgical computer
vision.
Read the full abstract ▾
All authors Valery Fischer, Alan Magdaleno, Anna-Katharina Calek, Nicola Cavalcanti, Nathan Hoffman, Christoph Germann, Joschua Wüthrich, Max Krähenmann, Mazda Farshad, Philipp Fürnstahl, Lilian Calvet