Computer Vision
Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion
Sathwik Karnik, Juyeop Kim, Sanmi Koyejo et al. (+2 more)
Text-to-image
diffusion models often memorize training data, revealing a fundamental failure to generalize beyond the training set. Current mitigation strategies typically sacrifice image quality or
prompt alignment to reduce memorization. To address this, we propose Reachability-Aware
Diffusion Steering (RADS), an inference-time framework that prevents memorization while preserving generation fidelity. RADS models the
diffusion denoising process as a dynamical system and applies concepts from reachability analysis to approximate the "backward reachable tube"–the set of intermediate states that inevitably evolve into memorized samples. We then formulate mitigation as a constrained reinforcement learning (RL) problem, where a policy learns to steer the trajectory away from memorization via minimal perturbations in the caption
embedding space. Empirical evaluations show that RADS achieves a superior Pareto frontier between generation diversity (SSCD), quality (FID), and
alignment (CLIP) compared to state-of-the-art baselines. Crucially, RADS provides robust mitigation without modifying the
diffusion backbone, offering a plug-and-play solution for safe generation. Our website is available at: https://s-karnik.github.io/rads-memorization-project-page/.
Read the full abstract ▾
All authors Sathwik Karnik, Juyeop Kim, Sanmi Koyejo, Jong-Seok Lee, Somil Bansal
Computer Vision
Towards Data-driven Nitrogen Estimation in Wheat Fields using Multispectral Images
Andreas Tritsarolis, Tomaž Bokan, Matej Brumen et al. (+2 more)
The modernization of agriculture has motivated the development of advanced analytics and decision-support systems to improve resource utilization and reduce environmental impacts. Targeted Spraying and Fertilization (TSF) is a critical operation that enables farmers to apply inputs more precisely, optimizing resource use and promoting environmental sustainability. However, accurate TSF is a challenging problem, due to external factors such as crop type, fertilization phase, soil conditions, and weather dynamics. In this paper, we present TerrAI, a Neural Network-based solution for TSF, which considers the spatio-temporal variability across different parcels. Our experimental study over a real-world remote sensing
dataset validates the soundness of TerrAI on data-driven agricultural practices.
Read the full abstract ▾
All authors Andreas Tritsarolis, Tomaž Bokan, Matej Brumen, Domen Mongus, Yannis Theodoridis
Computer Vision
Latent Replay Detection: Memory-Efficient Continual Object Detection on Microcontrollers via Task-Adaptive Compression
Bibin Wilson
Deploying object detection on microcontrollers (MCUs) enables intelligent edge devices but current models cannot learn new object categories after deployment. Existing continual learning methods require storing raw images far exceeding MCU
memory budgets of tens of kilobytes. We present Latent Replay Detection (LRD), the first framework for continual object detection under MCU
memory constraints. Our key contributions are: 1. Task-Adaptive Compression: Unlike fixed PCA, we propose learnable compression with FiLM (Feature-wise Linear Modulation) conditioning, where task specific
embeddings modulate the compression to preserve discriminative features for each task's distribution; 2. Spatial-Diverse Exemplar Selection: Traditional sampling ignores spatial information critical for detection - we select exemplars maximizing bounding box diversity via farthest-point sampling in IoU space, preventing localization bias in replay; 3. MCU-Deployable System: Our latent replay stores 150 bytes per sample versus >10KB for images, enabling a 64KB buffer to hold 400+ exemplars. Experiments on CORe50 (50 classes, 5 tasks) demonstrate that LRD achieves mAP@50 on the initial task and maintains strong performance across subsequent tasks - a significant improvement over naive fine-tuning while operating within strict MCU constraints. Our task-adaptive FiLM compression and spatial diverse exemplar selection work synergistically to preserve detection capabilities. Deployed on STM32H753ZI, ESP32-S3, and MAX78000 MCUs, LRD achieves 4.9-97.5ms latency per inference within a 64KB
memory budget-enabling practical continual detection on edge devices for the first time.
Read the full abstract ▾
Computer Vision
TinyVLM: Zero-Shot Object Detection on Microcontrollers via Vision-Language Distillation with Matryoshka Embeddings
Bibin Wilson
Zero-shot object detection enables recognising novel objects without task-specific training, but current approaches rely on large
vision language models (VLMs) like CLIP that require hundreds of megabytes of
memory - far exceeding the constraints of micro controller units (MCUs). We present TinyVLM, the first framework enabling zero-shot object detection on resource-constrained MCUs with less than 1MB of
memory. Our approach introduces three key innovations: (1) a decoupled architecture that separates visual inference from text encoding, allowing precomputed class
embeddings to be stored in flash
memory; (2) Matryoshka
distillation that trains nested
embeddings at multiple dimensions (16-256), enabling flexible accuracy-
memory trade-offs; and (3) quantized
embedding storage that reduces class prototype
memory by 4x with minimal accuracy loss. Trained on Conceptual Captions 3M (CC3M), TinyVLM achieves competitive zero-shot accuracy on COCO, Flowers102, and Food101 while requiring only 285KB of RAM and 892KB of flash
memory for the deployed
vision encoder. We demonstrate real-time inference at 26 FPS on STM32H7 and over 1,000 FPS on MAX78000 with its CNN accelerator, enabling practical zero-shot detection on edge devices for the first time.
Read the full abstract ▾
Computer Vision
Olbedo: An Albedo and Shading Aerial Dataset for Large-Scale Outdoor Environments
Shuang Song, Debao Huang, Deyan Deng et al. (+4 more)
Intrinsic image decomposition (IID) of outdoor scenes is crucial for relighting, editing, and understanding large-scale environments, but progress has been limited by the lack of real-world
datasets with reliable albedo and shading supervision. We introduce Olbedo, a large-scale aerial
dataset for outdoor albedo–shading decomposition in the wild. Olbedo contains 5,664
UAV images captured across four landscape types, multiple years, and diverse illumination conditions. Each view is accompanied by multi-view consistent albedo and shading maps, metric depth, surface normals, sun and sky shading components, camera poses, and, for recent flights, measured HDR sky domes. These annotations are derived from an inverse-rendering refinement pipeline over multi-view stereo reconstructions and calibrated sky illumination, together with per-pixel confidence masks. We demonstrate that Olbedo enables state-of-the-art
diffusion-based IID models, originally trained on synthetic indoor data, to generalize to real outdoor imagery: fine-tuning on Olbedo significantly improves single-view outdoor albedo prediction on the MatrixCity
benchmark. We further illustrate applications of Olbedo-trained models to multi-view consistent relighting of 3D assets, material editing, and scene change analysis for urban digital twins. We release the
dataset, baseline models, and an evaluation protocol to support future research in outdoor intrinsic decomposition and illumination-aware aerial
vision.
Read the full abstract ▾
All authors Shuang Song, Debao Huang, Deyan Deng, Haolin Xiong, Yang Tang, Yajie Zhao, Rongjun Qin
Computer Vision · Cross-listed from cs.LG
Shiwei Tan, Hengyi Wang, Weiyi Qin et al. (+3 more)
Multimodal Large
Language Models (MLLMs) deliver detailed responses on
vision-
language tasks, yet remain susceptible to object
hallucination (introducing objects not present in the image), undermining reliability in practice. Prior efforts often rely on heuristic penalties, post-hoc correction, or generic decoding tweaks, which do not directly intervene in the mechanisms that trigger object
hallucination and thus yield limited gains. To address this challenge, we propose a causal decoding framework that applies targeted causal interventions during generation to curb spurious object mentions. By reshaping the decoding dynamics to attenuate spurious dependencies, our approach reduces false object
tokens while maintaining descriptive quality. Across captioning and QA
benchmarks, our framework substantially lowers object-
hallucination rates and achieves state-of-the-art faithfulness without degrading overall output quality.
Read the full abstract ▾
All authors Shiwei Tan, Hengyi Wang, Weiyi Qin, Qi Xu, Zhigang Hua, Hao Wang
Computer Vision
Synergizing Understanding and Generation with Interleaved Analyzing-Drafting Thinking
Shengqiong Wu, Bobo Li, Xinkai Wang et al. (+6 more)
Unified
Vision-
Language Models (UVLMs) aim to advance
multimodal learning by supporting both understanding and generation within a single framework. However, existing approaches largely focus on architectural unification while overlooking the need for explicit interaction between the two capabilities during task solving. As a result, current models treat understanding and generation as parallel skills rather than synergistic processes. To achieve real synergy, we introduce the interleaved Analyzing-Drafting problem-solving loop (AD-Loop), a new think paradigm that dynamically alternates between analytic and drafting operations. By interleaving textual thoughts with visual thoughts, AD-Loop enables models to iteratively refine both comprehension and outputs, fostering genuine synergy. To train this mechanism, we design a two-stage strategy: supervised learning on interleaved thought data to initialize alternation, followed by reinforcement learning to promote adaptive and autonomous control. Extensive experiments demonstrate that AD-Loop consistently improves performance across standard
benchmarks for both understanding and generation, with strong transferability to various UVLMs architectures. Visual analyses further validate the effectiveness of implicit visual thoughts. These results highlight AD-Loop as a principled and broadly applicable strategy for synergizing comprehension and creation. The project page is at https://sqwu.top/AD-Loop.
Read the full abstract ▾
All authors Shengqiong Wu, Bobo Li, Xinkai Wang, Xiangtai Li, Lei Cui, Furu Wei, Shuicheng Yan, Hao Fei, Tat-seng Chua
Computer Vision
PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models
Binesh Sadanandan and Vahid Behzadan
Medical
Vision Language Models (VLMs) can change their answers when clinicians rephrase the same question, a failure mode that threatens deployment safety. We introduce PSF-Med, a
benchmark of 26,850 chest X-ray questions paired with 92,856 meaning-preserving paraphrases across MIMIC-CXR, PadChest, and VinDr-CXR, spanning clinical populations in the US, Spain, and Vietnam. Every paraphrase is validated by an LLM judge using a bidirectional clinical entailment rubric, with 91.6% cross-family agreement. Across nine VLMs, including general-purpose models, we find flip rates from 3% to 37%. However, low flip rate does not imply visual grounding: text-only baselines show that some models stay consistent even when the image is removed, suggesting they rely on
language priors. To study mechanisms in one model, we apply GemmaScope 2 Sparse Autoencoders (SAEs) to MedGemma 4B and analyze FlipBank, a curated set of 158 flip cases. We identify a sparse feature at layer 17 that correlates with
prompt framing and predicts decision margin shifts. In causal patching, removing this feature's contribution recovers 45% of the yes-minus-no logit margin on average and fully reverses 15% of flips. Acting on this finding, we show that clamping the identified feature at inference reduces flip rates by 31% relative with only a 1.3 percentage-point accuracy cost, while also decreasing text-prior reliance. These results suggest that flip rate alone is not enough; robustness evaluations should test both paraphrase stability and image reliance.
Read the full abstract ▾
Computer Vision
Automating Timed Up and Go Phase Segmentation and Gait Analysis via the tugturn Markerless 3D Pipeline
Abel Gonçalves Chinaglia, Guilherme Manna Cesar and Paulo Roberto Pereira Santiago
Instrumented Timed Up and Go (TUG) analysis can support clinical and research decision-making, but robust and reproducible markerless pipelines are still limited. We present tugturn.py, a Python-based workflow for 3D markerless TUG processing that combines phase
segmentation, gait-event detection, spatiotemporal metrics, intersegmental coordination, and dynamic stability analysis. The pipeline uses spatial thresholds to segment each trial into stand, first gait, turning, second gait, and sit phases, and applies a relative-distance strategy to detect heel-strike and toe-off events within valid gait windows. In addition to conventional kinematics, tugturn provides Vector Coding outputs and Extrapolated Center of Mass (XCoM)-based metrics. The software is configured through TOML files and produces reproducible artifacts, including HTML reports, CSV tables, and quality-assurance visual outputs. A complete runnable example is provided with test data and command-line instructions. This manuscript describes the implementation, outputs, and reproducibility workflow of tugturn as a focused software contribution for markerless biomechanical TUG analysis.
Read the full abstract ▾
Computer Vision
ECHOSAT: Estimating Canopy Height Over Space And Time
Jan Pauls, Karsten Schrödter, Sven Ligensa et al. (+7 more)
Forest monitoring is critical for
climate change mitigation. However, existing global tree height maps provide only static snapshots and do not capture temporal forest dynamics, which are essential for accurate carbon accounting. We introduce ECHOSAT, a global and temporally consistent tree height map at 10 m resolution spanning multiple years. To this end, we resort to multi-sensor satellite data to train a specialized
vision transformer model, which performs pixel-level temporal regression. A self-supervised growth loss regularizes the predictions to follow growth curves that are in line with natural tree development, including gradual height increases over time, but also abrupt declines due to forest loss events such as fires. Our experimental evaluation shows that our model improves state-of-the-art accuracies in the context of single-year predictions. We also provide the first global-scale height map that accurately quantifies tree growth and disturbances over time. We expect ECHOSAT to advance global efforts in carbon monitoring and disturbance assessment. The maps can be accessed at https://github.com/ai4forest/echosat.
Read the full abstract ▾
All authors Jan Pauls, Karsten Schrödter, Sven Ligensa, Martin Schwartz, Berkant Turan, Max Zimmer, Sassan Saatchi, Sebastian Pokutta, Philippe Ciais, Fabian Gieseke
Computer Vision
WildSVG: Towards Reliable SVG Generation Under Real-Word Conditions
Marco Terral, Haotian Zhang, Tianyang Zhang et al. (+8 more)
We introduce the task of SVG extraction, which consists in translating specific visual inputs from an image into scalable vector graphics. Existing
multimodal models achieve strong results when generating SVGs from clean renderings or textual descriptions, but they fall short in real-world scenarios where natural images introduce noise, clutter, and domain shifts. A central challenge in this direction is the lack of suitable
benchmarks. To address this need, we introduce the WildSVG
Benchmark, formed by two complementary
datasets: Natural WildSVG, built from real images containing company logos paired with their SVG annotations, and Synthetic WildSVG, which blends complex SVG renderings into real scenes to simulate difficult conditions. Together, these resources provide the first foundation for systematic benchmarking SVG extraction. We
benchmark state-of-the-art
multimodal models and find that current approaches perform well below what is needed for reliable SVG extraction in real scenarios. Nonetheless, iterative refinement methods point to a promising path forward, and model capabilities are steadily improving
Read the full abstract ▾
All authors Marco Terral, Haotian Zhang, Tianyang Zhang, Meng Lin, Xiaoqing Xie, Haoran Dai, Darsh Kaushik, Pai Peng, Nicklas Scharpff, David Vazquez, Joan Rodriguez
Computer Vision
Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentation
Asim Unmesh, Kaki Ramesh, Mayank Patel et al. (+2 more)
Temporal Action
Segmentation (TAS) requires dividing
videos into action segments, yet the vast space of activities and alternative breakdowns makes collecting comprehensive
datasets infeasible. Existing methods remain limited to closed vocabularies and fixed label sets. In this work, we explore the largely unexplored problem of Open-Vocabulary Zero-Shot Temporal Action
Segmentation (OVTAS) by leveraging the strong zero-shot capabilities of
Vision-
Language Models (VLMs). We introduce a training-free pipeline that follows a
segmentation-by-classification design: Frame-Action
Embedding Similarity (FAES) matches
video frames to candidate action labels, and Similarity-
Matrix Temporal
Segmentation (SMTS) enforces temporal consistency. Beyond proposing OVTAS, we present a systematic study across 14 diverse VLMs, providing the first broad analysis of their suitability for open-vocabulary action
segmentation. Experiments on standard
benchmarks show that OVTAS achieves strong results without task-specific supervision, underscoring the potential of VLMs for structured temporal understanding.
Read the full abstract ▾
All authors Asim Unmesh, Kaki Ramesh, Mayank Patel, Rahul Jain, Karthik Ramani
Computer Vision
FlowFixer: Towards Detail-Preserving Subject-Driven Generation
Jinyoung Jun, Won-Dong Jang, Wenbin Ouyang et al. (+2 more)
We present FlowFixer, a refinement framework for subject-driven generation (SDG) that restores fine details lost during generation caused by changes in scale and perspective of a subject. FlowFixer proposes direct image-to-image translation from visual references, avoiding ambiguities in
language prompts. To enable image-to-image training, we introduce a one-step denoising scheme to generate self-supervised training data, which automatically removes high-frequency details while preserving global structure, effectively simulating real-world SDG errors. We further propose a keypoint matching-based metric to properly assess fidelity in details beyond semantic similarities usually measured by CLIP or DINO. Experimental results demonstrate that FlowFixer outperforms state-of-the-art SDG methods in both qualitative and quantitative evaluations, setting a new
benchmark for high-fidelity subject-driven generation.
Read the full abstract ▾
All authors Jinyoung Jun, Won-Dong Jang, Wenbin Ouyang, Raghudeep Gadde, Jungbeom Lee
Computer Vision · Cross-listed from cs.LG
FedVG: Gradient-Guided Aggregation for Enhanced Federated Learning
Alina Devkota, Jacob Thrasher, Donald Adjeroh et al. (+2 more)
Federated Learning (FL) enables collaborative model training across multiple clients without sharing their private data. However, data heterogeneity across clients leads to client drift, which degrades the overall generalization performance of the model. This effect is further compounded by overemphasis on poorly performing clients. To address this problem, we propose FedVG, a novel gradient-based
federated aggregation framework that leverages a global validation set to guide the optimization process. Such a global validation set can be established using readily available public
datasets, ensuring accessibility and consistency across clients without compromising
privacy. In contrast to conventional approaches that prioritize client
dataset volume, FedVG assesses the generalization ability of client models by measuring the magnitude of validation gradients across layers. Specifically, we compute layerwise gradient norms to derive a client-specific score that reflects how much each client needs to adjust for improved generalization on the global validation set, thereby enabling more informed and adaptive
federated aggregation. Extensive experiments on both natural and medical image benchmarking
datasets, across diverse model architectures, demonstrate that FedVG consistently improves performance, particularly in highly heterogeneous settings. Moreover, FedVG is modular and can be seamlessly integrated with various state-of-the-art FL
algorithms, often further improving their results. Our code is available at https://github.com/alinadevkota/FedVG.
Read the full abstract ▾
All authors Alina Devkota, Jacob Thrasher, Donald Adjeroh, Binod Bhattarai, Prashnna K. Gyawali
Computer Vision
MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation
Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh et al. (+1 more)
Prompt learning has become a dominant paradigm for adapting
vision-
language models (VLMs) such as CLIP to downstream tasks without modifying pretrained weights. While extending
prompts to both
vision and text encoders across multiple
transformer layers significantly boosts performance, it dramatically increases the number of trainable parameters, with state-of-the-art methods requiring millions of parameters and abandoning the parameter efficiency that makes
prompt tuning attractive. In this work, we propose MMLoP (Multi-Modal Low-Rank Prompting), a framework that achieves deep multi-modal prompting with only 11.5K trainable parameters, comparable to early text-only methods like CoOp. MMLoP parameterizes
vision and text
prompts at each
transformer layer through a low-rank factorization that constrains
prompts to a compact subspace, providing parameter efficiency while motivating the need for our complementary regularization components. To further close the accuracy gap with state-of-the-art methods, we introduce three complementary components: a self-regulating consistency loss that anchors prompted representations to frozen zero-shot CLIP features at both the feature and logit levels, a uniform drift correction that removes the global
embedding shift induced by
prompt tuning to preserve class-discriminative structure, and a shared up-projection that couples
vision and text
prompts through a common low-rank factor to enforce cross-modal
alignment. Extensive experiments across three
benchmarks and 11 diverse
datasets demonstrate that MMLoP achieves a highly favorable accuracy-efficiency tradeoff, outperforming the majority of existing methods including those with orders of magnitude more parameters, while achieving a harmonic mean of 79.70% on base-to-novel generalization. Code is available at https://github.com/sajjad-ucsb/MMLoP.
Read the full abstract ▾
All authors Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani
Computer Vision
Momentum Memory for Knowledge Distillation in Computational Pathology
Yongxin Guo, Hao Lu, Onur C. Koyun et al. (+3 more)
Multimodal learning that integrates genomics and histopathology has shown strong potential in cancer diagnosis, yet its clinical translation is hindered by the limited availability of paired histology-genomics data. Knowledge
distillation (KD) offers a practical solution by transferring genomic supervision into histopathology models, enabling accurate inference using histology alone. However, existing KD methods rely on batch-local
alignment, which introduces instability due to limited within-batch comparisons and ultimately degrades performance. To address these limitations, we propose Momentum
Memory Knowledge
Distillation (MoMKD), a cross-modal
distillation framework driven by a momentum-updated
memory. This
memory aggregates genomic and histopathology information across batches, effectively enlarging the supervisory context available to each mini-batch. Furthermore, we decouple the gradients of the genomics and histology branches, preventing genomic signals from dominating histology feature learning during training and eliminating the modality-gap issue at inference time. Extensive experiments on the TCGA-BRCA
benchmark (HER2, PR, and ODX classification tasks) and an independent in-house testing
dataset demonstrate that MoMKD consistently outperforms state-of-the-art MIL and
multimodal KD baselines, delivering strong performance and generalization under histology-only inference. Overall, MoMKD establishes a robust and generalizable knowledge
distillation paradigm for computational pathology.
Read the full abstract ▾
All authors Yongxin Guo, Hao Lu, Onur C. Koyun, Zhengjie Zhu, Muhammet Fatih Demir, Metin Nafi Gurcan
Computer Vision
Towards Controllable Video Synthesis of Routine and Rare OR Events
Dominik Schneider, Lalithkumar Seenivasan, Sampath Rapuri et al. (+8 more)
Purpose: Curating large-scale
datasets of operating room (OR) workflow, encompassing rare, safety-critical, or atypical events, remains operationally and ethically challenging. This data bottleneck complicates the development of ambient intelligence for detecting, understanding, and mitigating rare or safety-critical events in the OR. Methods: This work presents an OR
video diffusion framework that enables controlled synthesis of rare and safety-critical events. The framework integrates a
geometric abstraction module, a conditioning module, and a fine-tuned
diffusion model to first transform OR scenes into abstract
geometric representations, then condition the synthesis process, and finally generate realistic OR event
videos. Using this framework, we also curate a synthetic
dataset to train and validate AI models for detecting near-misses of sterile-field violations. Results: In synthesizing routine OR events, our method outperforms off-the-shelf
video diffusion baselines, achieving lower FVD/LPIPS and higher SSIM/PSNR in both in- and out-of-domain
datasets. Through qualitative results, we illustrate its ability for controlled
video synthesis of counterfactual events. An AI model trained and validated on the generated synthetic data achieved a RECALL of 70.13% in detecting near safety-critical events. Finally, we conduct an ablation study to quantify performance gains from key design choices. Conclusion: Our solution enables controlled synthesis of routine and rare OR events from abstract
geometric representations. Beyond demonstrating its capability to generate rare and safety-critical scenarios, we show its potential to support the development of ambient intelligence models.
Read the full abstract ▾
All authors Dominik Schneider, Lalithkumar Seenivasan, Sampath Rapuri, Vishalroshan Anil, Aiza Maksutova, Yiqing Shen, Jan Emily Mangulabnan, Hao Ding, Jose L. Porras, Masaru Ishii, Mathias Unberath
Computer Vision · Cross-listed from physics.optics
A unified self-supervised framework for single-frame Fresnel CDI and overlapped ptychography
Oliver Hoidn, Steven Henke, Albert Vong et al. (+3 more)
Ptychographic imaging at synchrotron and X-ray free-electron
laser sources requires densely overlapping scans, which limits throughput and increases dose; extending coherent diffractive imaging to overlap-free operation on extended samples remains an open problem. We present a self-supervised inverse-mapping network for single-frame Fresnel coherent diffraction imaging (CDI) and overlapped ptychography with fixed, pre-estimated probes. The learned neural network reconstructs individual object patches from either one diffraction frame or several overlapping measurements at a time. In single-frame mode, the phase diversity provided by the curved-wavefront probe at the off-focus sample position removes the requirement for overlap constraints, enabling sparser scans and proportionally lower dose at fixed exposure. On synthetic line patterns, reconstructed amplitude SSIM exceeds 0.90 in single-frame mode with the curved probe and reaches 0.952-0.968 with overlap constraints. Optimization of the network via a Poisson negative log likelihood objective, rather than the more common mean absolute error, yields 10-fold improved
photon-dose efficiency at doses below $10^5$
photons per image, where shot noise typically limits resolution. In addition to these synthetic studies, we demonstrate robust single-frame reconstruction of extended samples using ptychographic
datasets from APS and LCLS, with end-to-end reconstruction of a 10,304-frame workload approximately $36\times$ faster than a highly optimized iterative solver. Together, these results unify single-frame Fresnel CDI and overlapped ptychography within one self-supervised framework, supporting dose-efficient, high-throughput imaging at modern light sources.
Read the full abstract ▾
All authors Oliver Hoidn, Steven Henke, Albert Vong, Aashwin Mishra, Apurva Mehta, Matthew Seaberg
Computer Vision · Cross-listed from eess.IV
RelA-Diffusion: Relativistic Adversarial Diffusion for Multi-Tracer PET Synthesis from Multi-Sequence MRI
Minhui Yu, Yongheng Sun, David S. Lalush et al. (+3 more)
Multi-tracer positron emission
tomography (PET) provides critical insights into diverse neuropathological processes such as tau accumulation, neuroinflammation, and $\beta$-amyloid deposition in the
brain, making it indispensable for comprehensive neurological assessment. However, routine acquisition of multi-tracer PET is limited by high costs, radiation exposure, and restricted tracer availability. Recent efforts have explored deep learning approaches for synthesizing PET images from structural MRI. While some methods rely solely on T1-weighted MRI, others incorporate additional sequences such as T2-FLAIR to improve pathological sensitivity. However, existing methods often struggle to capture fine-grained anatomical and pathological details, resulting in artifacts and unrealistic outputs. To this end, we propose RelA-
Diffusion, a Relativistic
Adversarial Diffusion framework for multi-tracer PET synthesis from multi-sequence MRI. By leveraging both T1-weighted and T2-FLAIR scans as complementary inputs, RelA-
Diffusion captures richer structural information to guide image generation. To improve synthesis fidelity, we introduce a gradient-penalized relativistic
adversarial loss to the intermediate clean predictions of the
diffusion model. This loss compares real and generated images in a relative manner, encouraging the synthesis of more realistic local structures. Both the relativistic formulation and the gradient penalty contribute to stabilizing the training, while
adversarial feedback at each
diffusion timestep enables consistent refinement throughout the generation process. Extensive experiments on two
datasets demonstrate that RelA-
Diffusion outperforms existing methods in both visual fidelity and quantitative metrics, highlighting its potential for accurate synthesis of multi-tracer PET.
Read the full abstract ▾
All authors Minhui Yu, Yongheng Sun, David S. Lalush, Jason P Mihalik, Pew-Thian Yap, Mingxia Liu
Computer Vision
Evan Kim, Hyunwoo Ryu, Thomas W. Mitchel et al. (+1 more)
Geometry-free view synthesis
transformers have recently achieved state-of-the-art performance in Novel View Synthesis (NVS), outperforming traditional approaches that rely on explicit geometry modeling. Yet the factors governing their scaling with compute remain unclear. We present a systematic study of scaling laws for view synthesis
transformers and derive design principles for training compute-optimal NVS models. Contrary to prior findings, we show that encoder-decoder architectures can be compute-optimal; we trace earlier negative results to suboptimal architectural choices and comparisons across unequal training compute budgets. Across several compute levels, we demonstrate that our encoder-decoder architecture, which we call the Scalable View Synthesis Model (SVSM), scales as effectively as decoder-only models, achieves a superior performance-compute Pareto frontier, and surpasses the previous state-of-the-art on real-world NVS
benchmarks with substantially reduced training compute.
Read the full abstract ▾
All authors Evan Kim, Hyunwoo Ryu, Thomas W. Mitchel, Vincent Sitzmann
Computer Vision
HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles
Yifan Wang, Francesco Pittaluga, Zaid Tasneem et al. (+3 more)
Controllable driving scene generation is critical for realistic and scalable autonomous driving simulation, yet existing approaches struggle to jointly achieve photorealism and precise control. We introduce HorizonForge, a unified framework that reconstructs scenes as editable Gaussian Splats and Meshes, enabling fine-grained 3D manipulation and
language-driven vehicle insertion. Edits are rendered through a noise-aware
video diffusion process that enforces spatial and temporal consistency, producing diverse scene variations in a single feed-forward pass without per-trajectory optimization. To standardize evaluation, we further propose HorizonSuite, a comprehensive
benchmark spanning ego- and
agent-level editing tasks such as trajectory modifications and object manipulation. Extensive experiments show that Gaussian-Mesh representation delivers substantially higher fidelity than alternative 3D representations, and that temporal priors from
video diffusion are essential for coherent synthesis. Combining these findings, HorizonForge establishes a simple yet powerful paradigm for photorealistic, controllable driving simulation, achieving an 83.4% user-preference gain and a 25.19% FID improvement over the second best state-of-the-art method. Project page: https://horizonforge.github.io/ .
Read the full abstract ▾
All authors Yifan Wang, Francesco Pittaluga, Zaid Tasneem, Chenyu You, Manmohan Chandraker, Ziyu Jiang
Computer Vision · Cross-listed from cs.LG
Uncertainty-Aware Diffusion Model for Multimodal Highway Trajectory Prediction via DDIM Sampling
Marion Neumeier, Niklas Roßberg, Michael Botsch et al. (+1 more)
Accurate and uncertainty-aware trajectory prediction remains a core challenge for autonomous driving, driven by complex multi-
agent interactions, diverse scene contexts and the inherently
stochastic nature of future motion.
Diffusion-based generative models have recently shown strong potential for capturing
multimodal futures, yet existing approaches such as cVMD suffer from slow sampling, limited exploitation of generative diversity and brittle scenario encodings. This work introduces cVMDx, an enhanced
diffusion-based trajectory prediction framework that improves efficiency, robustness and
multimodal predictive capability. Through DDIM sampling, cVMDx achieves up to a 100x reduction in inference time, enabling practical multi-sample generation for uncertainty estimation. A fitted Gaussian Mixture Model further provides tractable
multimodal predictions from the generated trajectories. In addition, a CVQ-VAE variant is evaluated for scenario encoding. Experiments on the publicly available highD
dataset show that cVMDx achieves higher accuracy and significantly improved efficiency over cVMD, enabling fully
stochastic,
multimodal trajectory prediction.
Read the full abstract ▾
All authors Marion Neumeier, Niklas Roßberg, Michael Botsch, Wolfgang Utschick
Computer Vision
StoryTailor:A Zero-Shot Pipeline for Action-Rich Multi-Subject Visual Narratives
Jinghao Hu, Yuhe Zhang, GuoHua Geng et al. (+2 more)
Generating multi-frame, action-rich visual narratives without fine-tuning faces a threefold tension: action text faithfulness, subject identity fidelity, and cross-frame background continuity. We propose StoryTailor, a zero-shot pipeline that runs on a single RTX 4090 (24 GB) and produces temporally coherent, identity-preserving image sequences from a long narrative
prompt, per-subject references, and grounding boxes. Three synergistic modules drive the system: Gaussian-Centered
Attention (GCA) to dynamically focus on each subject core and ease grounding-box overlaps; Action-Boost Singular Value Reweighting (AB-SVR) to amplify action-related directions in the text
embedding space; and Selective Forgetting Cache (SFC) that retains transferable background cues, forgets nonessential history, and selectively surfaces retained cues to build cross-scene semantic ties. Compared with baseline methods, experiments show that CLIP-T improves by up to 10-15%, with DreamSim lower than strong baselines, while CLIP-I stays in a visually acceptable, competitive range. With matched resolution and steps on a 24 GB GPU, inference is faster than FluxKontext. Qualitatively, StoryTailor delivers expressive interactions and evolving yet stable scenes.
Read the full abstract ▾
All authors Jinghao Hu, Yuhe Zhang, GuoHua Geng, Kang Li, Han Zhang
Computer Vision · Cross-listed from cs.LG
Test-Time Training with KV Binding Is Secretly Linear Attention
Junchen Liu, Sven Elflein, Or Litany et al. (+2 more)
Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear
attention operator. Beyond explaining previously puzzling model behaviors, this perspective yields multiple practical benefits: it enables principled architectural simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and provides a systematic reduction of diverse TTT variants to a standard linear
attention form. Overall, our results reframe TTT not as test-time memorization, but as learned linear
attention with enhanced representational capacity. Project page: https://research.nvidia.com/labs/sil/projects/tttla/.
Read the full abstract ▾
All authors Junchen Liu, Sven Elflein, Or Litany, Zan Gojcic, Ruilong Li
Computer Vision · Cross-listed from cs.RO
Squint: Fast Visual Reinforcement Learning for Sim-to-Real Robotics
Abdulaziz Almuzairee and Henrik I. Christensen
Visual reinforcement learning is appealing for robotics but expensive. Off-policy methods are sample-efficient yet slow while on-policy methods parallelize well but waste samples. Recent work has shown that off-policy methods can train faster than on-policy methods in wall-clock time for state-based control. Extending this to
vision remains challenging, where high-dimensional input images complicate training dynamics and introduce substantial storage and encoding overhead. To address these challenges, we introduce Squint, a visual Soft Actor Critic method that achieves faster wall-clock training than prior visual off-policy and on-policy methods. Squint achieves this via parallel simulation, a distributional critic, resolution squinting, layer normalization, a tuned update-to-data ratio, and an optimized implementation. We evaluate on the SO-101 Task Set, a new suite of eight manipulation tasks in ManiSkill3 with heavy domain randomization, and demonstrate sim-to-real transfer to a real SO-101
robot. We train policies for 15 minutes on a single RTX 3090 GPU, with most tasks converging in under 6 minutes.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.IR
Multi-Vector Index Compression in Any Modality
Hanxiang Qin, Alexander Martin, Rohan Jha et al. (+3 more)
We study efficient multi-vector
retrieval for late interaction in any modality. Late interaction has emerged as a dominant paradigm for information
retrieval in text, images, visual documents, and
videos, but its computation and storage costs grow linearly with document length, making it costly for image-,
video-, and audio-rich corpora. To address this limitation, we explore query-agnostic methods for compressing multi-vector document representations under a constant vector budget. We introduce four approaches for index compression: sequence resizing,
memory tokens, hierarchical pooling, and a novel
attention-guided clustering (AGC). AGC uses an
attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight
token aggregation. Evaluating these methods on
retrieval tasks spanning text (BEIR), visual-document (ViDoRe), and
video (MSR-VTT, MultiVENT 2.0), we show that
attention-guided clustering consistently outperforms other parameterized compression methods (sequence resizing and
memory tokens), provides greater flexibility in index size than non-parametric hierarchical clustering, and achieves competitive or improved performance compared to a full, uncompressed index. The source code is available at: github.com/hanxiangqin/omni-col-press.
Read the full abstract ▾
All authors Hanxiang Qin, Alexander Martin, Rohan Jha, Chunsheng Zuo, Reno Kriz, Benjamin Van Durme
Computer Vision · Cross-listed from cs.LG
Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs
Yining Hong, Huang Huang, Manling Li et al. (+4 more)
Embodied LLMs endow
robots with high-level task
reasoning, but they cannot reflect on what went wrong or why, turning deployment into a sequence of independent trials where mistakes repeat rather than accumulate into experience. Drawing upon human reflective practitioners, we introduce Reflective Test-Time Planning, which integrates two modes of reflection: reflection-in-action, where the
agent uses test-time scaling to generate and score multiple candidate actions using internal reflections before execution; and reflection-on-action, which uses test-time training to update both its internal reflection model and its action policy based on external reflections after execution. We also include retrospective reflection, allowing the
agent to re-evaluate earlier decisions and perform model updates with hindsight for proper long-horizon credit assignment. Experiments on our newly-designed Long-Horizon Household
benchmark and MuJoCo Cupboard Fitting
benchmark show significant gains over baseline models, with zero-shot generalization to photorealistic HM3D environments and real-
robot experiments on a Franka Panda arm. Ablations confirm that reflection-in-action and reflection-on-action are mutually dependent, and that retrospective reflection achieves better credit assignment than step-wise external feedback at lower computational overhead. Qualitative analyses further highlight behavioral correction through reflection.
Read the full abstract ▾
All authors Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Leonidas Guibas, Jiajun Wu, Yejin Choi
Computer Vision
Region of Interest Segmentation and Morphological Analysis for Membranes in Cryo-Electron Tomography
Xingyi Cheng, Julien Maufront, Aurélie Di Cicco et al. (+3 more)
Cryo-electron
tomography (cryo-ET) enables high resolution, three-dimensional reconstruction of biological structures, including membranes and membrane
proteins. Identification of regions of interest (ROIs) is central to scientific imaging, as it enables isolation and quantitative analysis of specific structural features within complex
datasets. In practice, however, ROIs are typically derived indirectly through full structure
segmentation followed by post hoc analysis. This limitation is especially apparent for continuous and geometrically complex structures such as membranes, which are segmented as single entities. Here, we developed TomoROIS-SurfORA, a two step framework for direct, shape-agnostic ROI
segmentation and morphological surface analysis. TomoROIS performs deep learning-based ROI
segmentation and can be trained from scratch using small annotated
datasets, enabling practical application across diverse imaging data. SurfORA processes segmented structures as point clouds and surface meshes to extract quantitative morphological features, including inter-membrane distances,
curvature, and surface roughness. It supports both closed and open surfaces, with specific considerations for open surfaces, which are common in cryo-ET due to the missing wedge effect. We demonstrate both tools using in vitro reconstituted membrane systems containing deformable vesicles with complex geometries, enabling automatic quantitative analysis of membrane contact sites and remodeling events such as invagination. While demonstrated here on cryo-ET membrane data, the combined approach is applicable to ROI detection and surface analysis in broader scientific imaging contexts.
Read the full abstract ▾
All authors Xingyi Cheng, Julien Maufront, Aurélie Di Cicco, Daniël M. Pelt, Manuela Dezi, Daniel Lévy
Computer Vision
Human Video Generation from a Single Image with 3D Pose and View Control
Tiantian Wang, Chun-Han Yao, Tao Hu et al. (+3 more)
Recent
diffusion methods have made significant progress in generating
videos from single images due to their powerful visual generation capabilities. However, challenges persist in image-to-
video synthesis, particularly in human
video generation, where inferring view-consistent, motion-dependent clothing wrinkles from a single image remains a formidable problem. In this paper, we present Human
Video Generation in 4D (HVG), a latent
video diffusion model capable of generating high-quality, multi-view, spatiotemporally coherent human
videos from a single image with 3D pose and view control. HVG achieves this through three key designs: (i) Articulated Pose Modulation, which captures the anatomical relationships of 3D joints via a novel dual-dimensional bone map and resolves self-occlusions across views by introducing 3D information; (ii) View and Temporal
Alignment, which ensures multi-view consistency and
alignment between a reference image and pose sequences for frame-to-frame stability; and (iii) Progressive Spatio-Temporal Sampling with temporal
alignment to maintain smooth transitions in long multi-view animations. Extensive experiments on image-to-
video tasks demonstrate that HVG outperforms existing methods in generating high-quality 4D human
videos from diverse human images and pose inputs.
Read the full abstract ▾
All authors Tiantian Wang, Chun-Han Yao, Tao Hu, Mallikarjun Byrasandra Ramalinga Reddy, Ming-Hsuan Yang, Varun Jampani
Computer Vision
Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning
Haoyi Jiang, Liu Liu, Xinjie Wang et al. (+5 more)
Vision-
language models excel at 2D visual understanding but remain limited in 3D spatial
reasoning. Existing approaches either depend on explicit 3D modalities, which limits scalability, or inject partial, view-conditioned
geometric priors and leave the
language model to recover global scene structure from sparse cues. We introduce Spa3R, a self-supervised framework that learns a unified, view-invariant spatial representation from unposed multi-view RGB images. Its Predictive Spatial Field Modeling objective compresses context views into a compact latent representation and predicts aligned
geometric and semantic feature fields at novel viewpoints, thereby encouraging coherent encoding of scene geometry and layout. We integrate the pre-trained Spa3R Encoder into a
vision-
language model through a lightweight residual cross-
attention adapter, yielding Spa3-VLM and grounding
language reasoning in global spatial context. Spa3-VLM achieves an average score of 58.6% on VSI-Bench and delivers leading or competitive performance across three additional spatial
reasoning benchmarks. These results demonstrate that predictive spatial representation learning provides an effective visual foundation for 3D
reasoning.
Read the full abstract ▾
All authors Haoyi Jiang, Liu Liu, Xinjie Wang, Yonghao He, Wei Sui, Zhizhong Su, Wenyu Liu, Xinggang Wang