Computer Vision · Cross-listed from eess.IV
Quantum walk inspired JPEG compression of images
Abhishek Verma, Sahil Tomar and Sandeep Kumar
This work proposes a
quantum inspired adaptive quantization framework that enhances the classical JPEG compression by introducing a learned, optimized Qtable derived using a
Quantum Walk Inspired Optimization (QWIO) search strategy. The optimizer searches a continuous parameter space of frequency band scaling factors under a unified rate distortion objective that jointly considers reconstruction fidelity and compression efficiency. The proposed framework is evaluated on MNIST, CIFAR10, and ImageNet subsets, using Peak Signal to Noise Ratio (PSNR), Structural Similarity Index (SSIM), Bits Per Pixel (BPP), and error heatmap visual analysis as evaluation metrics. Experimental results show average gains ranging from 3 to 6 dB PSNR, along with better structural preservation of edges, contours, and luminance transitions, without modifying decoder compatibility. The structure remains JPEG compliant and can be implemented using accessible scientific packages making it ideal for deployment and practical research use.
Read the full abstract ▾
Computer Vision
Stroke of Surprise: Progressive Semantic Illusions in Vector Sketching
Huai-Hsun Cheng, Siang-Ling Zhang and Yu-Lun Liu
Visual illusions traditionally rely on spatial manipulations such as multi-view consistency. In this work, we introduce Progressive Semantic Illusions, a novel vector sketching task where a single sketch undergoes a dramatic semantic transformation through the sequential addition of strokes. We present Stroke of Surprise, a generative framework that optimizes vector strokes to satisfy distinct semantic interpretations at different drawing stages. The core challenge lies in the "dual-constraint": initial prefix strokes must form a coherent object (e.g., a duck) while simultaneously serving as the structural foundation for a second concept (e.g., a sheep) upon adding delta strokes. To address this, we propose a sequence-aware joint optimization framework driven by a dual-branch Score
Distillation Sampling (SDS) mechanism. Unlike sequential approaches that freeze the initial state, our method dynamically adjusts prefix strokes to discover a "common structural subspace" valid for both targets. Furthermore, we introduce a novel Overlay Loss that enforces spatial complementarity, ensuring structural integration rather than occlusion. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines in recognizability and illusion strength, successfully expanding visual anagrams from the spatial to the temporal dimension. Project page: https://stroke-of-surprise.github.io/
Read the full abstract ▾
Computer Vision
UniT: Unified Multimodal Chain-of-Thought Test-time Scaling
Leon Liangyu Chen, Haoyu Ma, Zhipeng Fan et al. (+11 more)
Unified models can handle both
multimodal understanding and generation within a single architecture, yet they typically operate in a single pass without iteratively refining their outputs. Many
multimodal tasks, especially those involving complex spatial compositions, multiple interacting objects, or evolving instructions, require decomposing instructions, verifying intermediate results, and making iterative corrections. While test-time scaling (TTS) has demonstrated that allocating additional inference compute for iterative
reasoning substantially improves
language model performance, extending this paradigm to unified
multimodal models remains an open challenge. We introduce UniT, a framework for
multimodal chain-of-thought test-time scaling that enables a single unified model to reason, verify, and refine across multiple rounds. UniT combines agentic data synthesis, unified model training, and flexible test-time inference to elicit cognitive behaviors including verification, subgoal decomposition, and content
memory. Our key findings are: (1) unified models trained on short
reasoning trajectories generalize to longer inference chains at test time; (2) sequential chain-of-thought
reasoning provides a more scalable and compute-efficient TTS strategy than parallel sampling; (3) training on generation and editing trajectories improves out-of-distribution visual
reasoning. These results establish
multimodal test-time scaling as an effective paradigm for advancing both generation and understanding in unified models.
Read the full abstract ▾
All authors Leon Liangyu Chen, Haoyu Ma, Zhipeng Fan, Ziqi Huang, Animesh Sinha, Xiaoliang Dai, Jialiang Wang, Zecheng He, Jianwei Yang, Chunyuan Li, Junzhe Sun, Chu Wang, Serena Yeung-Levy, Felix Juefei-Xu
Computer Vision
MonarchRT: Efficient Attention for Real-Time Video Generation
Krish Agarwal, Zhuoming Chen, Cheng Luo et al. (+5 more)
Real-time
video generation with
Diffusion Transformers is bottlenecked by the quadratic cost of 3D self-
attention, especially in real-time regimes that are both few-step and autoregressive, where errors compound across time and each denoising step must carry substantially more information. In this setting, we find that prior sparse-
attention approximations break down, despite showing strong results for bidirectional, many-step
diffusion. Specifically, we observe that
video attention is not reliably sparse, but instead combines pronounced periodic structure driven by spatiotemporal position with dynamic, sparse semantic correspondences and dense mixing, exceeding the representational capacity of even oracle top-k
attention. Building on this insight, we propose Monarch-RT, a structured
attention parameterization for
video diffusion models that factorizes
attention using Monarch
matrices. Through appropriately aligned block structure and our extended tiled Monarch parameterization, we achieve high expressivity while preserving computational efficiency. We further overcome the overhead of parameterization through finetuning, with custom Triton kernels. We first validate the high efficacy of Monarch-RT over existing sparse baselines designed only for bidirectional models. We further observe that Monarch-RT attains up to 95%
attention sparsity with no loss in quality when applied to the state-of-the-art model Self-Forcing, making Monarch-RT a pioneering work on highly-capable sparse
attention parameterization for real-time
video generation. Our optimized implementation outperforms FlashAttention-2, FlashAttention-3, and FlashAttention-4 kernels on Nvidia RTX 5090, H100, and B200 GPUs respectively, providing kernel speedups in the range of 1.4-11.8X. This enables us, for the first time, to achieve true real-time
video generation with Self-Forcing at 16 FPS on a single RTX 5090.
Read the full abstract ▾
All authors Krish Agarwal, Zhuoming Chen, Cheng Luo, Yongqi Chen, Haizhong Zheng, Xun Huang, Atri Rudra, Beidi Chen
Computer Vision · Cross-listed from cs.NE
Energy-Aware Spike Budgeting for Continual Learning in Spiking Neural Networks for Neuromorphic Vision
Anika Tabassum Meem, Muntasir Hossain Nadid and Md Zesun Ahmed Mia
Neuromorphic
vision systems based on spiking neural networks (SNNs) offer ultra-low-power perception for event-based and frame-based cameras, yet catastrophic forgetting remains a critical barrier to deployment in continually evolving environments. Existing continual learning methods, developed primarily for artificial neural networks, seldom jointly optimize accuracy and energy efficiency, with particularly limited exploration on event-based
datasets. We propose an energy-aware spike budgeting framework for continual SNN learning that integrates experience replay, learnable leaky integrate-and-fire
neuron parameters, and an adaptive spike scheduler to enforce
dataset-specific energy constraints during training. Our approach exhibits modality-dependent behavior: on frame-based
datasets (MNIST, CIFAR-10), spike budgeting acts as a sparsity-inducing regularizer, improving accuracy while reducing spike rates by up to 47%; on event-based
datasets (DVS-Gesture, N-MNIST, CIFAR-10-DVS), controlled budget relaxation enables accuracy gains up to 17.45 percentage points with minimal computational overhead. Across five
benchmarks spanning both modalities, our method demonstrates consistent performance improvements while minimizing dynamic power consumption, advancing the practical viability of continual learning in neuromorphic
vision systems.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Training
Miaosen Zhang, Yishan Liu, Shuxia Lin et al. (+8 more)
Supervised fine-tuning (SFT) is computationally efficient but often yields inferior generalization compared to reinforcement learning (RL). This gap is primarily driven by RL's use of on-policy data. We propose a framework to bridge this chasm by enabling On-Policy SFT. We first present \textbf{Distribution Discriminant Theory (DDT)}, which explains and quantifies the
alignment between data and the model-induced distribution. Leveraging DDT, we introduce two complementary techniques: (i) \textbf{In-Distribution Finetuning (IDFT)}, a loss-level method to enhance generalization ability of SFT, and (ii) \textbf{Hinted Decoding}, a data-level technique that can re-align the training corpus to the model's distribution. Extensive experiments demonstrate that our framework achieves generalization performance surpassing prominent offline RL
algorithms, including DPO and SimPO, while maintaining the efficiency of an SFT pipeline. The proposed framework thus offers a practical alternative in domains where RL is infeasible. We open-source the code here: https://github.com/zhangmiaosen2000/Towards-On-Policy-SFT
Read the full abstract ▾
All authors Miaosen Zhang, Yishan Liu, Shuxia Lin, Xu Yang, Qi Dai, Chong Luo, Weihao Jiang, Peng Hou, Anxiang Zeng, Xin Geng, Baining Guo
Computer Vision
Best of Both Worlds: Multimodal Reasoning and Generation via Unified Discrete Flow Matching
Onkar Susladkar, Tushar Prakash, Gayatri Deshmukh et al. (+8 more)
We propose UniDFlow, a unified discrete flow-matching framework for
multimodal understanding, generation, and editing. It decouples understanding and generation via task-specific low-rank adapters, avoiding objective interference and representation
entanglement, while a novel reference-based
multimodal preference
alignment optimizes relative outcomes under identical conditioning, improving faithfulness and controllability without large-scale retraining. UniDFlpw achieves SOTA performance across eight
benchmarks and exhibits strong zero-shot generalization to tasks including inpainting, in-context image generation, reference-based editing, and compositional generation, despite no explicit task-specific training.
Read the full abstract ▾
All authors Onkar Susladkar, Tushar Prakash, Gayatri Deshmukh, Kiet A. Nguyen, Jiaxun Zhang, Adheesh Juvekar, Tianshu Bao, Lin Chai, Sparsh Mittal, Inderjit S Dhillon, Ismini Lourentzou
Computer Vision
DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing
Dianyi Wang, Ruihang Li, Feng Han et al. (+17 more)
Current unified
multimodal models for image generation and editing typically rely on massive parameter scales (e.g., >10B), entailing prohibitive training costs and deployment footprints. In this work, we present DeepGen 1.0, a lightweight 5B unified model that achieves comprehensive capabilities competitive with or surpassing much larger counterparts. To overcome the limitations of compact models in semantic understanding and fine-grained control, we introduce Stacked Channel Bridging (SCB), a deep
alignment framework that extracts hierarchical features from multiple VLM layers and fuses them with learnable 'think
tokens' to provide the generative backbone with structured,
reasoning-rich guidance. We further design a data-centric training strategy spanning three progressive stages: (1)
Alignment Pre-training on large-scale image-text pairs and editing triplets to synchronize VLM and DiT representations, (2) Joint Supervised Fine-tuning on a high-quality mixture of generation, editing, and
reasoning tasks to foster omni-capabilities, and (3) Reinforcement Learning with MR-GRPO, which leverages a mixture of reward functions and supervision signals, resulting in substantial gains in generation quality and
alignment with human preferences, while maintaining stable training progress and avoiding visual artifacts. Despite being trained on only ~50M samples, DeepGen 1.0 achieves leading performance across diverse
benchmarks, surpassing the 80B HunyuanImage by 28% on WISE and the 27B Qwen-Image-Edit by 37% on UniREditBench. By open-sourcing our training code, weights, and
datasets, we provide an efficient, high-performance alternative to democratize unified
multimodal research.
Read the full abstract ▾
All authors Dianyi Wang, Ruihang Li, Feng Han, Chaofan Ma, Wei Song, Siyuan Wang, Yibin Wang, Yi Xin, Hongjian Liu, Zhixiong Zhang, Shengyuan Ding, Tianhang Wang, Zhenglin Cheng, Tao Lin, Cheng Jin, Kaicheng Yu, Jingjing Chen, Wenjie Wang, Zhongyu Wei, Jiaqi Wang
Computer Vision
EO-VAE: Towards A Multi-sensor Tokenizer for Earth Observation Data
Nils Lehmann, Yi Wang, Zhitong Xiong et al. (+1 more)
State-of-the-art generative image and
video models rely heavily on tokenizers that compress high-dimensional inputs into more efficient latent representations. While this paradigm has revolutionized RGB generation, Earth observation (EO) data presents unique challenges due to diverse sensor specifications and variable spectral channels. We propose EO-VAE, a multi-sensor variational autoencoder designed to serve as a foundational tokenizer for the EO domain. Unlike prior approaches that train separate tokenizers for each modality, EO-VAE utilizes a single model to encode and reconstruct flexible channel combinations via dynamic hypernetworks. Our experiments on the TerraMesh
dataset demonstrate that EO-VAE achieves superior reconstruction fidelity compared to the TerraMind tokenizers, establishing a robust baseline for latent generative modeling in remote sensing.
Read the full abstract ▾
All authors Nils Lehmann, Yi Wang, Zhitong Xiong, Xiaoxiang Zhu
Computer Vision
DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation
Xu Guo, Fulong Ye, Qichao Sun et al. (+7 more)
Recent advancements in foundation models have revolutionized joint audio-
video generation. However, existing approaches typically treat human-centric tasks including reference-based audio-
video generation (R2AV),
video editing (RV2AV) and audio-driven
video animation (RA2V) as isolated objectives. Furthermore, achieving precise, disentangled control over multiple character identities and voice timbres within a single framework remains an open challenge. In this paper, we propose DreamID-Omni, a unified framework for controllable human-centric audio-
video generation. Specifically, we design a Symmetric Conditional
Diffusion Transformer that integrates heterogeneous conditioning signals via a symmetric conditional injection scheme. To resolve the pervasive identity-timbre binding failures and speaker confusion in multi-person scenarios, we introduce a Dual-Level Disentanglement strategy: Synchronized RoPE at the signal level to ensure rigid
attention-space binding, and Structured Captions at the semantic level to establish explicit attribute-subject mappings. Furthermore, we devise a Multi-Task Progressive Training scheme that leverages weakly-constrained generative priors to regularize strongly-constrained tasks, preventing overfitting and harmonizing disparate objectives. Extensive experiments demonstrate that DreamID-Omni achieves comprehensive state-of-the-art performance across
video, audio, and audio-visual consistency, even outperforming leading proprietary commercial models. We will release our code to bridge the gap between academic research and commercial-grade applications.
Read the full abstract ▾
All authors Xu Guo, Fulong Ye, Qichao Sun, Liyang Chen, Bingchuan Li, Pengze Zhang, Jiawei Liu, Songtao Zhao, Qian He, Xiangwang Hou
Computer Vision
TexSpot: 3D Texture Enhancement with Spatially-uniform Point Latent Representation
Ziteng Lu, Yushuang Wu, Chongjie Ye et al. (+7 more)
High-quality 3D texture generation remains a fundamental challenge due to the view-inconsistency inherent in current mainstream multi-view
diffusion pipelines. Existing representations either rely on UV maps, which suffer from distortion during unwrapping, or point-based methods, which tightly couple texture fidelity to
geometric density that limits high-resolution texture generation. To address these limitations, we introduce TexSpot, a
diffusion-based texture enhancement framework. At its core is Texlet, a novel 3D texture representation that merges the
geometric expressiveness of point-based 3D textures with the compactness of UV-based representation. Each Texlet latent vector encodes a local texture patch via a 2D encoder and is further aggregated using a 3D encoder to incorporate global shape context. A cascaded 3D-to-2D decoder reconstructs high-quality texture patches, enabling the Texlet space learning. Leveraging this representation, we train a
diffusion transformer conditioned on Texlets to refine and enhance textures produced by multi-view
diffusion methods. Extensive experiments demonstrate that TexSpot significantly improves visual fidelity,
geometric consistency, and robustness over existing state-of-the-art 3D texture generation and enhancement approaches. Project page: https://texlet-arch.github.io/TexSpot-page.
Read the full abstract ▾
All authors Ziteng Lu, Yushuang Wu, Chongjie Ye, Yuda Qiu, Jing Shao, Xiaoyang Guo, Jiaqing Zhou, Tianlei Hu, Kun Zhou, Xiaoguang Han
Computer Vision
FAIL: Flow Matching Adversarial Imitation Learning for Image Generation
Yeyao Ma, Chen Li, Xiaosong Zhang et al. (+2 more)
Post-training of flow matching models-aligning the output distribution with a high-quality target-is mathematically equivalent to imitation learning. While Supervised Fine-Tuning mimics expert demonstrations effectively, it cannot correct policy drift in unseen states. Preference optimization methods address this but require costly preference pairs or reward modeling. We propose Flow Matching
Adversarial Imitation Learning (FAIL), which minimizes policy-expert divergence through
adversarial training without explicit rewards or pairwise comparisons. We derive two
algorithms: FAIL-PD exploits differentiable ODE solvers for low-variance pathwise gradients, while FAIL-PG provides a black-box alternative for discrete or computationally constrained settings. Fine-tuning FLUX with only 13,000 demonstrations from Nano Banana pro, FAIL achieves competitive performance on
prompt following and aesthetic
benchmarks. Furthermore, the framework generalizes effectively to discrete image and
video generation, and functions as a robust regularizer to mitigate reward hacking in reward-based optimization. Code and data are available at https://github.com/HansPolo113/FAIL.
Read the full abstract ▾
All authors Yeyao Ma, Chen Li, Xiaosong Zhang, Han Hu, Weidi Xie
Computer Vision
PosterOmni: Generalized Artistic Poster Creation via Task Distillation and Unified Reward Feedback
Sixiang Chen, Jianyu Lai, Jialin Gao et al. (+6 more)
Image-to-poster generation is a high-demand task requiring not only local adjustments but also high-level design understanding. Models must generate text, layout, style, and visual elements while preserving semantic fidelity and aesthetic coherence. The process spans two regimes: local editing, where ID-driven generation, rescaling, filling, and extending must preserve concrete visual entities; and global creation, where layout- and style-driven tasks rely on understanding abstract design concepts. These intertwined demands make image-to-poster a multi-dimensional process coupling entity-preserving editing with concept-driven creation under image-
prompt control. To address these challenges, we propose PosterOmni, a generalized artistic poster creation framework that unlocks the potential of a base edit model for multi-task image-to-poster generation. PosterOmni integrates the two regimes, namely local editing and global creation, within a single system through an efficient data-
distillation-reward pipeline: (i) constructing multi-scenario image-to-poster
datasets covering six task types across entity-based and concept-based creation; (ii) distilling knowledge between local and global experts for supervised fine-tuning; and (iii) applying unified PosterOmni Reward Feedback to jointly align visual entity-preserving and aesthetic preference across all tasks. Additionally, we establish PosterOmni-Bench, a unified
benchmark for evaluating both local editing and global creation. Extensive experiments show that PosterOmni significantly enhances reference adherence, global composition quality, and aesthetic harmony, outperforming all open-source baselines and even surpassing several proprietary systems.
Read the full abstract ▾
All authors Sixiang Chen, Jianyu Lai, Jialin Gao, Hengyu Shi, Zhongying Liu, Tian Ye, Junfeng Luo, Xiaoming Wei, Lei Zhu
Computer Vision · Cross-listed from cs.GR
Iskra: A System for Inverse Geometry Processing
Ana Dodik, Ahmed H. Mahmoud and Justin Solomon
We propose a system for differentiating through solutions to geometry processing problems. Our system differentiates a broad class of
geometric algorithms, exploiting existing fast problem-specific schemes common to geometry processing, including local-global and ADMM solvers. It is compatible with machine learning frameworks, opening doors to new classes of inverse geometry processing applications. We marry the scatter-gather approach to mesh processing with tensor-based workflows and rely on the adjoint method applied to user-specified imperative code to generate an efficient backward pass behind the scenes. We demonstrate our approach by differentiating through mean
curvature flow, spectral conformal parameterization, geodesic distance computation, and as-rigid-as-possible deformation, examining usability and performance on these applications. Our system allows practitioners to differentiate through existing geometry processing
algorithms without needing to reformulate them, resulting in low implementation effort, fast runtimes, and lower
memory requirements than differentiable optimization tools not tailored to geometry processing.
Read the full abstract ▾
Computer Vision
AssetFormer: Modular 3D Assets Generation with Autoregressive Transformer
Lingting Zhu, Shengju Qian, Haidi Fan et al. (+6 more)
The digital industry demands high-quality, diverse modular 3D assets, especially for user-generated content~(UGC). In this work, we introduce AssetFormer, an autoregressive
Transformer-based model designed to generate modular 3D assets from textual descriptions. Our pilot study leverages real-world modular assets collected from online platforms. AssetFormer tackles the challenge of creating assets composed of primitives that adhere to constrained design parameters for various applications. By innovatively adapting module sequencing and decoding techniques inspired by
language models, our approach enhances asset generation quality through autoregressive modeling. Initial results indicate the effectiveness of AssetFormer in streamlining asset creation for professional development and UGC scenarios. This work presents a flexible framework extendable to various types of modular 3D assets, contributing to the broader field of 3D content generation. The code is available at https://github.com/Advocate99/AssetFormer.
Read the full abstract ▾
All authors Lingting Zhu, Shengju Qian, Haidi Fan, Jiayu Dong, Zhenchao Jin, Siwei Zhou, Gen Dong, Xin Wang, Lequan Yu
Computer Vision
GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learning
GigaBrain Team: Boyuan Wang, Bohan Li, Chaojun Ni et al. (+22 more)
Vision-
language-action (VLA) models that directly predict multi-step action chunks from current observations face inherent limitations due to constrained scene understanding and weak future anticipation capabilities. In contrast,
video world models pre-trained on web-scale
video corpora exhibit robust spatiotemporal
reasoning and accurate future prediction, making them a natural foundation for enhancing VLA learning. Therefore, we propose GigaBrain-0.5M*, a VLA model trained via world model-based reinforcement learning. Built upon GigaBrain-0.5, which is pre-trained on over 10,000 hours of robotic manipulation data, whose intermediate version currently ranks first on the international RoboChallenge
benchmark. GigaBrain-0.5M* further integrates world model-based reinforcement learning via RAMP (Reinforcement leArning via world Model-conditioned Policy) to enable robust cross-task adaptation. Empirical results demonstrate that RAMP achieves substantial performance gains over the RECAP baseline, yielding improvements of approximately 30% on challenging tasks including Laundry Folding, Box Packing, and Espresso Preparation. Critically, \textit{GigaBrain-0.5M$^*$} exhibits reliable long-horizon execution, consistently accomplishing complex manipulation tasks without failure as validated by real-world deployment
videos on our \href{https://gigabrain05m.github.io}{project page}.
Read the full abstract ▾
All authors GigaBrain Team: Boyuan Wang, Bohan Li, Chaojun Ni, Guan Huang, Guosheng Zhao, Hao Li, Jie Li, Jindi Lv, Jingyu Liu, Lv Feng, Mingming Yu, Peng Li, Qiuping Deng, Tianze Liu, Xinyu Zhou, Xinze Chen, Xiaofeng Wang, Yang Wang, Yifan Li, Yifei Nie, Yilong Li, Yukun Zhou, Yun Ye, Zhichao Liu, Zheng Zhu
Computer Vision · Cross-listed from cs.CL
DeepSight: An All-in-One LM Safety Toolkit
Bo Zhang, Jiaxuan Guo, Lijun Li et al. (+17 more)
As the development of Large Models (LMs) progresses rapidly, their safety is also a priority. In current Large
Language Models (LLMs) and
Multimodal Large
Language Models (MLLMs) safety workflow, evaluation, diagnosis, and
alignment are often handled by separate tools. Specifically, safety evaluation can only locate external behavioral risks but cannot figure out internal root causes. Meanwhile, safety diagnosis often drifts from concrete risk scenarios and remains at the explainable level. In this way, safety
alignment lack dedicated explanations of changes in internal mechanisms, potentially degrading general capabilities. To systematically address these issues, we propose an open-source project, namely DeepSight, to practice a new safety evaluation-diagnosis integrated paradigm. DeepSight is low-cost, reproducible, efficient, and highly scalable large-scale model safety evaluation project consisting of a evaluation toolkit DeepSafe and a diagnosis toolkit DeepScan. By unifying task and data protocols, we build a connection between the two stages and transform safety evaluation from black-box to white-box insight. Besides, DeepSight is the first open source toolkit that support the frontier AI risk evaluation and joint safety evaluation and diagnosis.
Read the full abstract ▾
All authors Bo Zhang, Jiaxuan Guo, Lijun Li, Dongrui Liu, Sujin Chen, Guanxu Chen, Zhijie Zheng, Qihao Lin, Lewen Yan, Chen Qian, Yijin Zhou, Yuyao Wu, Shaoxiong Guo, Tianyi Du, Jingyi Yang, Xuhao Hu, Ziqi Miao, Xiaoya Lu, Jing Shao, Xia Hu
Computer Vision
A DMD-Based Adaptive Modulation Method for High Dynamic Range Imaging in High-Glare Environments
Banglei Guan, Jing Tao, Liang Xu et al. (+5 more)
Background The accuracy of photomechanics measurements critically relies on image quality,particularly under extreme illumination conditions such as welding arc monitoring and polished metallic surface analysis. High dynamic range (HDR) imaging above 120 dB is essential in these contexts. Conventional CCD/CMOS sensors, with dynamic ranges typically below 70 dB, are highly susceptible to saturation under glare, resulting in irreversible loss of detail and significant errors in digital image correlation (DIC). Methods This paper presents an HDR imaging system that leverages the spatial modulation capability of a digital micromirror device (DMD). The system architecture enables autonomous regional
segmentation and adaptive exposure control for high-dynamic-range scenes through an integrated framework comprising two synergistic subsystems: a DMD-based optical modulation unit and an adaptive computational imaging pipeline. Results The system achieves a measurable dynamic range of 127 dB, effectively eliminating satu ration artifacts under high glare. Experimental results demonstrate a 78% reduction in strain error and improved DIC positioning accuracy, confirming reliable performance across extreme intensity variations. Conclusion The DMD-based system provides high fidelity adaptive HDR imaging, overcoming key limitations of conventional sensors. It exhibits strong potential for optical metrology and stress analysis in high-glare environments where traditional methods are inadequate.
Read the full abstract ▾
All authors Banglei Guan, Jing Tao, Liang Xu, Dongcai Tan, Pengju Sun, Jianbing Liu, Yang Shang, Qifeng Yu
Computer Vision
Projected Representation Conditioning for High-fidelity Novel View Synthesis
Min-Seop Kwak, Minkyung Kwon, Jinhyeok Choi et al. (+2 more)
We propose a novel framework for
diffusion-based novel view synthesis in which we leverage external representations as conditions, harnessing their
geometric and semantic correspondence properties for enhanced
geometric consistency in generated novel viewpoints. First, we provide a detailed analysis exploring the correspondence capabilities emergent in the spatial
attention of external visual representations. Building from these insights, we propose a representation-guided novel view synthesis through dedicated representation projection modules that inject external representations into the
diffusion process, a methodology named ReNoV, short for representation-guided novel view synthesis. Our experiments show that this design yields marked improvements in both reconstruction fidelity and inpainting quality, outperforming prior
diffusion-based novel-view methods on standard
benchmarks and enabling robust synthesis from sparse, unposed image collections.
Read the full abstract ▾
All authors Min-Seop Kwak, Minkyung Kwon, Jinhyeok Choi, Jiho Park, Seungryong Kim
Computer Vision
Can Local Vision-Language Models improve Activity Recognition over Vision Transformers? – Case Study on Newborn Resuscitation
Enrico Guerriero, Kjersti Engan and Øyvind Meinich-Bache
Accurate documentation of newborn resuscitation is essential for quality improvement and adherence to clinical guidelines, yet remains underutilized in practice. Previous work using 3D-CNNs and
Vision Transformers (ViT) has shown promising results in detecting key activities from newborn resuscitation
videos, but also highlighted the challenges in recognizing such fine-grained activities. This work investigates the potential of generative AI (GenAI) methods to improve activity recognition from such
videos. Specifically, we explore the use of local
vision-
language models (VLMs), combined with large
language models (LLMs), and compare them to a supervised TimeSFormer baseline. Using a simulated
dataset comprising 13.26 hours of newborn resuscitation
videos, we evaluate several zero-shot VLM-based strategies and fine-tuned VLMs with classification heads, including Low-Rank Adaptation (LoRA). Our results suggest that small (local) VLMs struggle with
hallucinations, but when fine-tuned with LoRA, the results reach F1 score at 0.91, surpassing the TimeSformer results of 0.70.
Read the full abstract ▾
Computer Vision
Spatial Chain-of-Thought: Bridging Understanding and Generation Models for Spatial Reasoning Generation
Wei Chen, Yancheng Long, Mingqiao Liu et al. (+9 more)
While
diffusion models have shown exceptional capabilities in aesthetic image synthesis, they often struggle with complex spatial understanding and
reasoning. Existing approaches resort to
Multimodal Large
Language Models (MLLMs) to enhance this capability. However, they either incur high computational costs through joint training or suffer from spatial information loss when relying solely on textual
prompts. To alleviate these limitations, we propose a Spatial Chain-of-Thought (SCoT) framework, a plug-and-play approach that effectively bridges the
reasoning capabilities of MLLMs with the generative power of
diffusion models. Specifically, we first enhance the
diffusion model's layout awareness by training it on an interleaved text-coordinate instruction format. We then leverage state-of-the-art MLLMs as planners to generate comprehensive layout plans, transferring their spatial planning capabilities directly to the generation process. Extensive experiments demonstrate that our method achieves state-of-the-art performance on image generation
benchmarks and significantly outperforms baselines on complex
reasoning tasks, while also showing strong efficacy in image editing scenarios.
Read the full abstract ▾
All authors Wei Chen, Yancheng Long, Mingqiao Liu, Haojie Ding, Yankai Yang, Hongyang Wei, Yi-Fan Zhang, Bin Wen, Fan Yang, Tingting Gao, Han Li, Long Chen
Computer Vision
Confidence-Uncertainty Boundary Calibration for Bayesian Deep Learning in Medical Image Analysis
Hua Xu, Julián D. Arias-Londoño and Juan I. Godino-Llorente
In critical decision support systems based on medical imaging, the reliability of AI-assisted decision-making is as relevant as predictive accuracy. Although deep learning models have demonstrated significant accuracy, they frequently suffer from miscalibration, manifested as overconfidence in erroneous predictions. To facilitate clinical acceptance, it is imperative that models quantify uncertainty in a manner that correlates with prediction correctness, allowing clinicians to identify unreliable outputs for further review. To address this necessity, this paper proposes a probabilistic optimization framework grounded in
Bayesian deep learning. Specifically, the Confidence-Uncertainty Boundary Curve (CUBC) is first explored as an intermediate operational target. Grounded in this target, a novel Confidence-Uncertainty Boundary Loss (CUB-Loss) is proposed to regularize the
alignment between prediction confidence and uncertainty estimates during training, imposing penalties on high-certainty errors and low-certainty correct predictions. Upon completion of training optimization, a Boundary Curve Calibration Error (BCCE) metric is further introduced to measure the degree of boundary
alignment in the calibrated model. Building on this measurement, a Dual Temperature Scaling (DTS) strategy is devised to perform post-hoc refinement, further adjusting the posterior predictive distribution across different confidence-uncertainty regions. The proposed framework is validated on three distinct medical imaging tasks: automatic screening of pneumonia, diabetic retinopathy detection, and identification of skin lesions. Empirical results demonstrate that the proposed approach improves uncertainty calibration across diverse modalities, maintains robust performance in data-scarce scenarios, and remains effective on severely imbalanced
datasets, underscoring its potential for real clinical deployment.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
UPDA: Unsupervised Progressive Domain Adaptation for No-Reference Point Cloud Quality Assessment
Bingxu Xie, Fang Zhou, Jincan Wu et al. (+3 more)
While no-reference point cloud quality assessment (NR-PCQA) approaches have achieved significant progress over the past decade, their performance often degrades substantially when a distribution gap exists between the training (source domain) and testing (target domain) data. However, to date, limited
attention has been paid to transferring NR-PCQA models across domains. To address this challenge, we propose the first unsupervised progressive domain adaptation (UPDA) framework for NR-PCQA, which introduces a two-stage coarse-to-fine
alignment paradigm to address domain shifts. At the coarse-grained stage, a discrepancy-aware coarse-grained
alignment method is designed to capture relative quality relationships between cross-domain samples through a novel quality-discrepancy-aware hybrid loss, circumventing the challenges of direct absolute feature
alignment. At the fine-grained stage, a perception fusion fine-grained
alignment approach with symmetric feature fusion is developed to identify domain-invariant features, while a conditional discriminator selectively enhances the transfer of quality-relevant features. Extensive experiments demonstrate that the proposed UPDA effectively enhances the performance of NR-PCQA methods in cross-domain scenarios, validating its practical applicability. The code is available at https://github.com/yokeno1/UPDA-main.
Read the full abstract ▾
All authors Bingxu Xie, Fang Zhou, Jincan Wu, Yonghui Liu, Weiqing Li, Zhiyong Su
Computer Vision
Benchmarking Vision-Language Models for French PDF-to-Markdown Conversion
Bruno Rigal, Victor Dupriez, Alexis Mignon et al. (+2 more)
This report evaluates PDF-to-Markdown conversion using recent
Vision-
Language Models (VLMs) on challenging French documents. Document parsing is a critical step for
Retrieval-Augmented Generation (RAG) pipelines, where transcription and layout errors propagate to downstream
retrieval and grounding. Existing
benchmarks often emphasize English or Chinese and can over-penalize benign formatting and linearization choices (e.g., line breaks, list
segmentation, alternative table renderings) that are largely irrelevant for downstream use. We introduce a French-focused
benchmark of difficult pages selected via model-disagreement sampling from a corpus of 60{,}000 documents, covering handwritten forms, complex layouts, dense tables, and graphics-rich pages. Evaluation is performed with unit-test-style checks that target concrete failure modes (text presence, reading order, and local table constraints) combined with category-specific normalization designed to discount presentation-only variance. Across 15 models, we observe substantially higher robustness for the strongest proprietary models on handwriting and forms, while several open-weights systems remain competitive on standard printed layouts.
Read the full abstract ▾
All authors Bruno Rigal, Victor Dupriez, Alexis Mignon, Ronan Le Hy, Nicolas Mery
Computer Vision
Synthesis of Late Gadolinium Enhancement Images via Implicit Neural Representations for Cardiac Scar Segmentation
Soufiane Ben Haddou, Laura Alvarez-Florez, Erik J. Bekkers et al. (+4 more)
Late gadolinium enhancement (LGE) imaging is the clinical standard for myocardial scar assessment, but limited annotated
datasets hinder the development of automated
segmentation methods. We propose a novel framework that synthesises both LGE images and their corresponding
segmentation masks using implicit neural representations (INRs) combined with denoising
diffusion models. Our approach first trains INRs to capture continuous spatial representations of LGE data and associated myocardium and fibrosis masks. These INRs are then compressed into compact latent
embeddings, preserving essential anatomical information. A
diffusion model operates on this latent space to generate new representations, which are decoded into synthetic LGE images with anatomically consistent
segmentation masks. Experiments on 133 cardiac MRI scans suggest that augmenting training data with 200 synthetic volumes contributes to improved fibrosis
segmentation performance, with the Dice score showing an increase from 0.509 to 0.524. Our approach provides an annotation-free method to help mitigate data scarcity.The code for this research is publicly available.
Read the full abstract ▾
All authors Soufiane Ben Haddou, Laura Alvarez-Florez, Erik J. Bekkers, Fleur V. Y. Tjong, Ahmad S. Amin, Connie R. Bezzina, Ivana Išgum
Computer Vision
DynaHOI: Benchmarking Hand-Object Interaction for Dynamic Target
BoCheng Hu, Zhonghan Zhao, Kaiyue Zhou et al. (+2 more)
Most existing hand motion generation
benchmarks for hand-object interaction (HOI) focus on static objects, leaving dynamic scenarios with moving targets and time-critical coordination largely untested. To address this gap, we introduce the DynaHOI-Gym, a unified online closed-loop platform with parameterized motion generators and rollout-based metrics for dynamic capture evaluation. Built on DynaHOI-Gym, we release DynaHOI-10M, a large-scale
benchmark with 10M frames and 180K hand capture trajectories, whose target motions are organized into 8 major categories and 22 fine-grained subcategories. We also provide a simple observe-before-act baseline (ObAct) that integrates short-term observations with the current frame via spatiotemporal
attention to predict actions, achieving an 8.1% improvement in location success rate.
Read the full abstract ▾
All authors BoCheng Hu, Zhonghan Zhao, Kaiyue Zhou, Hongwei Wang, Gaoang Wang
Computer Vision · Cross-listed from eess.IV
Learning Perceptual Representations for Gaming NR-VQA with Multi-Task FR Signals
Yu-Chih Chen, Michael Wang, Chieh-Dun Wen et al. (+4 more)
No-reference
video quality assessment (NR-VQA) for gaming
videos is challenging due to limited human-rated
datasets and unique content characteristics including fast motion, stylized graphics, and compression artifacts. We present MTL-VQA, a multi-task learning framework that uses full-reference (FR) quality metrics as supervisory signals to learn perceptually meaningful features without human labels during pretraining. By jointly optimizing multiple complementary proxy FR objectives with adaptive task weighting, our approach learns shared representations that transfer effectively to downstream NR-VQA. Experiments on gaming
video datasets show that MTL-VQA achieves competitive performance against state-of-the-art methods in both mean opinion score-supervised and label-efficient or self-supervised settings.
Read the full abstract ▾
All authors Yu-Chih Chen, Michael Wang, Chieh-Dun Wen, Kai-Siang Ma, Avinab Saha, Li-Heng Chen, Alan Bovik
Computer Vision · Cross-listed from cs.LG
Where Bits Matter in World Model Planning: A Paired Mixed-Bit Study for Efficient Spatial Reasoning
Suraj Ranganath, Anish Patnaik and Vaishak Menon
Efficient spatial
reasoning requires world models that remain reliable under tight precision budgets. We study whether low-bit planning behavior is determined mostly by total bitwidth or by where bits are allocated across modules. Using DINO-WM on the Wall planning task, we run a paired-goal mixed-bit evaluation across uniform, mixed, asymmetric, and layerwise variants under two planner budgets. We observe a consistent three-regime pattern: 8-bit and 6-bit settings remain close to FP16, 3-bit settings collapse, and 4-bit settings are allocation-sensitive. In that transition region, preserving encoder precision improves planning relative to uniform quantization, and near-size asymmetric variants show the same encoder-side direction. In a later strict 22-
cell replication with smaller per-
cell episode count, the mixed-versus-uniform INT4 sign becomes budget-conditioned, which further highlights the sensitivity of this transition regime. These findings motivate module-aware, budget-aware quantization policies as a broader research direction for efficient spatial
reasoning. Code and run artifacts are available at https://github.com/suraj-ranganath/DINO-MBQuant.
Read the full abstract ▾
Computer Vision
SynthRAR: Ring Artifacts Reduction in CT with Unrolled Network and Synthetic Data Training
Hongxu Yang, Levente Lippenszky, Edina Timko et al. (+1 more)
Defective and inconsistent responses in CT detectors can cause ring and streak artifacts in the reconstructed images, making them unusable for clinical purposes. In recent years, several ring artifact reduction solutions have been proposed in the image domain or in the sinogram domain using supervised deep learning methods. However, these methods require dedicated
datasets for training, leading to a high data collection cost. Furthermore, existing approaches focus exclusively on either image-space or sinogram-space correction, neglecting the intrinsic correlations from the forward operation of the CT geometry. Based on the theoretical analysis of non-ideal CT detector responses, the RAR problem is reformulated as an inverse problem by using an unrolled network, which considers non-ideal response together with linear forward-projection with CT geometry. Additionally, the intrinsic correlations of ring artifacts between the sinogram and image domains are leveraged through synthetic data derived from natural images, enabling the trained model to correct artifacts without requiring real-world clinical data. Extensive evaluations on diverse scanning geometries and anatomical regions demonstrate that the model trained on synthetic data consistently outperforms existing state-of-the-art methods.
Read the full abstract ▾
All authors Hongxu Yang, Levente Lippenszky, Edina Timko, Gopal Avinash
Computer Vision
DiffPlace: Street View Generation via Place-Controllable Diffusion Model Enhancing Place Recognition
Ji Li, Zhiwei Li, Shihao Li et al. (+3 more)
Generative models have advanced significantly in realistic image synthesis, with
diffusion models excelling in quality and stability. Recent multi-view
diffusion models improve 3D-aware street view generation, but they struggle to produce place-aware and background-consistent urban scenes from text, BEV maps, and object bounding boxes. This limits their effectiveness in generating realistic samples for place recognition tasks. To address these challenges, we propose DiffPlace, a novel framework that introduces a place-ID controller to enable place-controllable multi-view image generation. The place-ID controller employs linear projection, perceiver
transformer, and contrastive learning to map place-ID
embeddings into a fixed CLIP space, allowing the model to synthesize images with consistent background buildings while flexibly modifying foreground objects and weather conditions. Extensive experiments, including quantitative comparisons and augmented training evaluations, demonstrate that DiffPlace outperforms existing methods in both generation quality and training support for visual place recognition. Our results highlight the potential of generative models in enhancing scene-level and place-aware synthesis, providing a valuable approach for improving place recognition in autonomous driving
Read the full abstract ▾
All authors Ji Li, Zhiwei Li, Shihao Li, Zhenjiang Yu, Boyang Wang, Haiou Liu