Computer Vision
Colored Noise Diffusion Sampling
Hadar Davidson, Noam Issachar and Sagie Benaim
Diffusion models achieve state-of-the-art image synthesis, with their generative trajectories fundamentally exhibiting a spectral bias, resolving low-frequency global structures early and high-frequency fine details later. Conventional
stochastic differential equation (SDE) solvers fail to account for this dynamic, naively injecting uniform white noise throughout the entire process and misusing the finite energy budget. In this work, we establish a mathematical framework that reconsiders SDE inference as a targeted, frequency-decoupled energy transfer. Leveraging this framework, we introduce Colored Noise Sampling (CNS), a novel, training-free
stochastic solver. Rather than injecting uniform white noise, CNS utilizes a dynamic, timestep- and frequency-dependent schedule that more efficiently allocates injected energy toward structurally unresolved frequency bands. By actively exploiting the model's inherent spectral bias, CNS systematically steers the generated distribution toward the true data
manifold. Extensive experiments demonstrate that CNS significantly outperforms standard ODE and SDE baselines as a strictly plug-and-play, inference-time sampler substitution across diverse architectures (SiT, JiT, FLUX). Compared to standard sampling on ImageNet-256, CNS achieves substantial unguided FID reductions, improving from 8.26 to 6.27 on SiT-XL/2, 32.39 to 26.69 on JiT-B/16, and 11.88 to 8.31 on JiT-H/16, while yielding consistent relative FID improvements with Classifier-Free Guidance. Project page is available at https://hadardavidson.github.io/CNS/.
Read the full abstract ▾
Computer Vision
Supercharging Thermal Gaussian Splatting with Depth Estimation
Manoj Biswanath, Chenxin Cai, Hannah Schieber et al. (+2 more)
Efficient and robust 3D scene representation is crucial in autonomous driving, robotics, and related fields. While RGB images provide valuable content for 3D reconstruction, other modalities like thermal or depth can enable additional information on the environment. Lately, novel view synthesis methods like 3D Gaussian Splatting have started using multiple modalities to further boost their performance. But fusing or combining
multimodal data can make the process slower and can bring in additional challenges. Therefore, our project aims to use single modality based on thermal infrared domain, by removing the reliance on visible light as much as possible. This single modality can be expected to be faster as it does not rely on
multimodal data. We propose a method, Thermal-to-Depth Gaussian Splatting (TDg), that uses only thermal images and depth estimation in its architecture to derive the radiance fields. Our TDg method outperforms the MSMG (Multiple Single-Modal Gaussians) baseline in most cases on our test
datasets, RGBT-Scenes and ThermalMix. On average, the rendering quality metrics such as learned perceptual image patch similarity (LPIPS), structural similarity index measure (SSIM), and peak signal-to-noise ratio (PSNR) of TDg are 1.12%, 0.034%, and 0.01% better than the baseline MSMG values. It also reduces the training time significantly, by 12 mins 47 secs (55% improvement). Overall, our method is successful in deriving these thermal radiance fields, which can ultimately have several applications, such as identifying heat sources critical in surveillance, search or rescue operations, and industrial inspections where temperature is widely used to monitor machines.
Read the full abstract ▾
All authors Manoj Biswanath, Chenxin Cai, Hannah Schieber, Daniel Roth, Benjamin Busam
Computer Vision
Shihao Han, Hao Yang, Xinting Hu et al. (+3 more)
Scaling
Diffusion Transformers to generate high-resolution, long
videos is constrained by the quadratic cost of self-
attention, and existing sparse
attention methods degrade under high sparsity. We show empirically that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full
attention. Based on this insight, we propose Veda, a distilled sparse
attention framework that formulates tile selection as an explicit reconstruction problem from full
attention. Veda integrates statistics-aware tile scoring with head-aware tiling to reduce estimation error and structural mismatch, enabling aggressive sparsity. A hardware-efficient tile-skipping kernel converts theoretical sparsity into practical wall-clock speedups. Experiments on large
video diffusion models, including Waver and Wan2.1, demonstrate substantial acceleration with no noticeable degradation in generation quality. To generate 720P 10-second
videos on Waver-T2V-12B, Veda achieves a 5.1$\times$ end-to-end speedup and a 10.5$\times$ self-
attention speedup, reducing
attention overhead from 92% to 50%. Notably, the gains increase with sequence length, indicating that Veda scales favorably with spatiotemporal resolution across models.
Read the full abstract ▾
All authors Shihao Han, Hao Yang, Xinting Hu, Xiaofeng Mei, Yi Jiang, Xiaojuan Qi
Computer Vision
MonoPhysics: Estimating Geometry, Appearance, and Physical Parameters from Monocular Videos
Daniel Rho, Jun Myeong Choi, Matthew Thornton et al. (+2 more)
Existing inverse physics methods recover physical parameters from multi-view
videos, where
geometric constraints across views resolve scale and 3D structure. In
monocular settings, however, such constraints are absent, leading to severe scale ambiguity, inaccurate geometry, and weak coupling between appearance optimization and physical simulation. To address these challenges, we propose MonoPhysics, a framework for
monocular inverse physics estimation of deformable objects that jointly optimizes geometry, appearance, and physical parameters using a differentiable simulator and 3D Gaussian Splatting. Our key contribution is removing the multi-view capture requirement of existing methods, a necessary step toward handling in-the-wild
video. MonoPhysics introduces three visual-physical bridges: scene re-parameterization, physics-aware geometry refinement, and a differentiable position map. We evaluate on Vid2Sim, real-world captures, and a new
dataset of elastic and plasticine objects that we introduce. MonoPhysics outperforms
monocular baselines in future prediction and recovers Young's modulus on Vid2Sim with accuracy comparable to a multi-view baseline. Code and data are available at https://daniel03c1.github.io/MonoPhysics/.
Read the full abstract ▾
All authors Daniel Rho, Jun Myeong Choi, Matthew Thornton, Biswadip Dey, Roni Sengupta
Computer Vision · Cross-listed from cs.GR
Before the Shutter: Aesthetic and Actionable Portrait Photography Planning in 3D Scenes
Ruixiang Jiang and Chang Wen Chen
Portrait photography is largely decided before the shutter opens: the subject's pose, the camera configuration, and the lighting devices must be coordinated within the surrounding 3D scene. In contrast, most existing computational methods focus on post-production in 2D image space, such as retouching, relighting, or editing images that already exist; pre-capture photographic planning remains largely unexplored. We introduce 3D aesthetic portrait planning, the task of generating human pose, camera, lighting, and exposure plans that produce visually compelling portraits while satisfying
geometric and photometric feasibility in a 3D scene. Our approach builds a Photographic Scene Graph that represents scene affordances, subject-scene relations, and portrait-relevant lighting structure. Built on this representation, we perform aesthetic-guided comparative planning over previous attempts and current viewfinder observations. Experiments across diverse indoor and outdoor scenes show that our method produces portraits preferred by human raters and MLLM evaluators over competitive baselines, while maintaining high physical plausibility. Together, our results suggest a path from post-capture correction toward pre-capture computational portrait planning. Project repository: https://github.com/songrise/Before-the-Shutter
Read the full abstract ▾
Computer Vision
VPG: Visual Prefix Guidance for Autoregressive Image and Video Generation
Xinyao Liao, Qiyuan He, Yicong Li et al. (+4 more)
Autoregressive image and
video generators are trained with teacher-forced histories but must sample from their own generated prefixes at inference time, making them vulnerable to exposure bias and prefix drift. Existing remedies either modify training or apply sampling-time guidance aimed primarily at external semantic conditions, such as class labels or text
prompts, rather than testing whether a next-step prediction provides strong posterior support for the generated prefix itself. We propose Visual Prefix Guidance (VPG), a training-free inference-time guidance method for autoregressive image and
video generation. VPG improves next-step prediction by contrasting the model's output under the generated prefix with its output under a corrupted prefix, then extrapolating logits toward candidates that strengthen the posterior support of the generated prefix. Across class-conditional image generation with VAR, text-to-image generation with Infinity, and text-to-
video generation with InfinityStar, VPG improves generation quality without retraining the base model, reducing FID on VAR by 0.36 on average and improving
benchmark performance on both image and
video generation.
Read the full abstract ▾
All authors Xinyao Liao, Qiyuan He, Yicong Li, Jiayin Zhu, Xiaoye Qu, Wei Wei, Angela Yao
Computer Vision
Archon: A Unified Multimodal Model for Holistic Digital Human Generation
Chong Bao, Shichen Liu, Lijun Yu et al. (+9 more)
Digital humans are fundamental to immersive interaction, yet creating a unified model for holistic modalities, including text, audio, motion, and visual content, remains an open challenge. In this paper, we present Archon, a fully pretrained, human-centric unified
multimodal model for holistic avatar generation. Archon unifies seven modalities with modality-specific tokenizers, and a native autoregressive unified
multimodal model pretrained on synchronized modalities and 72 diverse tasks to model holistic joint distributions. To address the
token explosion challenge in high-fidelity talking
videos, we introduce a
memory-efficient semantic
video reparameterization, achieving 4x
token reduction while preserving fine-grained dynamics, coupled with a semantic-driven
video diffusion decoder. We further propose a "Thinking in Modality" that decomposes ambiguous cross-modal tasks into stepwise thinking in an alternative chain of modality, progressively enhancing fidelity and controllability. Extensive experiments demonstrate that Archon achieves superior or comparable performance across diverse digital human generation tasks, validating the effectiveness of our unified framework. Project page: https://zju3dv.github.io/archon/.
Read the full abstract ▾
All authors Chong Bao, Shichen Liu, Lijun Yu, David Futschik, Stylianos Moschoglou, Shefali Srivastava, Ziqian Bai, Feitong Tan, Guofeng Zhang, Zhaopeng Cui, Sean Fanello, Yinda Zhang
Computer Vision
City-Mesh3R: Simulation-Ready City-Scale 3D Mesh Reconstruction from Multi-View Images
Sayan Paul, Sourav Ghosh, Siddharth Katageri et al. (+3 more)
City-scale 3D surface reconstruction from multiview images for downstream 3D simulation, poses highly challenging problems due to the scale and complexity of urban scenes. Existing city-scale 3D reconstruction methods based on NeRF, Gaussian Splatting etc. often fail to recover 3D meshes ready for simulation due to incomplete/missing geometry and irregular, noisy surfaces. Scaling existing small-scale 3D reconstruction methods to arbitrarily large urban scenes is highly infeasible due to their computational complexity. We present City-Mesh3R, a scalable framework for reconstructing watertight surface meshes directly from large unordered image collections. Unlike recent methods which use global sparse SfM point-cloud initialization followed by a distributed 3D dense reconstruction of large-scale scenes, our method follows an end-to-end images-to-mesh 3D reconstruction approach using a divide-and-conquer strategy. The sparse city map is reconstructed via topological image clustering, cluster-wise independent sparse SfM and map merging, without need for exhaustive image feature matching. Then this map is partitioned spatially to perform geometry-aware camera selection, followed by dense surface reconstruction and surface refinement using
curvature-aware adaptive vertex density remeshing. These partition meshes are then stitched together to produce the global mesh of the city. The proposed end-to-end framework is evaluated on city-scale reconstruction
datasets. As demonstrated by our qualitative and quantitative results, our proposed method yields high-fidelity watertight 3D meshes with regular geometry, capturing fine surface details, and is suitable for scaling to arbitrarily large scenes owing to the end-to-end processing in a distributed setting.
Read the full abstract ▾
All authors Sayan Paul, Sourav Ghosh, Siddharth Katageri, Soumyadip Maity, Sanjana Sinha, Brojeshwar Bhowmick
Computer Vision
Grounded 3D-Aware Spatial Vision-Language Modeling
An-Chieh Cheng, Yang Fu, Yatai Ji et al. (+12 more)
We present GR3D, a spatial
vision language model equipped with three complementary grounding capabilities–explicit 2D grounding, implicit 2D grounding, and
monocular 3D grounding–within a single framework. GR3D introduces an implicit grounding mechanism that identifies entity mentions during generation and inserts the corresponding region
tokens into the text stream, allowing the model to reference visual evidence on the fly when producing spatial chain-of-thought responses. In parallel, a region-prompted
monocular 3D grounding design predicts 3D bounding boxes in the camera view from grounded region queries, supported by intrinsic-aware normalization and dense
geometric supervision. Together, these grounding capabilities enable GR3D to decompose complex spatial understanding problems into grounded 2D perception followed by 3D inference. GR3D achieves consistent improvements across grounded and non-grounded spatial
benchmarks, demonstrating grounding as an effective inductive bias for strengthening spatial understanding in VLMs. These grounding capabilities collectively enhance general spatial understanding beyond the grounding task itself.
Read the full abstract ▾
All authors An-Chieh Cheng, Yang Fu, Yatai Ji, Ligeng Zhu, Guanqi Zhan, Zhuoyang Zhang, Zhaojing Yang, Song Han, Yao Lu, Pavlo Molchanov, Vidya Nariyambut Murali, Jan Kautz, Xiaolong Wang, Hongxu Yin, Sifei Liu
Computer Vision
Boosting Image Quality Assessment Performance: Unsupervised Score Fusion by Deep Maximum a Posteriori Estimation
Zhongling Wang, Raymond Zhou, Shahrukh Athar et al. (+2 more)
Over the past decades, numerous Image Quality Assessment (IQA) models have emerged, aiming to predict the perceptual quality of images. However, individual models are often biased toward certain types of image content or distortions, depending on the design principle and process. An intuitive idea is to harness the strengths and mitigate the weaknesses of each IQA model, by fusing the scores of multiple models into a stronger one. Here we make one of the first attempts to seek an optimal solution for the idea and propose a general framework for unsupervised IQA score fusion using deep Maximum a Posteriori (MAP) estimation. The proposed model conducts fine-grained uncertainty estimation at the score level to increase the accuracy and reduce the uncertainty in fused predictions. Comprehensive experiments demonstrate the superiority of the proposed model over individual IQA models and other fusion methods. It also exhibits an interesting capability of rejecting “bad" models in the fusion process.
Read the full abstract ▾
All authors Zhongling Wang, Raymond Zhou, Shahrukh Athar, Wenbo Yang, Zhou Wang
Computer Vision
PhyGenHOI: Physically-Aware 4D Generation of Dynamic Human-Object Interactions
Omer Benishu, Gal Fiebelman and Sagie Benaim
We address the task of generating physically accurate and visually faithful 4D Human-Object Interaction (HOI). Given a static 3D human and target object represented as 3D Gaussian Splats (3DGS), our goal is to synthesize dynamic scenes where the human actively engages with the object through actions, such as punching or kicking, in accordance with a given input text. To this end, we introduce PhyGenHOI, a novel framework that couples generative human motion with an explicit physical object simulation. We model the human as a semantic
agent driven by a Motion
Diffusion Model (MDM) and the object as a physical
agent simulated via the Material Point Method (MPM), utilizing 3D Gaussians as a unified, differentiable representation. We supervise their interaction through three coupled mechanisms: (1) A Windowed Attraction Loss that temporally synchronizes generative motion to intercept the object; (2) A Contact-Driven Re-simulation step that triggers physically consistent momentum transfer upon impact; and (3) A Masked
Video-SDS objective that injects
video-based priors to enhance contact fidelity. Experiments show PhyGenHOI generates physically consistent 4D HOI across diverse actions, humans, and objects, outperforming baselines. Project page and
videos: https://omerbenishu.github.io/PhyGenHOI/
Read the full abstract ▾
Computer Vision
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
Feng Han, Zhixiong Zhang, Zheming Liang et al. (+2 more)
Vision-
Language Models (VLMs) have achieved substantial progress across a wide range of understanding and
reasoning tasks, driven by large-scale image-text training aimed at
multimodal fusion. Ideally, replacing a textual question with its rendered-image counterpart should leave model performance essentially unaffected. In practice, however, such modality substitution induces dramatic performance degradation. We attribute this "carrier sensitivity" issue to an inherent bias in current training corpora. Across prevalent
datasets such as image captioning, VQA, OCR, and web-sourced interleaved data, text and images are typically organized into distinct and asymmetric roles, with text serving as linguistic queries and images as visual references. Such data bias leads VLMs to exhibit distinct preferences for information acquisition across different modalities. Consequently, VLMs fail to align representations of semantically equivalent content across textual and visual carriers, making model
reasoning fragile under modality substitution. To address this, we propose Local Modality Substitution (LoMo), a lightweight, architecture-agnostic data curation paradigm designed to provide supervision for cross-modal representational invariance between semantically equivalent text and image carriers. LoMo achieves this by reformulating single-modality
prompts into seamlessly interleaved
multimodal sequences. It dynamically selects target text spans and recasts them as rendered images, thereby preserving the same semantics across "text, visual, text" carriers. Extensive experiments across 13 diverse
multimodal benchmarks demonstrate that LoMo significantly improves overall
multimodal reasoning and yields deeper cross-modal fusion. Specifically, it delivers consistent gains across foundational models, improving over standard SFT by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B.
Read the full abstract ▾
All authors Feng Han, Zhixiong Zhang, Zheming Liang, Yibin Wang, Jiaqi Wang
Computer Vision
minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models
Min Zhao, Hongzhou Zhu, Bokai Yan et al. (+13 more)
Recent
video diffusion foundation models have achieved remarkable progress in high-quality
video generation, yet turning them into real-time interactive
video world models remains challenging. Interactive world models require controllable, causal, and low-latency rollout, which in practice demands a full pipeline spanning data construction, controllable fine-tuning, autoregressive training, few-step
distillation, and streaming inference. In this work, we present minWM, a full-stack open-source framework for building real-time interactive
video world models. minWM provides an end-to-end pipeline that converts existing bidirectional T2V/TI2V
video foundation models into camera-controllable few-step autoregressive world models. Specifically, minWM first fine-tunes a bidirectional
video diffusion model with camera control, and then applies the Causal Forcing / Causal Forcing++ pipeline, including AR
diffusion training, causal ODE or causal consistency
distillation, and asymmetric DMD, to distill it into a few-step autoregressive generator for low-latency rollout. The framework is modular and architecture-extensible: we instantiate it on representative open backbones, including Wan2.1-T2V-1.3B and HY1.5-TI2V-8B, covering both cross-
attention-based condition injection and MMDiT-style architectures. minWM also supports adapting existing
video world models, such as HY-WorldPlay, to new data distributions, training recipes, and latency targets. Beyond releasing runnable scripts, checkpoints, documentation, and inference code, we provide practical ablations on camera trajectory quality, controllability training steps, and minimal batch-size requirements. We hope minWM serves as a reproducible and extensible recipe for building and adapting real-time interactive
video world models. Project Page: [https://github.com/shengshu-ai/minWM](https://github.com/shengshu-ai/minWM)
Read the full abstract ▾
All authors Min Zhao, Hongzhou Zhu, Bokai Yan, Zihan Zhou, Yimin Chen, Honglie Wang, Wenqiang Sun, Zhengwei Fang, Zizheng Xun, Zihao Li, Kaiwen Zheng, Guande He, Xiao Yang, Chongxuan Li, Fan Bao, Jun Zhu
Computer Vision · Cross-listed from cs.CL
How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
Ziwen Xu, Haiwen Hong, Linsong Yu et al. (+4 more)
Large
Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such
memory updates, existing studies mainly rely on qualitative downstream evaluations, leaving the quantitative capacity limits and underlying dynamics of exact parametric
memory largely unexplored. To bridge this gap, we employ LoRA as a controlled
memory capacity probe within the latent space to systematically quantify exact parametric
memory. We introduce the Parametric
Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length. At the
token level, fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for verbatim recall under greedy decoding. Driven by these insights, we introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold
tokens. Empirical evaluations demonstrate that MemFT can enhance
memory fidelity and efficiency. Code will be released at https://github.com/zjunlp/ParametricMemoryLaw.
Read the full abstract ▾
All authors Ziwen Xu, Haiwen Hong, Linsong Yu, Benglei Cui, Longtao Huang, Hui Xue, Ningyu Zhang
Computer Vision
Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning
Ciara Rowles, Reshinth Adithyan, Nikhil Pinnaparaju et al. (+2 more)
We present Stable-Layers, a reinforcement learning framework that eliminates the need for paired supervision by fine-tuning a pretrained layer decomposition model using only feedback from a
vision-
language model (VLM). Starting from Qwen-Image-Layered, we apply Flow-GRPO with LoRA adaptation, sampling multiple candidate decompositions per image, scoring them with a VLM, and optimising the policy from group-relative advantages. The key challenge lies in designing a reliable reward signal: VLMs scoring samples in isolation tend to compress their judgements into a narrow band, leaving GRPO with little within-group variance to learn from. We address this with a two-stage evaluation pipeline that pairs structured per-sample scoring across five edit-centric criteria with a grid-based calibration step in which the VLM re-scores all candidates side-by-side. Stable-Layers produces decompositions with stronger layer separation, fewer blank or artifact-heavy layers, and lower per-layer reconstruction error on the Crello
dataset compared to the base model.
Read the full abstract ▾
All authors Ciara Rowles, Reshinth Adithyan, Nikhil Pinnaparaju, Vikram Voleti, Mark Boss
Computer Vision
VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents
Amrita Mazumdar, Seonwook Park, Rajarshi Roy et al. (+6 more)
Natural human conversation is full-duplex and audio-visual: people simultaneously speak and listen while continuously interpreting and producing nonverbal cues, such as nods, smiles, and gestures. To support successful human-
agent interaction,
agents must model full-duplex audiovisual conversation; however, existing full-duplex
benchmarks evaluate only
speech. In this work, we present VideoFDB, the first
benchmark to evaluate full-duplex audio-visual-to-audio-visual (AV2AV) conversational
agents. VideoFDB contributes (i) 237 dyadic clips spanning 11 nonverbal conversational dynamics from real-world
video calls, (ii) a taxonomy separating perception from generation behaviors, and (iii) a rubric-based LM-as-judge evaluation framework with interpretable axes for assessing conversational quality with respect to nonverbal conversational dynamics. Across open- and closed-source
vision-
speech agents, we find systematic failure modes: captioning collapse and visual-stream ignorance, and we show that current systems exploit
vision for explicit visual question answering but not for the streaming joint audiovisual grounding required in natural conversation. We further evaluate cascaded
speech-to-avatar systems and find that their architecture fundamentally precludes the production of full-duplex nonverbal cues. As the first
benchmark for full-duplex AV2AV interaction, VideoFDB establishes a foundation for systematic evaluation and, we hope, will accelerate the advancement and development of next-generation
multimodal conversational
agents.
Read the full abstract ▾
All authors Amrita Mazumdar, Seonwook Park, Rajarshi Roy, Nikhil Srihari, Shengze Wang, Yuhao Zhou, Julia Wang, Koki Nagano, Shalini De Mello
Computer Vision
Ambient-robust Inverse Rendering using Active RGB-NIR Imaging
Hoon-Gyu Chung, Jinnyeong Kim, Hyunwoo Kang et al. (+1 more)
Inverse rendering aims to reconstruct geometry and reflectance of objects from images. Despite recent progress, existing methods often produces inaccurate reconstructions that are sensitive to ambient illumination conditions. Here we introduce an ambient-robust inverse rendering method enabled by active RGB-NIR imaging. Our key insight is to leverage near-infrared (NIR) flash illumination-imperceptible to human observers-to obtain stable point-light shading that is largely invariant to ambient illumination. By using multi-view RGB images illuminated by ambient light and NIR images acquired with active NIR flash illumination, we reconstruct accurate geometry and reflectance by exploiting the complementary benefits of RGB and NIR images via a three-stage inverse rendering method. To enable dense multi-view acquisition, we develop an active imaging system equipped with a RGB-NIR camera and a NIR flash mounted on a mobile base. Using this system, we collect the first multi-view RGB-NIR inverse rendering
dataset captured under multiple ambient illumination conditions. Experiments demonstrate that our method outperforms prior approaches, achieving accurate geometry and reflectance estimation across multiple ambient lighting scenarios.
Read the full abstract ▾
All authors Hoon-Gyu Chung, Jinnyeong Kim, Hyunwoo Kang, Seung-Hwan Baek
Computer Vision
GenClaw: Code-Driven Agentic Image Generation
Junyan Ye, Jun He, Zilong Huang et al. (+4 more)
Image generation models have evolved from text-conditioned pixel synthesis toward
multimodal agents endowed with visual comprehension and tool invocation capabilities. Yet, existing
agents remain at the mercy of underlying black-box image models. Their workflow is trapped in a repetitive cycle of
prompt rewriting for generation refinement, leaving them with no mechanism to directly manipulate the canvas. In essence, the potential of LLMs to serve as a genuine "brush" for precise visual construction remains largely untapped. In this paper, we propose GenClaw, a code-driven agentic image generation paradigm that empowers the
agent to create like a human artist: first conceptualizing, then sketching, and finally coloring. Specifically, the
agent first constructs the conceptual knowledge and context through search and
reasoning. It then utilizes code (e.g., SVG, HTML, ThreeJS) to render executable visual sketches. Finally, it employs an image generation model to supplement textures, materials, and photorealism. In this workflow, code serves as a controllable intermediate canvas bridging linguistic
reasoning and pixel synthesis, seamlessly integrating programmatic logic with the visual expressiveness of generative models. By transforming image generation from a black-box paradigm into a staged process akin to authentic human creation, GenClaw offers a step toward for highly controllable and interpretable visual generation systems.
Read the full abstract ▾
All authors Junyan Ye, Jun He, Zilong Huang, Dongzhi Jiang, Xuan Yang, Rui Chen, Weijia Li
Computer Vision
Reinforcement Learning with Robust Rubric Rewards
Ya-Qi Yu, Hao Wang, Fangyu Hong et al. (+15 more)
While Reinforcement Learning with Verifiable Rewards (RLVR) is effective for deterministically checkable tasks, many
vision-
language tasks are partially verifiable, demanding multi-criteria supervision (e.g., perceptual details,
reasoning steps, and constraints). Rubrics provide a natural interface for this fine-grained supervision, but their effectiveness depends on the execution accuracy during online RL. We propose Reinforcement Learning with Robust Rubric Rewards ($\text{RLR}^3$), extending RLVR from task-level verification to criterion-level verification. $\text{RLR}^3$ routes instance-specific rubrics through two execution paths: an LLM-as-an-extractor paired with a deterministic verifier, or an LLM-as-a-Judge for non-verifiable criteria. To ensure faithful scoring, $\text{RLR}^3$ introduce a minimal exposure strategy that masks ground truths from extractors and images from judges. Furthermore, $\text{RLR}^3$ employs hierarchical aggregation to prioritize essential criteria over additional criteria, and mitigates score saturation within rollout groups. Evaluated on Qwen3-VL-30B-A3B across 15
benchmarks, $\text{RLR}^3$ consistently outperforms RLVR, yielding a 4.7-point improvement over the base model and exceeding the official instruct-to-thinking model gap. Controlled audits confirm our deterministic verification and minimal exposure significantly reduce exploitable false positives.
Read the full abstract ▾
All authors Ya-Qi Yu, Hao Wang, Fangyu Hong, Xiangyang Qu, Gaojie Wu, Qiaoyu Luo, Nuo Xu, Huixin Wang, Wuheng Xu, Yongxin Liao, Zihao Chen, Haonan Li, Ziming Li, Dezhi Peng, Minghui Liao, Jihao Wu, Haoyu Ren, Dandan Tu
Computer Vision
CA-World: Multi-Object Counterfactual Alignment for Efficient Interactive-Ready Reconstruction
Xin Dong, Weijian Deng, Lihan Zhang et al. (+3 more)
Reconstructing interaction-ready 3D worlds is essential for physical simulation, virtual reality, robotics, and autonomous driving. However, existing methods mainly optimize static and holistic visual fidelity, with limited support for multi-object interaction. We argue that an interaction-ready reconstruction should anticipate potential scene changes and preserve
geometric completeness, visual quality, multi-object spatial relationship, and physical plausibility under potential interactions. To this end, motivated by the causal intervention, we propose CA-World, an efficient framework that integrates counterfactual
alignment learning into a decoupling-reintegration reconstruction pipeline. Specifically, we formulate foreground-background decoupling as a visual intervention, separate object generation and background inpainting as counterfactual generation, and scene reintegration as an inverse intervention. According to counterfactual consistency, reversing the intervention should recover the factual world, motivating three
alignment objectives between the reintegrated and original scenes: appearance, spatial, and physical consistency. This formulates interaction-ready reconstruction as counterfactual
alignment learning with direct supervision. Moreover, leveraging the locality of object-level interventions, CA-World constrains counterfactual states using the observed scene, enabling efficient and coherent reintegration without jointly optimizing all object states, thereby reducing computational cost and error accumulation. Experiments on object completeness, spatial accuracy, outdoor background completion, rendering quality, simulated dynamics, and downstream applications demonstrate the effectiveness of CA-World. Project page: https://chnxindong.github.io/ca-world/.
Read the full abstract ▾
All authors Xin Dong, Weijian Deng, Lihan Zhang, Tianru Dai, Wenfeng Deng, Yansong Tang
Computer Vision
BullingerDB: A Dataset for Handwritten Text Recognition and Writer Retrieval
Marco Peer, Anna-Scius Bertrand, Patricia Scheurer et al. (+1 more)
We present BullingerDB, a large-scale
benchmark dataset for historical document analysis based on the correspondence of Heinrich Bullinger (1504-1575). The corpus comprises 20,898 pages and 499,222 text lines written by 796 writers over six decades, featuring stylistic variation,
multilingual content (mostly Latin and Early New High German) as well as meta-information such as writer identity and time. We evaluate BullingerDB on text recognition and writer
retrieval. TrOCR, the best performing model, achieves a CER of 9.1%. For writer
retrieval, we introduce a temporal nDCG metric to assess time-aware
retrieval. While temporally coherent
retrieval is achievable, mAP (78.3%) scores indicate challenges due to long-term stylistic variation. With BullingerDB, we aim to establish a new
benchmark for
multilingual historical text recognition and temporally-aware writer analysis.
Read the full abstract ▾
All authors Marco Peer, Anna-Scius Bertrand, Patricia Scheurer, Andreas Fischer
Computer Vision
Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
Chun-Hsiao Yeh, Shengyi Qian, Manchen Wang et al. (+3 more)
Vision-
Language Models (VLMs) often struggle with robust 3D spatial
reasoning. Prevailing methods that rely on fine-tuning with 3D visual question-answering (VQA)
datasets may overfit
dataset-specific biases, while integrating specialized 3D visual encoders is often inflexible and cumbersome. In this paper, we argue that genuine spatial understanding should emerge from learning fundamental
geometric priors, not only from high-level VQA supervision. We propose GASP (
Geometric-Aware Spatial Priors), a framework that injects these priors directly into the LLM's
transformer layers. GASP employs a small correspondence head, applied as a deep supervision signal across all layers, and is trained with a dual objective leveraging ground-truth geometry from large-scale
video scenes: a contrastive loss on ground-truth point correspondences enforces 2D view-invariance, while a depth consistency supervision resolves 3D
geometric ambiguities. Our analysis first provides a diagnostic showing that standard VLMs' internal correspondence matching accuracy is very low (often below 5%). We then demonstrate that our training substantially improves this behavior, boosting peak layer-wise correspondence to over 70% and maintaining over 85% temporal robustness while baselines remain below 5%. These internal improvements translate to significant gains on downstream spatial
benchmarks including +18.2% on All-Angles Bench and +29.0% on VSI-Bench, all without training on any 3D VQA data. Our findings indicate that learning from fundamental
geometric priors is a promising and generalizable pathway towards VLMs with more reliable 3D spatial
reasoning.
Read the full abstract ▾
All authors Chun-Hsiao Yeh, Shengyi Qian, Manchen Wang, Yi Ma, Joseph Tighe, Fanyi Xiao
Computer Vision
IP-Adapter Is All You Need: Towards Fine-Tuning-Free Diffusion-Based Talking Face Generation
Hao Wu, Xiangyang Luo, Hao Wang et al. (+3 more)
With the rapid advancement of
diffusion models, talking face generation has made remarkable progress. However, existing
diffusion-based methods still require task-specific fine-tuning and large-scale audiovisual
datasets, resulting in high computational costs that hinder scalability and accessibility of
diffusion-based approaches across the research community. To address this, we propose a finetuning-free paradigm that directly performs talking face generation using the pretrained weights of Stable
Diffusion and IP-Adapter. This backbone leverages the visual
embedding capability of IP-Adapter to mine lip-related semantics from the pretrained Stable
Diffusion. To address the challenges of identity drift, synchronization errors, and temporal instability, we also design three trainable-parameterfree components: (1) the Structurist, which explicitly disentangles and reassembles lip and appearance features to mitigate identity drift and appearance distortion; (2) the Structure Controller, which adaptively refines
embeddings based on quasi-monotonic motion trends for precise lip synchronization; and (3) the Noise Sensor, which introduces Gaussian prior to detect and suppress flicker and jitter artifacts and enhance temporal consistency. Experimental results show that our method outperforms existing SOTA approaches in both lip-sync accuracy (at least 0.16 gain in PCLD) and visual fidelity (at least 0.7 improvement in FID), establishing a novel fine-tuning-free
diffusion framework for talking face generation.
Read the full abstract ▾
All authors Hao Wu, Xiangyang Luo, Hao Wang, Jiawei Zhang, Yi Zhang, Jinwei Wang
Computer Vision
Déjà View: Looping Transformers for Multi-View 3D Reconstruction
Alessandro Burzio, Tobias Fischer, Sven Elflein et al. (+9 more)
Recent feed-forward 3D reconstruction
transformers have scaled to over a billion parameters, following the broader trend of increasing model capacity in computer
vision. Yet emerging evidence suggests that contiguous
transformer layers often behave like repeated applications of similar operations, and multi-view reconstruction
transformers refine their predictions progressively across decoder depth. We posit that model depth partially buys iteration, paid for inefficiently in unique parameters, and instead make that iteration explicit in architecture. Our model, DéjàView, applies a single looped
transformer block recurrently to per-view features for K refinement steps. Trained once, it exposes K as an inference-time compute knob, matching or outperforming substantially larger feed-forward baselines across five reconstruction
benchmarks spanning indoor, outdoor, object-centric, and driving scenes, while using a fraction of their parameters and comparable or lower compute. Importantly, the same looped block formulation outperforms an otherwise identical variant with independent per-step parameters under matched training data and compute, suggesting that explicit iteration is not merely a compute-efficient substitute for capacity but a stronger inductive bias for multi-view 3D reconstruction.
Read the full abstract ▾
All authors Alessandro Burzio, Tobias Fischer, Sven Elflein, Qunjie Zhou, Riccardo de Lutio, Jiawei Ren, Jiahui Huang, Shengyu Huang, Marc Pollefeys, Laura Leal-Taixé, Zan Gojcic, Haithem Turki
Computer Vision
Cycle Consistency in Video Object-Centric Learning
Rongzhen Zhao, Zhiyuan Li, Ruonan Wei et al. (+2 more)
Self-supervised
video Object-Centric Learning (OCL) aims to discover distinct objects and associate them across time, whereas self-supervised Multi-Object Tracking (MOT) focuses on associating pre-defined object detections or
segmentations. Although well-established in MOT, Cycle Consistency (CC) cannot naively or explicitly apply to the latent slot space of OCL. Unlike the deterministic and ideal object representations in MOT, OCL slots are inherently
stochastic and ambiguous due to non-unique scene decompositions. Enforcing explicit cycle consistency (ECC) on slots imposes rigid mean seeking. This severely penalizes the model for exploring alternative but equally valid decompositions, thereby driving towards feature collapse. To resolve this dilemma, we propose Implicit Cycle Consistency (ICC), which shifts the cycle-consistency constraint from the restrictive slot space to the continuous reconstruction
manifold, encouraging slots to reach a soft consensus on collectively interpreting the visual scene rather than forcing rigid point-to-point feature
alignment. Extensive experiments on complex
video OCL
benchmarks demonstrate that ICC avoids feature collapse and outperforms ECC baselines. Our source code, model checkpoints and training logs are provided on https://github.com/Genera1Z/ICC.
Read the full abstract ▾
All authors Rongzhen Zhao, Zhiyuan Li, Ruonan Wei, Juho Kannala, Joni Pajarinen
Computer Vision
LiveSVG: Zero-Shot SVG Animation via Video Generation
Matan Levy, Ran Margolin, Bar Cavia et al. (+6 more)
We introduce LiveSVG, a zero-shot approach for generating Scalable Vector Graphics (SVG) animations using
video diffusion models. Current SVG animation methods struggle with complex motions: LLM-based code synthesis fails to express fine, non-rigid Bézier deformations, while Score
Distillation Sampling (SDS) provides noisy gradients and often requires category-specific priors like skeletons. In contrast, LiveSVG fits vector geometry directly to an explicitly generated target
video. Given an input SVG image and a motion
prompt, we generate a previewable target
video using a frozen image-to-
video model, then fit the original SVG to this
video via differentiable rendering. Our fitting stage is skeleton-free, utilizing a dual-level motion representation that combines per-group homographies for coarse articulation with per-path Bézier control-point offsets for local deformations. To resolve color-induced correspondence ambiguities during pixel-wise fitting, we introduce a novel sphere-packing recolorization strategy. We also present ChallengeSVG, a
benchmark of complex, multi-object scenes that exposes the limitations of prior work. Evaluations demonstrate that LiveSVG significantly outperforms existing methods on both AniClipart and ChallengeSVG, establishing direct reference-
video fitting as a practical, robust route to
prompt-aligned and fully editable vector animation.
Read the full abstract ▾
All authors Matan Levy, Ran Margolin, Bar Cavia, Dvir Samuel, Yael Pritch, Shmuel Peleg, Alex Rav Acha, Ariel Shamir, Dani Lischinski
Computer Vision · Cross-listed from cs.MM
Unveiling the Visual Counting Bottleneck in Vision-Language Models
Xingzhou Pang, Yifan Hou, Junling Wang et al. (+1 more)
While Large
Vision-
Language Models (VLMs) excel at interpolation, they suffer catastrophic failures in systematic generalization, most notably in visual counting. In this work, we investigate this extrapolation bottleneck by deconstructing visual counting into three cognitive stages: visual individuation, magnitude awareness, and symbolic mapping. Using synthetic Go boards and linear probes, we demonstrate that visual backbones maintain robust, linearly separable representations of quantity well into the extrapolation regime, ruling out perceptual failure. Furthermore, models retain latent magnitude awareness, successfully performing comparative
reasoning on quantities they fail to enumerate. We pinpoint the collapse to the symbolic mapping stage, where the model fails to project valid visual magnitudes onto symbolic
tokens. Our findings support a frac tured magnitude hypothesis: VLMs fail to acquire a universal number space, instead learning disjoint, modality-specific statistical
manifolds that prevent cross-modal grounding for unseen quantities. Validated on the state-of-the-art foundation model, our results suggest that bridging this gap requires inductive priors enforcing unified representations, as data scaling alone is insufficient.
Read the full abstract ▾
All authors Xingzhou Pang, Yifan Hou, Junling Wang, Mrinmaya Sachan
Computer Vision
OmniCD: A Foundational Framework for Remote Sensing Image Change Detection Guided by Multimodal Semantics
Chenhao Sun
Change detection (CD) in remote sensing is vital for applications such as urban monitoring and disaster assessment, yet traditional methods struggle with generalization across diverse scenarios. We present OmniCD, a foundational framework that unifies and enhances remote sensing CD through
multimodal semantic guidance. OmniCD incorporates image and text
prompts – such as textual descriptions, semantic maps, and geospatial metadata – into a unified architecture, supporting tasks from binary CD to zero-shot semantic change understanding. The framework integrates a hierarchical scene
retrieval module and a change detection module, reinforced by a style disentanglement mechanism for improved cross-domain robustness. We further introduce RSITCD, a large-scale
multimodal dataset with 300K+ annotated image-text pairs. Extensive experiments show that OmniCD achieves state-of-the-art performance across
benchmarks, demonstrating strong adaptability and setting a solid foundation for general-purpose CD systems in remote sensing.
Read the full abstract ▾
Computer Vision · Cross-listed from stat.ML
Visual Spatial Learning: Single-Field Spatial Interpolation Using Convolutional Neural Networks
Daniel Tinoco, Raquel Menezes, Carlos Baquero et al. (+1 more)
Predicting a complete spatially correlated field from sparse observations is a fundamental challenge in spatial statistics and environmental modelling. Classical interpolation methods such as Kriging rely on Gaussian process assumptions and variography, which can limit their effectiveness in non-stationary settings and require substantial domain expertise. In this work, we leverage an architecture based on convolutional neural networks (CNNs) for spatial interpolation that is trained and applied on a single partially observed field, without access to external data or prior fields. The model is supervised directly on the observed locations and learns to predict values at unobserved points on the user defined grid. Unlike Kriging, our method does not require explicit covariance modelling or variogram estimation, and it can flexibly capture local spatial patterns in a data-driven manner. This work demonstrates the potential of CNNs for single-instance spatial interpolation under sparse supervision, offering a practical alternative to classical geostatistical methods, and extending the use of CNNs to a new problem domain.
Read the full abstract ▾
All authors Daniel Tinoco, Raquel Menezes, Carlos Baquero, Alexandra Silva
Computer Vision
Why Far Looks Up: Probing Spatial Representation in Vision-Language Models
Cheolhong Min, Jaeyun Jung, Daeun Lee et al. (+5 more)
Vision-
language models (VLMs) achieve strong performance on spatial
reasoning benchmarks, yet it remains unclear whether this reflects structured 3D understanding or reliance on statistical shortcuts in natural images. We introduce a representation-level analysis framework that constructs minimal contrastive pairs to measure how spatial axes are organized and disentangled within VLM
embeddings. Our analysis across multiple model families reveals a consistent vertical-distance
entanglement: models conflate vertical image position with distance, mirroring the perspective bias of natural photographs. This bias produces a significant accuracy gap between perspective-consistent and counter-heuristic examples, and intensifies under data scaling even as overall
benchmark accuracy improves. We further show that models with similar
benchmark scores can exhibit different internal representations, and that these differences predict accuracy and robustness across diverse spatial
reasoning benchmarks. To isolate this bias from evaluation-set skew, we introduce SpatialTunnel, a synthetic
benchmark designed to expose spatial shortcut biases by removing common correlations present in natural images. Experiments confirm that the
entanglement is model-intrinsic, and that models with well-separated spatial axes exhibit greater robustness, suggesting that well-structured spatial representations lead to more reliable spatial
reasoning across diverse
benchmarks. Code and
benchmark are available on the project page: https://cheolhong0916.github.io/whyfarlooksup.github.io/.
Read the full abstract ▾
All authors Cheolhong Min, Jaeyun Jung, Daeun Lee, Hyeonseong Jeon, Yu Su, Jonathan Tremblay, Chan Hee Song, Jaesik Park