Computer Vision
Prior Directions: Separating Geometric Identification from Behavioral Efficacy in Visual Revision
Weile Gong, Zijian Lu, Mingcai Chen et al. (+3 more)
Vision-
language models can remain anchored to stale verbal priors even when current visual evidence supports a different answer. We use this controlled revision failure to ask whether the subspace that best captures recurrent prior-induced representation change is also uniquely more behaviorally effective. Across four models, paired prior and reference states define Prior Directions, a compact low-rank geometry that, at rank 32, spans only 0.78% of the state dimensions yet captures 49.7% to 64.0% of held-out displacement energy. Matched correspondence improves held-out capture at every tested rank in all four models; across 20 permutation seeds, this advantage remains positive in all 15 available combinations of model and rank. Yet the
geometric advantage does not translate into a comparable intervention advantage: matched and Pairing-Permuted edits restore 60/66 and 59/66 clean prior-induced failures, respectively. We trace this discrepancy to the geometry of the realized edits. Pairing-Permuted edits remain strongly aligned with the matched Prior Directions; their aligned component restores 59/66 cases, whereas an equally norm-matched orthogonal residual restores only 20/66. These results show that the construction that best identifies recurrent representation change need not be uniquely more behaviorally effective, distinguishing global subspace identification from the geometry actually used by an intervention.
Read the full abstract ▾
All authors Weile Gong, Zijian Lu, Mingcai Chen, Yiping Zuo, Xin He, Weibei Fan
Computer Vision
CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents
Qianru Li, Xuyang Chen, Erkin Türköz et al. (+5 more)
Automatically generating cinematically expressive camera trajectories through 3D scenes from natural
language descriptions is a challenging task of high practical value, with applications ranging from real-estate advertising to virtual tour creation. Existing methods either lack true 3D spatial awareness by relying on 2D image priors, or treat trajectory generation as a
geometric path planning problem divorced from cinematographic semantics. We present CinemaTraj, a framework that reframes camera trajectory planning as a
language-grounded spatial
reasoning problem. Given a set of RGB-D images and a user
prompt, CinemaTraj equips an LLM
agent with a structured 3D scene graph: the
agent decomposes the
prompt into a sequence of atomic cinematographic movements (dolly, orbit, crane, pan, tilt, zoom, arc). Each movement is instantiated via a novel parametric trajectory representation that is both cinematographically expressive and optimizable for collision avoidance. The scene graph acts as a structured spatial prior, grounding the
agent's
reasoning in accurate
geometric and semantic knowledge of the environment. CinemaTraj further generates synchronized voiceover and subtitles aligned with camera motion, producing narrated cinematic
video outputs. We evaluate CinemaTraj on real-world ScanNet++ environments, and show that it produces
prompt-faithful, collision-free trajectories with high cinematographic quality, outperforming existing approaches on
prompt alignment, trajectory quality, and safety metrics.
Read the full abstract ▾
All authors Qianru Li, Xuyang Chen, Erkin Türköz, Lu Liu, Xuqin Wang, Liqiu Meng, Tao Wu, Yanfeng Zhang
Computer Vision · Cross-listed from cs.GR
StructureGS: Structure-aware Gaussian Splatting for Articulated Object Reconstruction
Gahye Lee, Gyoonseo Kim, Wonjong Jang et al. (+2 more)
Reconstructing articulated objects with multiple movable parts is essential for understanding object structure and enabling physical interaction. However, this reconstruction task poses significant challenges due to the
entanglement of geometry, appearance, and motion parameters during optimization. Existing methods rely primarily on photometric supervision, which commonly fails to disentangle these interdependent components, resulting in poor part decomposition with blurred boundaries and
geometric artifacts. To address this limitation, we introduce StructureGS, a reconstruction framework for articulated objects that integrates structure-aware guidance into 3D Gaussian Splatting. Our approach leverages oriented bounding boxes of object parts to enforce two key structural properties: spatial coherence, which constrains each part's geometry to remain compact and spatially coherent within its designated region, and structural connectivity, which enforces physically plausible contact relationships between adjacent parts. These properties are realized through structure-aware losses that inject explicit structural constraints into the optimization process. Extensive experiments demonstrate that our method achieves state-of-the-art performance in articulated object reconstruction, producing high-quality results with well-defined part geometries.
Read the full abstract ▾
All authors Gahye Lee, Gyoonseo Kim, Wonjong Jang, Jooeun Son, Seungyong Lee
Computer Vision
Hearsay: Vision-Language Medical Diagnoses Without an Image
Siddharth Vohra
When asked to describe a medical image that was never attached, frontier
vision-
language models do not abstain: they confabulate a diagnosis. We show that this confabulation is not random. It is structured by who the patient is said to be. Across chest X-ray,
brain MRI, and dermatology, Claude Opus-4.7, GPT-5.4, and Gemini-3.1-Pro are each queried with only a demographic descriptor and no image, and changing the descriptor systematically shifts the diagnosis returned. Claude concentrates sharply: a 65-year-old white man asking about a skin mole receives Melanoma in nearly every response, and a 32-year-old Black woman asking about her chest X-ray receives a Sarcoidosis diagnosis whose
reasoning reads "suspected, based on demographics and classic pattern.” GPT-5.4's effect is broader, fabricating across every demographic
cell we test, most conspicuously naming Sarcoidosis for young Black patients on chest X-ray. Two structural findings sharpen the problem. A hedged regime appears in which the prose acknowledges the missing image while the structured diagnosis field nevertheless names a disease, a dissociation invisible to prose-only audits. And Claude's dermatology effect collapses entirely when 'skin mole' is swapped for 'skin lesion' while GPT-5.4's is preserved, indicating that mirage is a family of distinct failure modes rather than a single phenomenon. Trustworthy VLM deployment in clinical pipelines requires
auditing the structured output channel directly, and probe-word sensitivity should be treated as a first-class evaluation dimension
Read the full abstract ▾
Computer Vision
SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation
Yunzhan Fu, Enyu Bao, Xiangyu Shen et al. (+4 more)
Vision-
language pre-training (VLP) serves as a cornerstone for medical
multimodal representation learning. However, existing medical VLP frameworks are often constrained by the limited context windows and shallow representational capacities of lightweight text encoders when processing lengthy, terminology-dense clinical reports. While integrating medical large
language models (LLMs) offers unprecedented clinical
reasoning capabilities, it introduces three major bottlenecks: (i) the anisotropic representational collapse of generative LLMs under standard contrastive objectives, (ii) the prohibitive
memory overhead of joint end-to-end training with large batch sizes, and (iii) the medical
hallucinations induced by vanilla contrastive losses that ignore fine-grained anatomical laterality and negation modifiers. To address these challenges, we propose SCALPEL, a Semantic Cross-modal
Alignment framework via LLM-Powered Encoder Learning. First, Clinical Report Contrastive fine-tuning converts a generative LLM into an isotropic encoder via domain-specific clinical text adaptation. Second, an asymmetric
alignment strategy leverages offline feature caching to enable efficient training. Critically, we formulate an Anatomy-Negation Aware Objective that explicitly penalizes mismatched image-text pairs involving laterality confusion or false negations. Extensive experiments across MIMIC-CXR, CheXpert, and IU X-Ray
benchmarks demonstrate that SCALPEL achieves state-of-the-art performance in cross-modal
retrieval, zero-shot disease classification and medical visual question answering.
Read the full abstract ▾
All authors Yunzhan Fu, Enyu Bao, Xiangyu Shen, Yihao Wu, Chunbo Jiang, Fangli Guan, Liqi Yan
Computer Vision
ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figures
Fahad Ahmed, Sören Auer and Jennifer D'Souza
Scientific figure comprehension and
reasoning using
multimodal AI requires integrating visual perception with domain-specific
reasoning to extract meaningful knowledge, often not presented in the text of a research publication. The Sci-ImageMiner
benchmark dataset, accompanied by a community-driven competition, raises the bar over prior scientific competitions by curating a comprehensive, expert-annotated
dataset across four end-to-end complementary tasks. The competition attracted 68 active participants and 1,263 public/private submissions from 9th January 2026 to 8th April 2026. Our results show that state-of-the-art
multimodal models perform well on classification and summarization tasks but struggle with data extraction and scientific
reasoning, particularly in visual question-answering. These findings reveal key limitations and highlight challenges and opportunities for improving domain-aware
multimodal AI systems. Overall, the Sci-ImageMiner
benchmark and competition establish a rigorous platform for advancing research in scientific figure comprehension and
reasoning and demonstrate the potential of state-of-the-art approaches for a challenging and complex research area.
Read the full abstract ▾
Computer Vision
BATS: Resource-Efficient Volumetric Segmentation with Boundary-Aware Mixed-Resolution Tokens
David Hagerman, Roman Naeem and Fredrik Kahl
Many high-performing volumetric
segmentation models maintain dense multi-scale feature maps, leading to high activation
memory and inference cost. We present BATS (Boundary-Aware
Token Selection), a 3D medical image
segmentation architecture that concentrates fine-resolution processing near predicted class boundaries. A dense boundary predictor identifies where additional resolution is needed, while a fine-first context cascade constructs an input-dependent mixed-resolution hierarchy. Homogeneous regions are represented coarsely, with finer
tokens retained around boundaries, thin structures, and small targets. The sparse hierarchy is refined and rasterised into a dense
segmentation. BATS predicts boundary relevance independently at every resolution level, preventing an erroneous coarse-scale decision from suppressing fine-scale evidence. Parent cluster
attention further injects hierarchical ancestor
tokens into local
attention neighbourhoods, providing cross-scale context without dense multi-scale feature maps or cross-scale neighbour search. We evaluate BATS on five public CT and MRI
datasets using the standardised nnU-Net Revisited protocol. BATS achieves the highest LiTS Dice among the compared methods and averages within 0.37 Dice points of the strongest dense baseline, MedNeXt-L, across the five
datasets. Relative to MedNeXt-L, it reduces peak allocated GPU
memory by more than 53% on KiTS, LiTS, and BraTS. Inference is up to 30% faster on KiTS and LiTS, which retain fewer
tokens, but slower on the more
token-dense BraTS. Mixed-resolution processing therefore provides consistent
memory savings, while runtime and accuracy gains depend on
dataset boundary density.
Read the full abstract ▾
Computer Vision
Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory
Yanbo Ding, Zhizhi Guo, Quanyue Song et al. (+4 more)
Audio-
video generative models achieve impressive quality but suffer from high latency, making them unsuitable for real-time applications. Although several streaming audio-
video generation methods have been proposed, they remain costly and fail to support long-form generation. To address this, we propose Ripple, a real-time joint audio-
video generation system with a cross-modal recurrent
memory mechanism. To enable efficient streaming inference while preserving long-term context, Ripple combines a fixed-length sliding-window
attention with modality-specific
memory states that continuously summarize audio and
video context. Cross-modal
memory interaction is further introduced to enhance audio-visual synchronization. To learn this
memory-augmented model effectively, we devise a three-stage training recipe: (1) adapting a bidirectional audio-
video teacher to block-wise causal
attention with simulated
memory, (2) optimizing the
memory construction and interaction pipeline through end-to-end
distillation, and (3) applying online reinforcement post-training tailored for streaming audio-
video generation. As a result, Ripple achieves ~28 FPS at 480P resolution, over faster than the teacher, while capable of coherent long-form generation. Extensive experiments on both short-
video and long-
video benchmarks demonstrate our superior performance over existing offline and online joint audio-
video generation methods.
Read the full abstract ▾
All authors Yanbo Ding, Zhizhi Guo, Quanyue Song, Yishan He, Zhixiang He, Yongxiang Li, Yali Wang
Computer Vision · Cross-listed from cs.RO
From Uncertainty to Determinism: Coarse-to-Fine Visual Floorplan Localization without Ray Matching
Shiyong Meng, Bolei Chen, Ping Zhong et al. (+4 more)
Visual Floorplan Localization (FLoc) has emerged as a promising solution for indoor localization by matching
egocentric images against minimalist structural maps. However, due to cross-modal information asymmetry and repetitive indoor layouts, visual FLoc is fundamentally challenged by
multimodal pose distributions, where visually identical observations map to distinct, spatially separated locations. Existing ray-matching-based methods tackle this by explicitly predicting sparse
geometric or semantic rays, which inherently incur information loss and demand resource-intensive preprocessing alongside exhaustive matching during inference. In this paper, we bypass the intermediate ray-matching paradigm and propose a coarse-to-fine visual FLoc framework that progresses from uncertainty to determinism. In the coarse stage, we design an image-conditioned pose
diffusion model to parameterize the continuous
multimodal pose distribution, effectively routing stochastically initialized pose particles toward distinct candidate modes. In the refinement stage, we propose a localized refiner that predicts bounded sub-meter pose residuals from candidate-centered floorplan crops, where structural ambiguities are largely eliminated. Our method effectively balances global multi-hypothesis tracking and local sub-meter refinement without requiring any offline map preprocessing or test-time lookup tables. Comprehensive results on the S3D (full) and ZInD
benchmarks demonstrate that our approach achieves state-of-the-art accuracy and robustness.
Read the full abstract ▾
All authors Shiyong Meng, Bolei Chen, Ping Zhong, Yang Wan, Rongzhi Wang, Jiazhi Xia, Jianxin Wang
Computer Vision
DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation
Jiaxing Li, Kai Zou, Cindy Zhou et al. (+7 more)
Existing autoregressive
video distillation methods commonly adopt a Distribution Matching
Distillation (DMD)-based multi-stage pipeline. However, they typically decouple the initialization and DMD stages – which then pursue different target distributions – and judge the intermediate student mainly by visual scores such as VBench. In this paper, we revisit this design from a distributional perspective. Given the mode-seeking nature of the distribution matching loss, a good initialization should match the mode coverage of the target DMD teacher, rather than merely pursuing high quality. To analyze this, we introduce a distributional evaluation protocol that measures precision and coverage between student and teacher distributions in a shared latent space. It exposes differences hidden by visual scores: some initializations reach high precision but low coverage, leading to suboptimal refinement, while mode-covering ones preserve broader support. Furthermore, even when the target distributions are aligned, DMD's reverse-KL objective can still drive the student toward high-probability teacher regions in late training, reducing coverage and diversity. To address this, we propose joint
distillation, which combines DMD's mode-seeking objective with a Consistency
Distillation-based mode-covering constraint. Experiments show that our method improves generation quality, coverage, and diversity; notably, even with a Wan-1.3B DMD teacher, it outperforms baselines refined with Wan-14B, underscoring the importance of distributional
alignment in autoregressive
video distillation.
Read the full abstract ▾
All authors Jiaxing Li, Kai Zou, Cindy Zhou, Kaichen Huang, Junyao Gao, Zile Wang, Yang Liu, Bin Liu, Bo An, Yangguang Li
Computer Vision
PRISM-Net: Patient-specific reference-guided inter-breast symmetry matching for three-class breast DCE-MRI classification
Boya Zhang, Shuaiwen Zhou, Di Kong et al. (+7 more)
Breast DCE-MRI AI is increasingly being explored for breast-level classification of no-lesion, benign, and malignant findings, beyond conventional lesion-centered diagnosis. Within this broader diagnostic scope, however, patient-specific background variability remains a major source of imaging confounding across classification tasks. Existing approaches predominantly focus on unilateral or lesion-centric analysis, whereas bilateral methods offer limited explicit modeling of spatially adaptive cross-breast correspondence. We propose PRISM-Net, a registration-free bilateral framework that leverages contralateral breast features as patient-specific references for background-aware representation learning. PRISM-Net integrates bilateral feature matching and asymmetry-aware
attention to establish adaptive inter-breast correspondence and enhance representations of discriminative asymmetric patterns. On ODELIA, Macro AUC, Micro AUC, and quadratic weighted kappa were $84.11 \pm 2.33$, $90.64 \pm 1.61$, and $60.94 \pm 5.64$ on the in-distribution test set, and $68.51 \pm 4.54$, $80.74 \pm 2.68$, and $43.45 \pm 7.10$ on the held-out institution, respectively, outperforming the evaluated baseline methods across the primary evaluation metrics. PRISM-Net further demonstrated performance on independent institutional and background-complexity evaluations. Ablation experiments revealed that both bilateral relation modeling and asymmetry-aware reweighting contributed to improved classification performance. These findings highlight patient-specific bilateral reference modeling as a clinically grounded strategy for DCE-MRI interpretation, improving asymmetric pattern discrimination through explicit modeling of background complexity.
Read the full abstract ▾
All authors Boya Zhang, Shuaiwen Zhou, Di Kong, Mingxu Wang, Wenbiao Du, Yiman Zhong, Yuexin Duan, Xiawei Yue, Liuquan Cheng, Xiru Li
Computer Vision
See2Think: Do Multimodal Models Really Use Intermediate Visual States?
Siyu Yan, Zhuoran Yan, Haiying Xu et al. (+10 more)
Multimodal large
language models increasingly use sketches, annotations, tools, and intermediate images during
reasoning, but it remains unclear whether they truly rely on these visual states. Existing
benchmarks are limited both by task collections with narrow coverage or partially text-solvable samples and by evaluations that emphasize final answers without diagnosing how intermediate visual states are generated, rendered, and used. We introduce See2Think, a unified evaluation framework comprising See2ThinkBench and Visual Action-of-Thought (VAoT). See2ThinkBench contains 1,200 open-ended, visually dependent problems across 12 task categories spanning 2D structured, 3D scene, and real-world
reasoning. VAoT records textual thoughts, visual actions, rendered states, and subsequent
reasoning under four controlled inference settings. Evaluating representative proprietary and open-source
multimodal models, we find that visual
reasoning is strongly model- and environment-dependent, with no single setting consistently dominating across tasks. Process analysis further shows that models usually select relevant visual operations, while faithful rendering remains the clearest bottleneck and high feedback uptake does not necessarily translate into accuracy gains. Under task-relevant corrupted feedback, models exhibit behavioral dependence on visual states, with accuracy dropping by over 10 percentage points in controlled interventions.
Read the full abstract ▾
All authors Siyu Yan, Zhuoran Yan, Haiying Xu, Panhao Zhou, Jingyu Chen, Chenhao Ji, Shuo Cao, Yongheng Zhang, Haoze Liu, Siyu Zhang, Xiwen Gu, Yihao Liu, Alex Jinpeng Wang
Computer Vision
Searching for Robust Augmentations to Improve Out-of-Domain Generalization in Dermoscopic Skin Cancer Classification
Alexander Kozachok, Ilya Latyshev, Evgeny Karpulevich et al. (+3 more)
Background/Objectives: Dermoscopic skin-lesion classifiers lose accuracy when images arrive from a new clinic or a new device. We asked which data augmentations reduce that loss, and measured the effect under a protocol that keeps policy selection separate from policy evaluation. Methods: A ConvNeXt-Large binary malignant-versus-non-malignant classifier was trained on six dermoscopic sources (25,903 images); HAM10000 and ISIC 2016-2020 were held out of training entirely. Single augmentations, photometric combinations and eleven composite policies were ranked on a development split of 1511 held-out images. The winning policy was then evaluated on a confirmation set of 8073 held-out images that took no part in that ranking and from which we removed every image sharing a lesion identifier with the training data and every image contributed by an institution represented in training. Both policies were retrained with four random seeds each and compared with an exact permutation test. Results: The mix policy raised confirmation-set ROC-AUC from 0.787 to 0.826 (+0.039; per-seed ranges 0.772-0.797 and 0.815-0.840, non-overlapping; exact permutation p=0.029), with the same direction on each contributing source. At matched sensitivity the gain is larger in clinical terms: specificity rose from 0.612 to 0.713 at a sensitivity of 0.80, and from 0.284 to 0.397 at a sensitivity of 0.95. In-domain ROC-AUC was preserved (0.938 to 0.941). On an independent clinical cohort acquired with a different device at a different institution (472 images, 22 malignant), performance was maintained (0.934 versus 0.930). Conclusions: Augmentations that model the physical causes of domain shift improve cross-source transfer at no cost to in-domain accuracy, and the improvement survives a selection-disjoint, contamination-free evaluation.
Read the full abstract ▾
All authors Alexander Kozachok, Ilya Latyshev, Evgeny Karpulevich, Elena Kozachok, Egor Ushakov, Oleg Samovarov
Computer Vision
Jiahao You, Xu Han, Jinfeng Xu et al. (+1 more)
Dataset distillation compresses large-scale
datasets into compact synthetic sets while preserving their training utility, enabling efficient 3D point cloud training. Current point cloud
dataset distillation methods only tackle
geometric and representation challenges while ignoring the distributional imbalance prevalent in point cloud
datasets where both training and test splits follow long-tailed class distributions. To our knowledge, we present the first study on long-tailed point cloud
dataset distillation. Rather than focusing primarily on
geometric and representation properties or simply constructing a class-balanced synthetic set, our framework explicitly accounts for long-tailed class distributions via two core modules. First, we design Adaptive Synthetic Budgeting to allocate class-wise synthetic budgets according to class quantity and the expected benefit of additional synthetic samples. Given the allocated budgets, we further design 3D Long-Tailed Distribution Matching to optimize synthetic point clouds through Global-Local Feature
Alignment and Prior-Aware Supervision. The former preserves both global class distributions and diverse intra-class structures, while the latter provides class-dependent expert supervision to keep tail-class samples recognizable while maintaining diverse head-class patterns. Extensive experiments demonstrate the effectiveness of our method, lifting classification accuracy by 7.0 points on ShapeNet55 against state-of-the-art methods.
Read the full abstract ▾
All authors Jiahao You, Xu Han, Jinfeng Xu, Xianzhi Li
Computer Vision
StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation
Zijun Lin, Zeqing Wang, Cheston Tan et al. (+2 more)
Recent game world models can generate visually realistic and interactive environments conditioned on player actions. However, games are not defined by pixels alone; they are governed by explicit mechanics, namely state-dependent rules that control health reduction, skill activation, and game termination. These mechanics depend on precise internal states, such as health points, skill meters, and timers, which are tightly coupled with visual observations and determine how gameplay evolves. Without modeling these state dynamics, existing game world models may generate visually plausible rollouts but violate the underlying game rules. In this paper, we propose StatePlay, a novel state-aware game world model that jointly predicts visual content and game states to promote mechanics-consistent generation. StatePlay adopts a mixture-of-
transformers (MoT)-style architecture that preserves specialized visual and state representations while enabling cross-modal interaction, allowing predicted states to guide frame generation. Each branch is further optimized with a distinct objective suited to its modality. Experiments show that StatePlay achieves an average normalized L1 distance below 0.06 for state prediction. Furthermore, compared with models without explicit state modeling, our method improves mechanics fidelity in generated game rollouts by 18.6%. Overall, our work highlights the importance of state-aware game world modeling and advances beyond pixel-level realism toward complete and mechanically faithful game generation.
Read the full abstract ▾
All authors Zijun Lin, Zeqing Wang, Cheston Tan, Bihan Wen, Yeying Jin
Computer Vision
Multimodal fusion of visual and morphometric features for avian bone classification
Nevio Dubbini, Lisa Yeomans, Marco Pavia et al. (+4 more)
Artificial intelligence has shown considerable potential for archaeological applications, yet its use in zooarchaeology remains limited, particularly for the identification of avian skeletal remains. This study presents a proof-of-concept
multimodal framework that integrates convolutional neural network-based image analysis with osteometric measurements for the classification of bird bones. Using a
dataset of more than 10,000 images from multiple museum and research collections, two classification tasks were investigated: skeletal element identification and family-level taxonomic classification. Prior to classification, images were automatically segmented using a two-stage pipeline combining BiRefNet and SAM2. Visual features extracted with a pre-trained EfficientNet_V2_S backbone were fused with standardized morphometric data through a feature-level
multimodal architecture. The model achieved 86% accuracy on the test set for bone-type classification, demonstrating reliable recognition of skeletal elements. Family-level classification proved more challenging, reaching 51% top-1 accuracy but 75% top-3 accuracy, indicating that correct taxa were frequently included among the most probable predictions. These results demonstrate the feasibility of combining visual and morphometric information within a unified deep-learning framework and establish a methodological baseline for future AI-assisted zooarchaeological identification. The approach contributes to ongoing efforts to develop scalable, interpretable, and archaeologically meaningful tools for the study of avian remains.
Read the full abstract ▾
All authors Nevio Dubbini, Lisa Yeomans, Marco Pavia, Ramazan Parmaksiz, Ayse Atas Hooglugt, Gabriele Gattiglia, Beatrice Demarchi
Computer Vision
Dual Inversion for Text-to-Image Diffusion Models: From Both Prompt and Noise Perspectives
Xiaolong Liu, Junjian Li, Yuan Xiao et al. (+4 more)
Prompt inversion, as a typical reverse engineering technique, enables text-to-image (T2I)
diffusion models to generate the desired target images without extensive
prompt engineering. However, existing
prompt inversion methods suffer from significant limitations: (1) gradient-based methods are unstable and uninterpretable, often resulting in generated images with severe artifacts; (2) gradient-free methods yield human-readable
prompts but still fail to preserve visual fidelity due to the lack of fine-grained detail
alignment. We contend that the limitations stem from treating
prompt inversion as a sufficient condition for reverse engineering, ignoring the critical role of the latent noise that encodes structural information. Consequently, we propose Dualin (Dual inversion), a two-stage method that jointly recovers both the semantic
prompt and latent noise of the target image. In the first stage, we integrate
vision-
language model, CLIP and large
language model to invert a faithful, human-interpretable hard
prompt. In the second stage, unconditional DDIM inversion reconstructs the exact latent noise of the target image, guaranteeing the consistency at the structural information level. Theoretically, we prove that the inverted noise enables flexible image editing without re-optimization. Extensive experiments on diverse
datasets demonstrate that Dualin simultaneously generates high-quality inverted
prompts and achieves state-of-the-art image fidelity. Additionally, Dualin can establish a robust foundation for the precise and controllable image editing.
Read the full abstract ▾
All authors Xiaolong Liu, Junjian Li, Yuan Xiao, Jiaqi Deng, Dayong Ye, Tianqing Zhu, Huan Huo
Computer Vision
Online Handwriting Trajectory Reconstruction from Kinematic Sensors using Temporal Convolutional Network
Wassim Swaileh, Florent Imbert, Yann Soullard et al. (+2 more)
Handwriting with digital pens is a common way to facilitate human-computer interaction through the use of Online Handwriting (OH) trajectory reconstruction. In this work, we focus on a digital pen equipped with sensors from which one wants to reconstruct the OH trajectory. Such a pen allows to write on any surface and to get the digital trace, which can help learning to write, by writing on paper, and can be useful for many other applications such as collaborative meetings, etc. In this paper, we introduce a novel processing pipeline that maps the sensor signals of the pen to the corresponding OH trajectory. Notably, in order to tackle the difference of sampling rates between the pen and the tablet (which provides ground truth information), our preprocessing pipeline relies on Dynamic Time Warping to align the signals. We introduce a dedicated neural network architecture, inspired by a Temporal Convolutional Network, to reconstruct the online trajectory from the pen sensor signals. Finally, we also present a new
benchmark dataset on which our method is evaluated both qualitatively and quantitatively, showing a notable improvement over its most notable competitor.
Read the full abstract ▾
All authors Wassim Swaileh, Florent Imbert, Yann Soullard, Romain Tavenard, Eric Anquetil
Computer Vision
CASIAL: Geometric Distortion Robust Image Watermarking
Yupeng Qiu, Han Fang and Ee-Chien Chang
Deep learning-based watermarking has shown strong robustness against non-
geometric distortions, yet its performance under
geometric transformations remains limited. Such transformations induce two fundamental failure modes: region removal, such as cropping or masking, which eliminates the information carried by removed pixels, and desynchronization, such as scaling or rotation, which misaligns pixel positions and disrupts decoding. We argue that achieving
geometric robustness requires two essential properties: (1) global spread of the watermark message, ensuring resilience even when large regions are removed, and (2) geometry-invariant representations, enabling decoding to remain synchronized despite spatial transformations. Building on these insights, we propose CASIAL, a
geometric distortion-robust watermarking framework with cover image-aware message spreading (CAS) strategy and invariance
alignment learning (IAL) module. CAS tightly couples watermark bits with cover image features and distributes them adaptively across the entire image, enhancing per-pixel information capacity and robustness to region removal. IAL leverages spatial
attention to capture cross-pixel dependencies and align perturbed features into a shared geometry-invariant representation space, mitigating failures due to desynchronization. Across six challenging
geometric transformations, CASIAL achieves substantially stronger robustness than eleven prior baselines while preserving high visual quality. It also maintains competitive performance under six signal distortions and four photometric transformations. Notably, although trained only with white-box distortions, CASIAL also exhibits strong transfer robustness to unseen black-box distortions. Comprehensive experiments demonstrate the broad robustness and superior visual quality of our method.
Read the full abstract ▾
Computer Vision
FreeShadow: Training-Free Shadow Removal via Illumination Transfer and Selective Content Preservation in Diffusion Models
Yinan Wang, Yan Huang, Yong Xu et al. (+1 more)
Existing supervised and unsupervised shadow removal methods often suffer from limited generalization due to the insufficient diversity of available training
datasets, while zero-shot methods tend to produce artifacts and require time-consuming test-time optimization. To address these issues, we propose FreeShadow, a training-free shadow removal method built upon pretrained
diffusion models, which exploits
diffusion priors for shadow removal without any training or optimization. For illumination recovery, we propose an illumination transfer
attention (ITA), which re-weights the self-
attention maps in
diffusion model to transfer illumination cues from non-shadow to shadow regions. For content preservation, we analyze the effects of illumination variations on self-
attention maps and latent high-frequency features in
diffusion model, and selectively preserve illumination-invariant components to maintain content fidelity while suppressing residual shadows. We further propose local texture-preserving relighting (LTPR) to mitigate local texture misalignment caused by VAE compression. Extensive experiments demonstrate that our method achieves strong generalization and produces realistic shadow-free images.
Read the full abstract ▾
All authors Yinan Wang, Yan Huang, Yong Xu, Patrick Le Callet
Computer Vision
TPD: Temporal Prior Decoupling for Text-to-Video Diffusion Models
Taewon Kang and Matthias Zwicker
Text-to-
video diffusion models generate temporally coherent content from natural
language, yet when a
prompt describes an early scene that persists while a new event emerges on top of it—such as "a tall sandcastle standing on a beach where a wave rushes in and washes it away"—generation frequently fails to realize the late-segment event in the corresponding frames. We identify this failure as Temporal Prior Suppression (TPS): the dominant prior of the early segment captures the cross-
attention trajectory across the temporal axis and suppresses the guidance signal needed for late-segment realization, a competing tendency existing guidance mechanisms do not model. We introduce Temporal Prior Decoupling (TPD), a training-free framework that restores suppressed late-segment signals during
diffusion sampling. TPD constructs a temporal counterfactual by conditioning on the early segment alone, and defines the discrepancy between the full-
prompt and counterfactual trajectories as a suppressed signal direction. Rather than removing this direction as in prior subtractive projection methods, TPD restores it through a frame-selective lower-bound constraint resolved jointly over
diffusion timestep and
video frame, realizing the suppressed event in the late frames without disrupting early-segment coherence: where prior work enforces upper-bound feasibility to remove unwanted semantics, TPD enforces lower-bound feasibility to guarantee suppressed-signal contribution. TPD runs entirely within standard
diffusion sampling without retraining, and is defined purely in classifier-free guidance space, making it backbone-agnostic by construction. Experiments show that TPD significantly improves late-concept realization while preserving temporal coherence and visual fidelity, and that the targeted suppression recurs across distinct text-to-
video backbones.
Read the full abstract ▾
Computer Vision
Sequence-SOD: Bio-inspired Sequence-aware Spiking ObjectDetection for Event Cameras
Katharina Bendig, René Schuster and Didier Stricker
Event cameras follow a retina-inspired sensing principle, reporting local intensity changes asynchronously with hightemporal resolution and a wide dynamic range. Spiking Neural Networks (SNNs) complement these sparse event streams through
brain-inspired dynamics, using sparse spikes and leaky membrane potentials to integrate information over time. However, many SNN object detectors process isolated event intervals with a single label and reset the network state after each prediction, thereby underusing temporal information in continuous event streams. We introduce Sequence-SOD, a sequence-aware SNN object detector that processes extended event sequences containing labels at multiple time points. Events are accumulated into short intervals, discretized into temporal steps, and fed sequentially to an SSD-style Spiking DenseNet while preserving membrane potentials across intervals within a sequence, so that detection is driven by an evolving neural state instead of independently reset input windows. On the Gen1 Automotive Detection
Dataset, sequence-aware training improves mAP from 23.38 for single-interval training to 25.30 without augmentation and to 26.88 withevent-data augmentation. The model achieves a theoretical prediction frequency of 40 Hz. Training and evaluating SNN object detectors on extended event sequences improves their ability to exploit temporal cues while preserving the energy-efficiency benefits of sparse spiking computation. The results highlight sequence-aware training as a complementary direction to architectural improvements for event-based SNN detection.
Read the full abstract ▾
Computer Vision
Visko Orbis 1.0: A Live Model for Real-Time Interactive Long Video Generation
Xiangbo Gao, Siyuan Yang, Ping He et al. (+12 more)
We present Visko Orbis 1.0, a Live Model for real-time, interactive long
video generation. Users can change the
prompt at any moment during generation, and the update becomes visible in real time. Visko Orbis 1.0 supports long-form text-to-
video, image-to-
video, and
video continuation, with
multilingual prompts and
prompt switching while generation is in progress. A bounded multi-scale
memory preserves subjects, scenes, and style across chunks, sustaining hour-scale rollouts without evident quality or color drift. The generator is factorized causally in time, matching the causal structure of physical dynamics, and is aligned with a latent world-model reward for predictive consistency. Built on a distilled chunk-wise streaming generator and a streaming
video upscaler, Visko Orbis 1.0 delivers 4K
video generation at 24 FPS in real time, using an optimized GPU serving engine. In quantitative evaluations, Visko Orbis 1.0 achieves the best DOVER aesthetic and technical scores and the best VideoAlign visual and motion quality, and leads three physical-plausibility protocols (VideoPhy-2, Physics-IQ, and VBench-2.0 Physics); in long-form Arena comparisons, it obtains the highest overall-preference and temporal-stability ratings among all the state-of-the-art real-time interactive
video generation systems.
Read the full abstract ▾
All authors Xiangbo Gao, Siyuan Yang, Ping He, Mingyang Wu, Yuheng Wu, Yushen Zuo, Jiongze Yu, Ryan Cui, Hongyuan Hua, Devin Ma, Xiao Jin, Yubo Ruan, Qing Yin, Jie Yang, Zhengzhong Tu
Computer Vision · Cross-listed from cs.RO
Enfold: Folding World Model Imagination into Predictive Representations for Ultra-Efficient Embodied Control
Weili Zeng, Yitong Xing, Fulong Liu et al. (+10 more)
World generative models are typically used through what they produce: a rendered future, a
video-conditioned action, or latent context computed by a costly generative branch. We argue that their more reusable asset is the computation that constructs a future. As a generator transforms a corrupted future into a coherent trajectory, its intermediate states organize appearance, spatial layout, and interaction across levels of abstraction. Can this future-generative computation be internalized in a representation inferred from the present alone? We present Enfold, which transfers this computation into a representation predicted from the current visual context and
language instruction. During training, multi-level states exposed as the generator processes the observed future supervise a current-only encoder. The learned representation is fed back to condition future generation and is read by task heads without allowing task gradients to reshape the encoder. At deployment, action prediction no longer executes the generator. Across LIBERO, RoboTwin2.0, and real-
robot tasks, Enfold supports strong control while reducing action latency by $3.7\times$ relative to Fast–WAM, Enfold-Flash reaches $10.1\times$. Representation analyses show that it suppresses nuisance variation and preferentially captures changes that emerge over longer horizons. When the current scene is altered by human intervention, both the generated continuation and the executed actions adapt, which is inconsistent with fixed trajectory replay. These results recast a world generator as a source of predictive control representations: its future need not be materialized at every step if its internal structure can be enfolded into the present.
Read the full abstract ▾
All authors Weili Zeng, Yitong Xing, Fulong Liu, Chengqun Yang, Antao Xiang, Feng Tian, Jingnan Gao, Jisong Cai, Xin Wang, Xiaomin Wu, Yao Mu, Xiaokang Yang, Yichao Yan
Computer Vision
Physically Real-time Infrared Attack against Optical Flow Estimation Networks
Shen You, Wei Jiang, Jiarui Liu et al. (+4 more)
With the promising performance of deep neural networks on image-based tasks, different real-world applications such as autonomous driving and motion detection have become increasingly mature and relevant to human lives. In particular, Optical Flow Estimation Networks (OFENs), as upstream models, play a critical role in different domains. Its outputs are heavily assumed and adopted for different downstream tasks, and it is essential to test its robustness to prevent safety accidents. We present an approach for real-time attacks on OFENs in the physical world, leveraging infrared lights for their stealthiness. By generating a large number of
Adversarial Examples in advance, our approach computes AEs in real time and dynamically displays them, which allows our method to facilitate precise and targeted attacks without modifying the victim system. Unlike previous digital-to-physical attack techniques, our method directly attacks victim models within the physical world, thereby overcoming the limitations associated with the ineffectiveness of AEs. Experimental results demonstrate the efficacy of our approach in compromising OFENs across diverse lighting conditions, varying object motion velocities, and different object placements, ultimately impairing the network's ability to accurately estimate optical flow.
Read the full abstract ▾
All authors Shen You, Wei Jiang, Jiarui Liu, Yijian Ye, Qiuzhen Lin, Xiangtao Li, Ka-Chun Wong
Computer Vision
Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time
Xinyi Wang, Yuyang Huang, Yalin Su et al. (+4 more)
While text-to-image
diffusion models achieve impressive visual quality, they frequently struggle to maintain precise
alignment with complex compositional
prompts. An effective strategy is to improve the inference process of
diffusion models, thereby better leveraging their pretrained priors to address misalignment. Existing training-free methods can be divided into two categories. The first category focuses on improving the randomly sampled initial noise, either performing costly search over noise pools or manipulating sampled noise without ensuring reliable semantic injection. The second category focuses on improving the denoising trajectory, lacking explicit mechanisms to timely diagnose and correct semantic errors. we propose AnchorSteer, a training-free framework that exerts fine-grained control over both initialization and the denoising trajectory. AnchorSteer consists of two synergistic components: Semantic Anchoring replaces uninformative Gaussian noise with text-aligned initializations via CLIP-based prior extraction and a novel Latent-Prior Score
Distillation Sampling (LP-SDS) objective. Specifically, LP-SDS distills CLIP visual priors into the knowledge distribution of
diffusion models, mitigating the domain gap between CLIP-based priors and
diffusion-based priors. Reflective Steering transforms passive denoising with an active Think–Erase–Retouch loop that enables mid-generation self-correction. It leverages VLM-based diagnosis to detect semantic deviations and performs targeted latent refinement to suppress erroneous content and recover missing attributes. Extensive experiments on GenEval and T2I-CompBench++ demonstrate that AnchorSteer consistently outperforms existing baselines in text–image
alignment while preserving high visual quality.
Read the full abstract ▾
All authors Xinyi Wang, Yuyang Huang, Yalin Su, Pengcheng Luan, Tao Zhang, Feiming Wei, Wenxian Yu
Computer Vision
Genie Sim PanoWorld: An Infinite Indoor 3D World Generation Pipeline via Panoramic Scene Modeling and Simulation
Yongxin Su, Linjie Hou, Feng Wang et al. (+4 more)
We address the problem of reconstructing a high-fidelity, freely navigable 3D scene from a single $360^\circ$ panorama, without per-scene optimization or multi-view capture. Existing methods either lack metric trajectory control, which hinders reliable downstream 3D reconstruction, or struggle with large disocclusions under long-range camera motion while requiring high-end multi-GPU servers.We present Genie Sim PanoWorld, a two-stage feed-forward pipeline that bridges generation and reconstruction via an explicit, trajectory-controllable panoramic
video. A NavMesh-planned $\mathrm{SE}(3)$ roaming trajectory is injected into a latent
video diffusion model through dense geometry-warped conditioning; long–short trajectory mixed training and a self-consistency objective based on shortcut models together yield high-fidelity
video in four CFG-free denoising steps. A feed-forward panoramic reconstructor then lifts the generated
video into a high-fidelity 3D Gaussian scene that supports real-time, free-viewpoint roaming and can be directly used as a simulation-ready asset for
embodied AI applications. Experiments show that Genie Sim PanoWorld outperforms geometry-conditioned baselines in both panoramic
video generation and downstream 3D reconstruction, while generalizing zero-shot to unseen indoor scenes.
Read the full abstract ▾
All authors Yongxin Su, Linjie Hou, Feng Wang, Jialin Tang, Zhijun Li, Qian Wang, Maoqing Yao
Computer Vision
FPSGen: Flexible Point Cloud Scene Generation with BEV-Supported Transport Flows
Wenzhe He, Meng Wang, JiaWei Qian et al. (+3 more)
Existing point-based generative methods for outdoor scenes primarily focus on
LiDAR-conditioned completion. During training, noisy point clouds are constructed by perturbing complete ground-truth scenes, whereas during inference, they are initialized by adding noise to duplicated partial scans. This train-inference mismatch inherits the sparsity and visibility bias of partial scans, leading to sparse distant regions and incomplete geometry in occluded areas. Moreover, the reliance on partial scans restricts generation when
LiDAR observations are unavailable or replaced by layout cues. We present FPSGen, a flexible framework that constructs point sources independently of partial scans. FPSGen first predicts a bird's-eye-view (BEV) prior with density, height, and mask channels from the active cues. The density map is then sampled to form a BEV-supported point source, enabling both unconditional and conditioned initialization. A teacher-student approximate optimal transport scheme then uses teacher-predicted endpoints to learn a velocity field that induces straighter transport paths. By integrating BEV point source construction with path-straightening transport, FPSGen provides a unified framework for unconditional and flexible cue-conditioned scene generation. Extensive experiments show that FPSGen achieves state-of-the-art JSD and voxel IoU performance on SemanticKITTI completion while maintaining strong performance with a single point transport step. On KITTI-360 unconditional generation, it also achieves the best Coverage (COV) among the compared methods.
Read the full abstract ▾
All authors Wenzhe He, Meng Wang, JiaWei Qian, Jinfeng Xu, Ying Liu, Ruihui Li
Computer Vision
FakeIDet3-DB: Refining Digital Attacks and Patch Extraction for Secure ID Benchmarking
Javier Muñoz-Haro, Andres Teruel, Ruben Tolosana et al. (+4 more)
Identity document (ID) authentication relies on the structural integrity of complex, high-frequency security patterns. However, advanced Generative AI models can now inject localized, high-fidelity manipulations, creating deceptive attacks that bypass standard verification. Training robust image forensic models to detect these anomalies is hindered by
privacy regulations, forcing reliance on synthetic templates lacking the intricate visual patterns of real IDs. To bridge this domain gap, we introduce FakeIDet3-DB, the first comprehensive database of digital manipulations on real, government-issued IDs. FakeIDet3-DB encompasses classical (e.g., copy-move) and Generative AI-driven manipulations (e.g., face-swapping, inpainting) enhanced with advanced image refinement procedures to suppress visual artifacts. In addition, to comply with strict data protection regulations (e.g., GDPR), we adopt a recently-proposed framework based on patches. In order to maximize forensic utility, we formulate
privacy-aware patch extraction from a real ID as a geometrically constrained image processing problem. We propose PACE, a Pseudo-Anonymized Contextual patch Extraction
algorithm, which leverages Integral Image mapping and distance-driven Non-Maximum Suppression (NMS). PACE efficiently contours anonymization masks that prevent Personally Identifiable Information (PII) leakage while maximizing semantic density in peri-censorship regions, yielding almost 5.2M patches extracted from more than 6.4K images from real/fake IDs. Furthermore, an extensive evaluation of the proposed FakeIDet3-DB is performed using state-of-the-art models, showcasing they all struggle to detect and locate attacks coming from generative and classic techniques (32.45% EER in detection and 83.48% AUC-ROC in localization).
Read the full abstract ▾
All authors Javier Muñoz-Haro, Andres Teruel, Ruben Tolosana, Daniel DeAlcala, Ruben Vera-Rodriguez, Aythami Morales, Julian Fierrez
Computer Vision
Understanding Knowledge Transfer Mechanism in Heterogeneous MLLM Fusion: A Simple Linear Approach
Yinghao Hou, Jiahe Fan, Yuanhao Pu et al. (+2 more)
Training-free fusion of heterogeneous
multimodal large
language models (MLLMs) provides a direct route for cross-scale capability transfer, yet improvements in aggregate performance do not reveal what a smaller model actually inherits. Existing studies are largely designed and evaluated on limited task sets or aggregate metrics; as evaluation expands to broader task collections, whether different capabilities can transfer across scales remains poorly understood. To investigate this question, we introduce Cross-Scale Directional Parameter Injection (CDPI), a simple linear probe to analyze cross-scale knowledge transfer during heterogeneous fusion. A local theoretical analysis indicates that knowledge transfer selectivity is determined at first order by capability-dependent responses to a shared injection direction, while second-order
curvature effects constrain the effective transfer regime. Across four Qwen3-VL model pairs and twelve
multimodal benchmarks, our experiments reveal a consistent pattern of selectivity: gains concentrate on
reasoning, particularly high-level
reasoning, whereas perception performance remains close to that of the original target model. Component-wise ablations further show that high-level
reasoning gains arise primarily from the
language model, while ratio analysis finds that positive selective transfer occurs mainly in the small-ratio regime. These findings recast cross-scale heterogeneous MLLM fusion as selective
language-side
reasoning transfer within a narrow, low-interference regime, rather than broad capability inheritance.
Read the full abstract ▾
All authors Yinghao Hou, Jiahe Fan, Yuanhao Pu, Zongyuan Chen, Hong Xie