Computer Vision
RACE-FPP: A Robust AI-assisted Characterisation Enhancement for Fringe Projection Profilometry
Osman Ali, Xiangjun Kong, Tibebe Yalew et al. (+2 more)
Fringe Projection Profilometry (FPP) requires precise system characterisation to achieve reliable three-dimensional (3D) reconstructions; however, characterisation accuracy strongly depends on robust checkerboard feature localisation, which can deteriorate under challenging imaging conditions such as lens blur and characterisation target orientations. Existing deep learning-based corner detectors are typically assessed using detection metrics and camera reprojection error alone, without considering their wider impact on projector characterisation, camera-projector stereo characterisation consistency, or overall measurement accuracy. In this work, we introduce a complete FPP characterisation pipeline that incorporates deep learning-based corner detection into the standard camera characterisation workflow. We also characterise the projector by sampling phase values at the centres of the white squares in the characterisation target. Rather than treating corner detection as an isolated task, the proposed framework explicitly analyses how localisation errors propagate throughout the entire FPP characterisation chain. Performance is evaluated using detection metrics (e.g., precision and recall), camera and projector reprojection errors, and the camera and projector stereo characterisation. Across a mixed
dataset of clean and degraded images, the camera reprojection error is reduced from 1.237 pixels to 0.259 pixels, while the projector reprojection error is reduced by roughly 50%. Dimensional evaluation of reconstructed artefacts shows improved
geometric accuracy compared with those resulting from the conventional pipeline. Overall, the findings indicate increased robustness of system-level characterisation under challenging imaging conditions, thereby enabling more reliable industrial FPP measurements.
Read the full abstract ▾
All authors Osman Ali, Xiangjun Kong, Tibebe Yalew, Waiel Elmadih, Samanta Piano
Computer Vision
MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos
Souptik Sen and Zahra Ahmadi
Audio-visual models improve egocentric action recognition by exploiting complementary cues, yet typically assume that both streams remain available at inference. Existing missing-modality methods operate on trimmed, single-event clips in which a stream is entirely present or absent, whereas real sensors fail and recover within long, untrimmed observations. We redefine egocentric modality missingness as temporally localized sensor outages within untrimmed, multi-event observations, with whole-clip absence as the limiting case. We introduce MacJEPA, a missing-modality-robust Masked-context query JEPA that recognizes visual actions and acoustic events from supplied interval queries over audio-visual context. Window-local modality dropout simulates these sensor outages during training. MacJEPA further repurposes masking in JEPA from a self-supervised pretext into a supervised robustness objective, aligning masked and clean latent representations of both
multimodal content
tokens and the task-conditioned queries. All objectives are optimized jointly with recognition in a single stage, requiring no test-time adaptation. Across Epic-Kitchens-100 and Epic-Sounds, a single checkpoint remains competitive under complete input and consistently surpasses published missing-modality baselines when either the dominant or auxiliary stream is removed. MacJEPA thus unifies strong full-input recognition with temporal missing-modality robustness in a single model operating on untrimmed multi-event
videos.
Read the full abstract ▾
Computer Vision
PIE-PS: Photometric Stereo from Physical Irradiance Event Streams
Xiangze Meng, Guangyu Li, Jing Li et al. (+4 more)
Event cameras record asynchronous log-image-irradiance changes with microsecond latency and high dynamic range. These properties are useful for photometric stereo under moving illumination, but raw events are sparse and depend on an unknown contrast threshold. We start from the event trigger model and derive a physical relation between adjacent events, light motion, and surface normals. This relation gives a direct physics-only solver, but the solver needs the threshold, enough events at each pixel, and independent per-pixel optimization. To address these limits, we introduce PIE-PS, a learning-based framework for dense surface normal reconstruction from raw event streams and known lighting. We form Physical Irradiance Events (PIEs) by pairing two adjacent events at the same pixel with their corresponding light directions. Each PIE provides a Physical Irradiance Event Feature (PIEF), defined as the signed event rate. PIEF does not require the unknown contrast threshold. To share spatial and temporal context across nearby PIEs, we introduce PIE-GNN, which treats each PIE as a graph node and encodes it with its light-pair geometry. Since the reliability of PIE observations can vary with local appearance, illumination geometry, and sensor noise, Reliability-Grading
Attention (RGA) predicts reliability weights to down-weight unreliable PIEs. Pixel aggregation then produces dense normals. Experiments on synthetic and real data show that PIE-PS outperforms prior event-based photometric stereo methods and the direct solver baseline.
Read the full abstract ▾
All authors Xiangze Meng, Guangyu Li, Jing Li, Di Mei, Songchen Ma, Mingkun Xu, Rui Ma
Computer Vision
View Matters: Keyframe-Guided Text-Driven 3D Gaussian Editing
Kaizhe Zhang, Yijie Zhou, Weizhan Zhang et al. (+3 more)
Text-driven 3D Gaussian editing commonly does not distinguish the editing reliability of rendered views, although different viewpoints provide supervision of substantially different quality. Views that clearly show the scene and match the edit instruction provide reliable guidance, while less informative views may weaken the edit when all views are treated equally. We present View Matters, a view-importance-aware framework that conducts editing around reliable keyframes. Keyframe Importance Estimation (KIE) identifies reliable views using
geometric visibility, semantic distinctiveness, and edit relevance. Keyframe-Guided Editing (KGE) then propagates their editing signals asymmetrically to non-keyframes without noisy reverse influence, while Importance-Aware Optimization (IAO) preserves this reliability preference during 3DGS optimization. Across 23 scene-
prompt pairs, View Matters achieves the highest average CLIP text-image similarity of 0.2822 and directional similarity of 0.2564 among the evaluated methods, with a four-minute editing time. Additional adjacent-view analysis indicates that the fidelity-oriented editing process maintains cross-view coherence.
Read the full abstract ▾
All authors Kaizhe Zhang, Yijie Zhou, Weizhan Zhang, Xuanyu Wang, Feng Lei, Sha Gong
Computer Vision · Cross-listed from cs.MA
Visual Orchestration Tax in Agentic VLM Pipelines: Auditing and Certifying Visual Evidence Reuse
Lingteng Zeng
Agentic VLM pipelines increasingly pass the same static visual evidence through multiple specialist
agents and tools. This design creates an orchestration-level redundancy mode: semantically unchanged images are repeatedly reconstructed as image-conditioned requests at the VLM API boundary. We call this phenomenon visual orchestration tax and develop a measurement-to-certification framework for visual evidence reuse in agentic VLM pipelines. The audit side defines $\mathrm{M1}_{\mathrm{trace}}$ to count raw visual-evidence touches and M2 to measure structural touch redundancy, with query-level distributions, bootstrap confidence intervals, and paired quality tests. Across SeeingEye and MAMMQA on chart, document, general-VQA, and multi-modal-QA tasks, audits reveal 66.8-75.6% visual-evidence touch redundancy, and every audited query exceeds the predefined gate. The certification side introduces SharedVisCache, a contract-aware evidence reuse hook keyed by image content, preprocessing fingerprint, and encoder assumptions. On SeeingEye, contract validation certifies 75.0-75.5% repeated touches as reusable while preserving 350/350 output strings and $\Delta\mathrm{M5}{=}0$. At the physical layer, certified hits reduce $F_{\mathrm{vision}}$ from 800 to 200 in ChartQA-200 trace replay and from 200 to 50 inside live SeeingEye translator-stage physical integration, preserving 800/800 replay strings and 200/200 integrated call outputs. The results position visual reuse as a measurable, behavior-preserving property of
agent orchestration and define an
agent-layer contract that makes backend prefix or
token reuse semantically interpretable.
Read the full abstract ▾
Computer Vision
The Failure Is in the Readout: Fine-Grained Emotion Recognition Benchmarks Measure Elicitation, Not Perception
Tobias Hallmen, Fabian Deuser, Robin-Nico Kampa et al. (+2 more)
Fine-grained emotion recognition supports therapy tools and social
robots, but it needs facial data, which raises
privacy and data-protection concerns. EmoNet-Face-HQ answers that with generated portraits, expert-rated over a $40$-category taxonomy far finer than the usual six to eight basic emotions. Under the protocol it ships with,
vision-
language models (VLMs) score poorly on that taxonomy, and the
benchmark concludes that a dedicated fine-tuned model is necessary: Empathic-Insight-Face (EIF; Small/Large). We show that off-the-shelf VLMs match or beat that fine-tuned model when the answer is not generated but read from the logits, as one binary query per category. We keep the
benchmark's images, taxonomy and ratings, and change only how the answer is read. Experts agree at $\kappa_w = 0.468$ on the five categories they measure most reliably. Generatively, no interval among eleven open-weight VLMs lies entirely above that anchor ($\kappa_w=0.268$-$0.486$). Under verification all eleven clear it, each of them significantly better at $\kappa_w=0.507$-$0.586$. Three also significantly beat EIF sitting at $\kappa_w = 0.551$ (Small; $0.534$ Large). The gain comes from the graded probability and not from asking a yes/no question: as a control, thresholding those same probabilities to yes/no costs 142% of the average gains and drops binarization below generative elicitation to $\kappa_w=0.254$-$0.423$. A replication on real photographs (FACES) is weaker and mixed: of the ten models that pass a validity gate, six gain, three are neutral to positive and one is negative, so the effect is not confined to synthetic data.
Read the full abstract ▾
All authors Tobias Hallmen, Fabian Deuser, Robin-Nico Kampa, Norbert Oswald, Elisabeth André
Computer Vision · Cross-listed from cs.CR
Rethinking Visual Provenance: Detection and Watermarking Across Direct Visual Generation and LLM-Driven Code Rendering
Zheng Gao, Xiaoyu Li, Zhicheng Bao et al. (+1 more)
AI systems create images and
videos with image/
video generation models or by writing code and graphics descriptions that are then rendered. These routes can produce similar visible artifacts but expose different representations, intervention points, and provenance evidence. We develop a production-centered framework that compares detection and watermarking across both routes. An explicit verification specification distinguishes passive inference, message recovery, and authenticated provenance. We organize image,
video, source-code, and rendering-aware watermarks by production stage. We examine the different requirements of generated images and
video, plots and SVG, programmable
video, and
agent-composed workflows. Documented Claude, OpenAI, and rendering-tool interfaces connect the framework to concrete systems. We pose ten scoped research questions on identifiability, observability, fair comparison across stages, recoverable payload, reconstruction, synchronization, composition, hybrid local contribution, and private production-event authentication. The result is a conceptual research agenda grounded in published methods, inspected interfaces, and elementary boundary examples. It reports no experiments and claims no new
theorems; its appendix results are elementary calculations, and documentation and source inspection establish interfaces, not empirical robustness.
Read the full abstract ▾
All authors Zheng Gao, Xiaoyu Li, Zhicheng Bao, Jiaojiao Jiang
Computer Vision · Cross-listed from cs.RO
VLA-ACL: Action-Consistent Visual Token Pruning for Efficient Vision-Language-Action Models
Owen Du, Yang Yue, Jie Zhang et al. (+3 more)
Vision-
Language-Action (VLA) models achieve strong robotic manipulation performance but incur high computational costs from processing long
token sequences at every control step, limiting real-time deployment. Visual
token pruning offers a direct solution, as visual patches dominate the input sequence and contain considerable redundancy. Existing approaches, however, either rely on indirect training-free heuristics, such as
attention scores and motion thresholds, or require costly fine-tuning of the base VLA model. We introduce VLA-ACL (Action Consistency Learning), which learns a lightweight visual
token pruning policy through action-level supervision while keeping the base VLA model entirely frozen. The training objective encourages actions produced from pruned visual contexts to remain consistent with the full-context teacher, with ground-truth actions as auxiliary supervision. This directly ties
token selection to its effect on the downstream control output. Experiments on LIBERO and real-world manipulation tasks show that VLA-ACL prunes up to 87.5% of visual
tokens while retaining competitive performance, reduces computation by up to 75%, and achieves a 1.5x inference speedup. These results establish a stronger performance-efficiency trade-off than existing frozen-VLA
pruning methods and demonstrate the value of action-level supervision for visual
token selection. Code is available at https://github.com/du-owen/VLA-ACL.
Read the full abstract ▾
All authors Owen Du, Yang Yue, Jie Zhang, Jiaqi Pi, Chi Bene Chen, Gao Huang
Computer Vision · Cross-listed from cs.LG
Mu-DisCoCat: A Variational Pipeline for Compositional Generalization on Quantum Processors
Mina Abbaszadeh, Matilda Karabina Moore, Raem Haq et al. (+2 more)
Achieving compositional concept generalization (CoCoGen), the ability to understand novel situations by recombining learned primitives, remains a fundamental challenge in artificial intelligence. Compositional semantic models such as Compositional Distributional Semantics (DisCoCat) offer solutions by generalising vectors to tensors, but suffer from scaling bottlenecks when learning the tensors. Mapping DisCoCat onto Variational
Quantum Circuits (VQCs) resolves this limitation for text, yet the methodology has not been expanded to
multimodal situations such as the ones involved in CoCoGen. This paper introduces Mu-DisCoCat: a
multimodal variational
quantum learning framework for DisCoCat that achieves CoCoGen. The framework first learns stable object representations from single-object image-text pairs, then fixes these and uses them to learn the relations between them in multi-object situations. In classical simulations, the model used Uhlmann state fidelity to compute the overlap between the
multimodal circuit representations and achieved higher relational OOD accuracy than the evaluated CLIP baseline. Its deployment was evaluated using the destructive SWAP test across noisy
quantum emulators, including a range of IBM fake backends, IQM FakeAphrodite, and the IBM Marrakesh
quantum processor. Despite real-world device noise, the hardware-executed models maintained a strong positive correlation with simulated fidelities, reliably distinguishing unseen similar and dissimilar pairs. Our work establishes a framework for executing CoCoGen on VQCs, demonstrating a viable use case for near-term
quantum hardware.
Read the full abstract ▾
All authors Mina Abbaszadeh, Matilda Karabina Moore, Raem Haq, Martha Lewis, Mehrnoosh Sadrzadeh
Computer Vision
Supermarket Product Detection and Recognition: Utilizing Deep Learning with Rectified Imagery
Mayank Sah and Jimson Mathew
Product Identification has sprung up to become one of the most challenging problems in the automation of the retail industry. With the new industry 5.0 standards, automated inventory management, and catalog creation tasks are vitally important. Object identification models have emerged as a viable answer with their unprecedented identification and localization accuracy. However, the close-knit rack design of supermarkets generates the problem of angle variation in capturing images. The angle-variant densely packed images(a single image contains many objects) become overwhelming for these models alone. In this paper, we try to supplement object detection models with traditional Hough transform (HT) and homogeneous estimation concepts. We study the effect of rectified images using homography estimation and hough transform and their limitations on the problem of grocery identification. We make a case for creating a new
dataset to test the effects of such rectification and produce analytical results on different scenarios of angle variation and object densities per image. Extensive experiments on different object detection models suggest that image rectification of angled images improves the detection accuracy of grocery products in images. The results also highlight the limitation of rectification on the angle of image capture and the object density of the image.
Read the full abstract ▾
Computer Vision
Beyond Training from Scratch: Foundation Models for Data-Efficient and Generalizable Cardiac MRI Reconstruction
Anam Hashmi, Mayug Maniparambil, Julia Dietlmeier et al. (+2 more)
Cardiac
magnetic resonance imaging reconstruction aims to recover high-quality images from undersampled acquisitions, enabling faster scans while preserving diagnostic fidelity. Recent reconstruction methods are typically trained from scratch and often require large amounts of task-specific data, limiting their robustness under data scarcity and distribution shifts. In this work, we investigate whether pretrained
vision foundation models can serve as effective priors for accelerated cardiac MRI reconstruction. We propose a reconstruction framework that integrates frozen and parameter-efficiently adapted visual encoders, including CLIP, BiomedCLIP, and DINOv2, within a
transformer-based reconstruction architecture. Extensive experiments on the CMRxRecon2023 and CMRxRecon2024
benchmarks demonstrate that pretrained representations consistently outperform a
transformer trained from scratch across multiple acceleration factors. We further evaluate performance under limited supervision and cross-
dataset transfer, showing that foundation models provide superior data efficiency and generalization. While frozen representations are particularly effective in extreme low-data regimes, Low-Rank Adaptation (LoRA) yields additional gains when moderate amounts of training data are available. Among the evaluated backbones, DINOv2 achieves the strongest overall performance. These findings highlight the potential of
vision foundation models as robust and transferable priors for cardiac MRI reconstruction.
Read the full abstract ▾
All authors Anam Hashmi, Mayug Maniparambil, Julia Dietlmeier, Kathleen M. Curran, Noel E. O'Connor
Computer Vision
Multi-Dataset Diagnostic Utility of Clinical Visual Concepts in AI Systems for Dermatology
Linda Wermelinger, Simone Lionetti, Fabian Gröger et al. (+6 more)
The clinical integration of AI systems in digital dermatology relies heavily on human trust. Clinically interpretable visual concepts can act as intermediate representations enhancing trust and reliability. However, research in this domain is currently limited by scattered, heterogeneous
dataset annotations. In this work, we introduce SkinLex, a harmonized
dataset of 48 clinical morphological attributes across four public
datasets (SkinCon, DermaCon-IN, MM-Skin, and PASSION) for a total of 20,411 records. Supervised nine-partition classification of skin conditions shows that limiting features to specific visual groups, like shapes or colors alone, reduces diagnostic accuracy. Bootstrapped backward elimination reveals that the set of 48 visual concepts has some degree of redundancy for algorithmic nine-partition diagnosis on the examined
dataset. This demonstrates that coarse diagnosis on the selected
dataset requires a relatively small but varied combination of clinical concepts, and motivates further research to improve concept taxonomy. Results can be translated into clinical benefits by reducing inputs for concept-based models, improving efficiency for annotation and modeling, and further enhancing interpretability. Code and
prompt templates are available at https://github.com/Digital-Dermatology/SkinLex.
Read the full abstract ▾
All authors Linda Wermelinger, Simone Lionetti, Fabian Gröger, Nipun Ranasekara, Philippe Gottfrois, Ludovic Amruthalingam, Labelling Consortium, Marc Pouly, Alexander A. Navarini
Computer Vision · Cross-listed from cs.LG
Optimization Encoders: Rethinking Second-Order Meta-Learning for Neural Fields
Rudolf L.M. van Herten, Soufiane Ben Haddou, Rachit Saluja et al. (+1 more)
Conditional neural fields represent signals continuously, but their effectiveness depends on how the conditional latent representations are inferred from observed data. In meta-learning, this encoding occurs through gradient updates induced by the decoder, tying representation learning directly to decoder design. We formalize this connection by interpreting latent optimization as an optimization encoder, unifying the roles of second-order differentiation, latent parameterization, and task supervision. This concept enables second-order meta-learning for end-to-end training of the encoding procedure alongside the decoder, and clarifies which learning pathway first-order approximations discard. Guided by this view, we introduce Attentive Latent Fields (MetaLF), an equivariant
transformer-based neural field that contextualizes a latent pointcloud through self-
attention. These interactions shape both field predictions and the updates that construct their representation, allowing local observations to inform coherent non-local structure. Disentangling the inner encoding objective from outer task supervision unifies reconstruction, classification, and segmentation within an end-to-end meta-learning framework, using reconstruction-only latent adaptation at test time. Controlled experiments on
polynomial fields link latent coordination to lower effective rank and stronger
alignment with the underlying function space. Across image and 3D shape reconstruction, MetaLF improves fidelity within three to five gradient updates, while supporting semantic prediction across images, shapes, and volumes. Together, these findings position the optimization encoder perspective as a unified basis for designing neural fields around how representations are constructed, coordinated, and used.
Read the full abstract ▾
All authors Rudolf L.M. van Herten, Soufiane Ben Haddou, Rachit Saluja, Johannes C. Paetzold
Computer Vision
Two Halves are More than One: Phase-wise Velocity Distillation for Fast and High-Quality Image Generation
Zhen Guo, Rongyuan Wu, Qiaosi Yi et al. (+3 more)
Recent
diffusion-based image generation backbones have grown substantially in scale, making the network inference cost increase rapidly. While
diffusion distillation techniques can reduce the number of inference steps, high-quality image generation within a single full-backbone-forward compute budget remains challenging. Existing one-step methods typically allocate this budget to a single evaluation of a monolithic student. However, approximating the heterogeneous coarse-to-fine transport with a single monolithic mapping is difficult and often leads to over-smoothed outputs. To address this issue, we propose Phase-wise Velocity
Distillation (PVD), which partitions the generation timeline into a coarse and a fine phase, and models the transition within each phase via the average velocity. A dedicated half-sized expert is assigned to each phase, decoupling structural composition from detail refinement while keeping the cumulative computation equivalent to one full-backbone forward pass. We show that the use of two half-sized phase-specific experts outperforms a single full-size monolithic student. On class-conditional image generation, PVD achieves an FID of 1.48 on ImageNet 256 x 256. On more complex text-to-image (T2I) tasks, PVD-distilled models (Stable
Diffusion 3.5-Medium, FLUX.1-dev, Qwen-Image) produce results competitive with their multi-step teachers, significantly outperforming prior
distillation methods. Moreover, across the evaluated T2I backbones, PVD reduces active parameters by 49.10-50.89% and peak VRAM by 45.76-48.36% compared to the corresponding teachers. Source code and distilled models are available at https://github.com/PolyU-VCLab/PVD.
Read the full abstract ▾
All authors Zhen Guo, Rongyuan Wu, Qiaosi Yi, Chenxi Xie, Xinyu Wei, Lei Zhang
Computer Vision
PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation
Guo Tang and Yongtao Wang
Realistic physical interaction is a cornerstone of embodied intelligence, yet collecting paired visual–tactile data remains costly. Visual-to-tactile synthesis offers a promising approach to augmenting such data, but learning this mapping is complicated by the gap between visual appearance and contact-related material properties, as well as spatial misalignment in paired observations. To address these challenges, we present PhysTacGen, a visual-to-optical-tactile image generation framework that integrates material-aware descriptions with
geometric conditioning. First, we introduce Group Tactile Policy Optimization (GTPO), a reinforcement learning strategy that refines a
vision–
language model to generate structured material descriptions using task-specific rewards. Second, we combine DINOv2-based pair curation with monocular relative-depth estimation to select training pairs and provide
geometric priors. Finally, an SDXL ControlNet synthesizes optical tactile images conditioned on RGB, relative depth, and GTPO-generated text. Experiments on curated SSVTP data demonstrate improved structural similarity over the compared baselines, while a blinded user study shows a preference for GTPO-generated descriptions. Generated tactile inputs also improve performance on an attribute-derived force-coefficient prediction proxy. Together, these results demonstrate the effectiveness of PhysTacGen for optical tactile image synthesis and its utility in the evaluated downstream task.The code will be available at https://github.com/VDIGPKU/PhysTacGen.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic
Sin-Han Yang, Shih-Cheng Huang, Chieh-Yen Lin et al. (+3 more)
Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tasks, most existing merging methods use an additional
dataset to find the coefficients for the best linear combination of task-specific weight updates. However, we identify an implicit regularization in this standard practice: searching over coefficients restricts the candidate models to a subspace spanned by task-specific weight updates. In this work, we investigate whether this regularization is actually useful. Surprisingly, empirical results show that optimizing merged-model weights without this regularization significantly boosts the performance of common merging methods across multiple architectures, domains, and even in an extremely data-limited scenario where only one instance is available per class. Moreover, directly optimizing the pretrained model weights even outperforms some existing merging methods. Analysis shows that better multi-task weights exist outside the subspace and can be found using multiple methods. We study different strategies for using the additional
dataset, discussing their practical use and implications for model merging. Overall, this work calls for revisiting the existing model-merging pipeline, motivating a broader exploration of the weight space and a reconsideration of the implicit regularization induced by task arithmetic.
Read the full abstract ▾
All authors Sin-Han Yang, Shih-Cheng Huang, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, Hung-yi Lee
Computer Vision
VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs
Yuan Feng, Qize Yang, Ruizhe Chen et al. (+9 more)
Multimodal large
language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch
tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Downsampling sacrifices this detail, while existing
token pruning and adaptive approaches remain limited in content-adaptive granularity, task generalization, and integration with modern MLLMs and serving infrastructure. Overcoming these limitations calls for foundation models that learn, end to end, where-and at what granularity-to allocate visual representations, a native capability we term elastic visual representation weaving. We introduce VisionWeave, establishing this capability in frontier-level MLLMs through large-scale training. It combines two components: a gated spatial pooler constructs coarse-grained representations alongside native fine-grained representations within a shared MRoPE coordinate, while a granularity router learns their content-adaptive allocation. Through self-
distillation alone, we validate this capability on Qwen3.5-4B and scale to Qwen3.8-27B with over 30K A100 GPU-hours. Based on Qwen3.8-27B, VisionWeave adaptively adjusts
token savings to visual content, saving 43.0%
tokens on average while retaining 98.9% native performance across eight
benchmarks, versus only 88% performance preserved for
token pruning baselines with a fixed 50% savings target. Extensive evaluations confirm robust efficiency-quality trade-offs across diverse tasks, resolutions and
video frames. When deployed on SGLang serving engine, our method achieves a 2.3x throughput gain while reducing mean TTFT by 54.4% and mean TPOT by 60.6%. Together, we believe these results position elastic visual weaving as a promising capability for next-generation
multimodal models.
Read the full abstract ▾
All authors Yuan Feng, Qize Yang, Ruizhe Chen, Sibo Song, Haolin He, Muzhi Zhu, Zihan Liu, Yunfei Chu, Xize Cheng, Yuxuan Wang, Jin Xu, Xike Xie
Computer Vision
Decide Before You Look: Learning Which Retrieved Memories Deserve Pixels
Youxing LI
Multimodal assistants answer questions from long-term
memories that contain images. After
retrieval, each retrieved image reaches the answering model either as pixels, at about a thousand visual
tokens per image, or as a stored text proxy that often misses the detail the question asks about. We find that the benefit of pixels usually comes from one or two retrieved
memories, and that it can be predicted before the answering model runs, without reading any full-resolution image. In PixelTriage, a plug-in placed after
retrieval, a small model that does not generate text reads the dialogue, a short note and a thumbnail of each retrieved
memory and predicts how much its pixels would add. It is trained on synthetic
memory episodes labeled by a frozen 27B model that answers each question with and without each
memory's pixels. With a 7B answering model, PixelTriage lies on the accuracy–cost frontier of M$^3$Exam, DMV and MemEye and uses 11–23% of the visual
tokens without a significant loss of accuracy. On DMV it answers 2.9 times faster than opening all images. It outperforms
retrieval order and uniform down-sizing at equal budgets and transfers to other
memory systems and to a 397B answering model.
Read the full abstract ▾
Computer Vision
M3SunAgent: Monocular 3D Spatial Understanding Agent for Metric Depth Estimation and 3D Visual Grounding
Jinsong Zhang, Kejun Wu, Ming Zhu et al. (+3 more)
Monocular metric depth estimation and 3D visual grounding represent the two complementary cornerstones of monocular 3D spatial understanding (M3Sun), from which the fundamental 3D spatial information required by M3Sun can be acquired. However, these complementary tasks are generally conducted by separate frameworks, which pose challenges of inflexible and unaligned spatial information access for embodied intelligence systems. In this paper, we propose a unified
agent for monocular 3D spatial understanding (M3SunAgent) that leverages a large
language model (LLM) as a task planner for spatial visual programming, which flexibly generate structured programs and coordinate tools. For instance-level metric depth estimation task, M3SunAgent invokes an object detector tool to locate the target, estimates depth at selected points with a depth estimation tool, and aggregates these predictions into an instance-level depth estimate. We also construct the M3Sun Instance (M3SI)
dataset, a
benchmark with 2,910 samples for evaluation. For monocular 3D visual grounding task, M3SunAgent uses a
vision-
language model (VLM) tool to locate the target and output basic spatial attributes, then combines back-projection tool with a dimension-lifting tool to predict its 3D bounding box. Experimental results demonstrate the superior performance of M3SunAgent. Specifically, in evaluations of instance-level monocular metric depth estimation, M3SunAgent achieves the best performance among all compared models, 52.61% of predicted instances are distributed below depth error 0.25 ($\delta < 0.25$). In evaluations of monocular 3D visual grounding, M3SunAgent demonstrates overall competitive performance than
vision and VLM models, reaching a 3D mean intersection over union (mIoU) of 41.73% and exceeding the state-of-the-art MonoVLM model by 3.62%.
Read the full abstract ▾
All authors Jinsong Zhang, Kejun Wu, Ming Zhu, Renjie Qiao, Chengtao Cai, Zhengguo Li
Computer Vision
EmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation
Yikai Qin, Yifei Deng, Mingjian Liang et al. (+13 more)
Scaling robotic foundation models requires diverse training data and reliable evaluation environments. Simulation offers a scalable solution, yet existing generation pipelines remain constrained by predefined assets and skills, a disconnect between scene generation and task generation, and limited support for complex embodiments and physics. We introduce EmbodiedSmith, a framework for scalable embodied data generation through recursive self-improvement (RSI). EmbodiedSmith unifies asset, scene, and task generation in a pipeline that supports autonomous creation and
language-driven customization. Its core is an agentic refinement loop: scene generation anticipates downstream task requirements, while task generation guides targeted scene edits, allowing scenes and tasks to iteratively improve one another. This joint refinement improves task generation success, including for long-horizon tasks. The framework further supports mobile manipulators, humanoids, and dexterous hands, as well as interactions involving deformable objects and fluids, broadening the range of behaviors and physical phenomena represented in generated data. Together, these capabilities provide a flexible simulation engine for both
robot pretraining and evaluation. Extensive experiments validate the quality, diversity, and generation efficiency of the resulting data, while downstream policy experiments demonstrate that increased data diversity improves generalization.
Read the full abstract ▾
All authors Yikai Qin, Yifei Deng, Mingjian Liang, Wenxuan Song, Zepeng Lin, Zhiyi Jiang, Jiajun Fu, Qiao Sun, Huashuo Lei, Xicheng Gong, Jiayi Chen, Han Zhao, Shuanghao Bai, Pengxiang Ding, Pengwei Wang, Haoang Li
Computer Vision
DensiTok: Making Feed-Forward 3D Gaussian Splatting See More Views Than It Is Given
Minhyeok Lee, Jungho Lee, Minseok Kang et al. (+3 more)
Feed-forward 3D Gaussian Splatting (3DGS) reconstructs a scene in a single forward pass, replacing per-scene optimization with a network trained across many scenes. Its quality, however, degrades sharply as the number of input images drops. The bottleneck is upstream of the reconstruction heads: from a few unposed views, the internal representation they read carries no evidence for unobserved regions, leaving holes, floaters, and blur. The common remedy supplies that evidence as pixels, synthesizing extra views with an image or
video generator and re-encoding them, which is costly and not 3D-consistent by construction. We instead densify the evidence itself. We present DensiTok, a plug-in module for pretrained feed-forward 3DGS models that densifies their internal geometry
tokens directly, making a frozen backbone behave as though it had observed many more views than it was given. DensiTok compresses those
tokens into a compact latent space, completes the latents of the unobserved viewpoints in a single flow-matching step conditioned on camera geometry, and decodes them back into
tokens that the original reconstruction heads. The same module design can be integrated into different pretrained predictors while keeping each backbone and its reconstruction heads frozen. Completion in a low-dimensional latent space requires no image synthesis or additional encoder passes. Across three pretrained backbones and two
benchmarks, DensiTok consistently improves sparse-view reconstruction and recovers much of the gap to dense-view reconstruction.
Read the full abstract ▾
All authors Minhyeok Lee, Jungho Lee, Minseok Kang, Heeseung Choi, Ig-Jae Kim, Sangyoun Lee
Computer Vision
Revisiting Numerical Forecasting Models for Language-Based Trajectory Prediction
JunGyu Lee, Inhwan Bae and Hae-Gon Jeon
Language-based trajectory predictors represent coordinates as discrete
tokens and learn auxiliary tasks such as destination and group
reasoning. This formulation enables the model to capture behavioral intent and social context beyond coordinate dynamics alone. However,
token-level objectives provide only indirect guidance for continuous coordinate-space dynamics. To address this limitation, we introduce MoRE (Mixture of Reward Experts), a refinement framework that transfers numerical forecasting priors into a pretrained
language-based predictor through reinforcement learning. Five frozen numerical predictors provide complementary coordinate-level knowledge of motion and interactions. Their predictions are converted into expert rewards and combined through an uncertainty-weighted consensus that penalizes disagreement. A ground-truth reward anchors the prediction to the target trajectory. To focus refinement on difficult cases, MoRE refines the policy using the top 1% of training samples ranked by predictive
entropy. Expert predictions are computed once and cached before PPO training, so the experts are not run during policy updates or inference. In this way, MoRE combines the contextual modeling of the
language-based predictor with coordinate-level feedback from numerical experts. On ETH-UCY, MoRE reduces ADE from 0.22 to 0.20 m and FDE from 0.32 to 0.29 m. Relative to the base policy, ADE decreases by 17.9% on SDD and 12.7% on NBA. On ETH-UCY, MoRE also reduces collision rates and better matches ground-truth pedestrian spacing, without increasing measured inference
memory or latency. The project page is available at https://jungyu0413.github.io/MoRE/.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.GR
UltraDiff: Differentiable Ray Tracing in Ultrasound for Shape Optimization
Felix Duelmer, Magdalena Wysocki, Nassir Navab et al. (+1 more)
Physically-based differentiable rendering enables gradient-based optimization of scene parameters by matching rendered images to measurements, but has so far mainly focused on light transport. We extend this paradigm to medical ultrasound, where image formation resembles transient rendering: echoes are binned by time-of-flight rather than projected onto an image plane. We present UltraDiff, a modular framework for differentiable ultrasound ray tracing. UltraDiff formulates ultrasound image formation as a path-space integral, gated by travel time between the transducer and tissue interfaces, and derives a Monte Carlo
estimator of both the forward model and its gradients with respect to scene parameters. We demonstrate this on an inverse geometry estimation: starting from a sphere, an SDF is optimized until simulated echoes match measured ones, recovering vertebral surfaces from simulated B-mode sweeps and from a real robotic acquisition of a spine phantom. Unlike state-of-the-art ultrasound shape reconstruction methods, which rely on pre-segmented images, our approach operates unsupervised on B-mode images through analysis-by-synthesis, while achieving competitive
geometric accuracy. Implemented on top of Mitsuba 3, UltraDiff brings differentiable path tracing to a new sensing modality and provides a foundation for inverse problems in acoustic imaging.
Read the full abstract ▾
All authors Felix Duelmer, Magdalena Wysocki, Nassir Navab, Mohammad Farid Azampour
Computer Vision
CCDF: A Benchmark Dataset for Deepfake Detection in Real-World Surveillance Footage
Baptiste Chopin, Thomas Swearingen, Arun Ross et al. (+2 more)
Due to rapid advances in Generative AI, commercial
video generation tools can be used to produce fabricated surveillance footage that can fool both human viewers and automated synthetic
video detectors. Since these tools are so widely accessible, a malicious user can create a harmful
video clip at minimal cost. The production and dissemination of such
videos in high-stakes settings, such as crime reporting and elections, can misdirect emergency response efforts or distort political discourse. Existing deepfake
video datasets, used by the research community to develop deepfake detection
algorithms, exhibit two limitations: (1) they emphasize benign web content rather than footage of possibly malicious activity, and (2) they rely on older or open-source generators that do not represent recent advances in generative systems. We assemble CCtv DeepFakes (CCDF), a
video deepfake
dataset, to address both gaps. CCDF contains 1840
videos (460 real and 1380 generated) spanning 16 crime and accident categories, with generated content produced using three leading commercial systems: Grok Imagine, Google VEO 3.1, and OpenAI Sora 2. CCDF is a highly realistic, small-scale, manually annotated
dataset targeting evaluation of detection models. We release three versions of the
dataset: the raw generated data, a cleaned version in which
video metadata are standardized between real and synthetic samples to prevent detectors from exploiting trivial cues, and an altered version simulating low-effort post-processing attacks. We evaluate CCDF with ten recent state-of-the-art detectors covering different detection approaches. Our results suggest that these approaches do not reliably distinguish CCDF's generated
videos from real ones, despite their strong reported performance on existing
datasets. These results further confirm that existing
datasets are not well-suited to evaluating certain threats.
Read the full abstract ▾
All authors Baptiste Chopin, Thomas Swearingen, Arun Ross, Antitza Dantcheva, Christian Rathgeb
Computer Vision
Jinghao Xu, Zhenhua Guo, Xiaofeng Zhu et al. (+1 more)
Dynamic
multimodal learning aims to learn robust representations by adaptively modeling information discrepancies across modalities. However, existing methods still suffer from two limitations: (i) static cross-modal
alignment strategies usually impose uniform constraints on all samples while overlooking sample-wise variations, potentially leading to unreasonable over-
alignment; and (ii) confidence- or uncertainty-aware fusion methods often fail to adequately account for feature magnitude and confidence differences across modalities. For modality pairs with significant feature magnitude differences or small confidence gaps, it might be unreliable to strictly align fusion weights according to confidence. To address these issues, we propose an
Alignment- and Calibration-driven
Multimodal Learning framework (ACML). Specifically, ACML incorporates a dynamic cross-modal triplet
alignment module, which enforces strong semantic consistency for high-confidence positive pairs while encouraging diverse representation learning between high- and low-confidence positive pairs according to their confidence gaps. Additionally, ACML introduces a difference-aware
attention calibration strategy that adaptively adjusts
attention regularization based on feature magnitude and confidence differences across modalities, thereby mitigating biases caused by unreasonable fusion constraints. Extensive experiments on multiple
multimodal benchmark datasets demonstrate that ACML consistently achieves superior performance and robustness over recent state-of-the-art methods.
Read the full abstract ▾
All authors Jinghao Xu, Zhenhua Guo, Xiaofeng Zhu, Xiaoshuang Shi
Computer Vision
TF-PRVR: Training-Free Partially Relevant Video Retrieval
Giyeol Kim and Chanho Eom
Partially Relevant
Video Retrieval (PRVR) aims to retrieve untrimmed
videos containing moments relevant to a given text query. Despite recent progress, existing PRVR methods suffer from two key limitations: a fixed
video decomposition scheme that causes semantic dilution, and source-domain overfitting induced by task-specific training. In this paper, we propose TF-PRVR, the first training-free framework for PRVR. TF-PRVR leverages frozen
vision-
language features to construct
video-specific hierarchical representations. It derives temporal semantic signals from frame-level features and applies frequency-based multi-scale analysis to identify adaptive temporal boundaries, producing hierarchical segments with coherent event-level semantics. Built on these segments, TF-PRVR constructs a unified multi-scale graph and propagates query relevance across temporally and semantically related nodes. A moment-aware scoring strategy then aggregates temporally aligned relevance across scales, emphasizing consistently supported moments while suppressing isolated false responses. Without task-specific training, TF-PRVR preserves the general-purpose
alignment capability of pre-trained
vision-
language models and avoids
dataset-specific overfitting. Extensive experiments demonstrate consistent performance across
datasets with diverse visual and temporal characteristics, suggesting a practical direction for training-free PRVR.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
OpenWAM: An Open Framework for Composable World-Action Models
Heng Yu, David D. Yuan, Juze Zhang et al. (+7 more)
World-action models (WAMs) couple future prediction with
robot control, yet existing systems often vary the
video backbone, interaction structure, supervision, and inference procedure simultaneously, making their design choices difficult to compare. We introduce OPENWAM, an open world-action modeling framework built around a common causal
robot-
video foundation and configurable
video-action interaction. Starting from Wan2.2-5B, we perform causal
robot-
video pretraining on over 10,000 hours of
video, then integrate an action expert through a shared Mixture-of-
Transformers architecture that supports joint,
video-then-action, action-then-
video, and decoupled generation. OPENWAM achieves high success rates on four LIBERO suites and real-world bimanual tasks;
robot-
video training with causal adaptation improves VTA success on LIBERO-Long from 68.4% to 97.8%. The same configurable architecture naturally extends to inverse and forward dynamics, allowing us to study how counterfactual transitions improve independently trained dynamics models beyond demonstrations alone. When only the
video predictor is adapted to a new task, a frozen local-context inverse dynamics model trained on counterfactual data and demonstrations achieves 84.0% mean success across four held-out LIBERO-90 tasks, compared with 47.0% for a full-context inverse model and 21.5% for a local-context model trained only on demonstrations. For forward dynamics, counterfactual supervision reduces RGB prediction error by 34.5% and raises outcome identification from 21.1% to 71.3% among 16 same-state outcomes. OPENWAM provides a common testbed for comparing WAM interaction designs and for studying dynamics learning from
video data beyond successful demonstrations.
Read the full abstract ▾
All authors Heng Yu, David D. Yuan, Juze Zhang, Changan Chen, Yao Feng, Michelle Baldonado, Steve Cousins, Li Fei-Fei, Jiajun Wu, Ehsan Adeli
Computer Vision
Can We Model the Artifacts Explicitly? Disentangle Artifacts via Pairwise Edit Relations for Image Manipulation Localization
Xuekang Zhu, Kaiwen Feng, Ruifeng Wang et al. (+10 more)
Image Manipulation Localization (IML) is commonly formulated as a fully supervised learning task that estimates the optimal manipulation mask $y$ for a given image $x$. In this work, we first reveal the latent nature of artifacts and thus reinterpret IML as a latent-variable problem, $P(y|x)=\int P(y|z)\,P(z|x)\,dz$, where $z$ denotes the artifacts. Following this interpretation, we pinpoint the cause for the current IML models' insufficiency as their implicit artifacts modeling strategy, highlighting the necessity of modeling $z$ in an explicit manner. Without direct labels, feature disentanglement is the most appropriate solution for this explicit modeling. Accordingly, we propose a two-stage learning paradigm with the Pairwise Artifacts Learning (PAL) and Standard Localization (SL) phases to estimate $P(z|x)$ and $P(y|z)$ via edit relations. To support our edit-relation-based learning, we further curate EditGroup-45K, a source-anchored
dataset organized into edit groups for pair construction. Extensive experiments show that our PAL paradigm yields consistent improvements across diverse IML architectures, and empirical analyses further verify that PAL does capture artifacts explicitly through feature disentanglement. Code and
dataset are available at https://github.com/venus-guangjian/PAL
Read the full abstract ▾
All authors Xuekang Zhu, Kaiwen Feng, Ruifeng Wang, Xiwen Wang, Xiaochen Ma, Bo Du, Changjiang Jiang, Chenfan Qu, Songyu Ye, Xia Du, Wentao Feng, Jian Liu, Ji-Zhe Zhou
Computer Vision
Multimodal Knowledge Distillation for Gastric Adenocarcinoma Classification from Whole-Slide Images
Shrihari Dumbre and Bikash Santra
Gastric adenocarcinoma (GA) is a leading cause of cancer-related mortality worldwide, and accurate histopathological subtype classification from whole-slide images (WSIs) is essential for effective treatment planning. While
multimodal approaches that integrate pathology report text with WSIs can improve classification, existing methods often depend on computationally expensive
transformer architectures and large
language models. We propose a
multimodal knowledge
distillation (MKD) framework that combines a pretrained WSI image encoder and a clinical text encoder using Low-Rank
Multimodal Fusion (LMF) to efficiently model cross-modal interactions during training. Each WSI is represented as a bag of patches paired with a slide-level diagnostic caption. The teacher model learns fused image-text representations for subtype classification, while the student model distills this knowledge to enable accurate image-only inference. We evaluate our method on the PatchGastric
benchmark dataset and achieve at least 3.35% higher mean accuracy than state-of-the-art approaches, without relying on
transformer-based fusion, multi-task learning, or large
language models. The source code is available at https://github.com/helomelo1/MKD-LMF.
Read the full abstract ▾
Computer Vision
Diverse Motion Customization via Control-based Dynamic Optimization
Youngyoon Choi, Kihyun Kim, Jeongwoo Shin et al. (+1 more)
Despite recent advances in
video generation, motion customization remains challenging due to content leakage, where appearance attributes from the reference
video unintentionally propagate into the generated output. We identify this issue as a consequence of the generative process collapsing toward the reference
video, which arises from formulating the learning objective as a direct regression on the reference. To address this, we propose Control-based Motion Customization (CMC), a principled training framework that is structurally robust to content leakage. Our key idea is to steer generative dynamics toward desired motion while avoiding collapse toward the reference
video, which we formalize using
Stochastic Optimal Control (SOC). Under this formulation, customized
videos acquire the target motion yet remain within the pre-trained model's
prompt-conditional distribution, where appearance is determined by the text
prompt rather than the reference
video. Furthermore, to improve efficiency, we tailor the SOC formulation to motion customization by eliminating the need for an explicit reward and introducing a timestep-adaptive motion cost that focuses only on early generative stages, accelerating training by 2.5 times. Extensive experiments demonstrate that CMC effectively mitigates content leakage and achieves competitive motion fidelity while preserving the diversity of the base model across diverse scenarios.
Read the full abstract ▾
All authors Youngyoon Choi, Kihyun Kim, Jeongwoo Shin, Joonseok Lee