Computer Vision
Test-Time Scaling in Multimodal Foundation Models: A Comprehensive Survey of Generation and Reasoning
Cong Wan, Ying He, Zhongzhan Huang et al. (+1 more)
Test-time Scaling (TTS) has emerged as a pivotal research direction for enhancing model performance by dynamically allocating computational resources during inference. Recent advancements have adapted this paradigm to
Multimodal Foundation Models (MFMs), unlocking their potential in
multimodal reasoning and generation. Despite rapid progress, the field lacks a systematic survey and unified theoretical framework to delineate the developmental landscape of
multimodal TTS. To bridge this gap, we present the first comprehensive review of TTS research for MFMs, proposing a unified taxonomic framework that categorizes existing methodologies into three distinct strategies: sampling-based, feedback-based, and search-based approaches. We further summarize representative applications and
benchmarks commonly utilized to evaluate
multimodal TTS capabilities in generation and
reasoning tasks. Finally, this survey discusses open challenges and outlines future research directions, providing a systematic roadmap for subsequent studies in this rapidly evolving field.
Read the full abstract ▾
All authors Cong Wan, Ying He, Zhongzhan Huang, Hefeng Wu
Computer Vision
SegmentAnyTreeV2: Scaling Transformer-Based Tree Instance Segmentation Across Sensors, Platforms, and Forests
Maciej Wielgosz, Stefano Puliti and Rasmus Astrup
We present SegmentAnyTreeV2, a sensor- and platform-agnostic framework for semantic and instance
segmentation of forest point clouds. The model combines a serialization-based Point
Transformer v3 backbone with a lightweight semantic head and a tree-focused cross-
attention mask decoder. Semantic predictions restrict instance decoding to tree-class voxels, while instance-aware query initialization, one-to-many seed supervision, and asymmetric mask scoring improve separation in dense and structurally complex stands. We further introduce FOR-instance v3, an expanded
benchmark comprising 427 scenes and 26,496 annotated trees across diverse biomes, forest structures, and
LiDAR platforms. On the FOR-instanceV2 test split, SegmentAnyTreeV2 achieves 90.5% precision, 80.2% recall, 85.0% F1, 90.7% coverage, and 87.6% semantic mIoU, outperforming previous learning-based methods in both instance detection and mask completeness. Zero-shot evaluation on independent sites further demonstrates strong cross-domain generalization.
Read the full abstract ▾
Computer Vision
Empowering Feed-Forward Reconstruction Models with Metric Scale via Satellite Images
Xianghui Ze, Yongjian Luo, Mengjun Chao et al. (+3 more)
Feed-forward 3D reconstruction models have recently shown strong generalization across diverse scenes, yet most of them recover geometry only up to an unknown global scale. This scale ambiguity limits their use in applications that require metric understanding of the environment. Existing metric reconstruction methods commonly rely on large-scale metric annotations or accurate camera calibration, both of which are costly or unreliable in many real-world settings. We propose a satellite-guided framework for resolving scale ambiguity in feed-forward 3D reconstruction. The key idea is to use readily available satellite imagery as a global metric reference. Given a coarse camera pose, our method retrieves a local satellite patch and integrates it with a feed-forward reconstruction backbone through bidirectional cross-view interaction. By enforcing consistency between the reconstructed scene and the satellite reference, the model infers absolute scale, refines scene geometry, and estimates camera pose in a metric coordinate frame. Experiments on KITTI, nuScenes, and Oxford RobotCar show consistent improvements in metric depth estimation, multi-view point-cloud reconstruction, and cross-view camera localization, while preserving strong generalization across
datasets and geographic regions.
Read the full abstract ▾
All authors Xianghui Ze, Yongjian Luo, Mengjun Chao, Zhenbo Song, Jianfeng Lu, Yujiao Shi
Computer Vision · Cross-listed from cs.LG
Neural Field Tokenizations with Hierarchy and Spatial Locality Priors
Alonso Urbano, David W. Romero, Max Zimmer et al. (+1 more)
Neural fields parameterize data as functions from coordinates to values, providing a unified framework for representation learning across modalities. Existing approaches are dominated by per-sample meta-learning, which scales poorly due to
memory-intensive inner-loop optimization. The natural alternative – feed-forward encoding – typically introduces modality-specific assumptions, sacrificing the generality that makes learning with neural fields attractive. We argue that locality and hierarchy are useful priors for learning field representations that can be injected without compromising modality-agnosticism. We propose LH-NeF, a framework to learn general-purpose tokenized representations of continuous signals. A locality-preserving hierarchical encoder maps raw coordinate-value field observations to structured
tokens, from which the field is reconstructed during training. By replacing meta-learning's inner loop with a single forward pass, LH-NeF uses 42x less
memory and supports 133x larger batches than the strongest modality-agnostic baseline. Across images, 3D shapes, and
climate fields, our learned representations match or exceed performance of modality-agnostic, modality-specific, and specialized generative neural field baselines on both reconstruction and downstream tasks.
Read the full abstract ▾
All authors Alonso Urbano, David W. Romero, Max Zimmer, Sebastian Pokutta
Computer Vision
How Much MRI Preprocessing Is Enough? A Cost-Utility Study for Brain MRI Foundation Models
Jiangshuan Pang, Wangyang Tang, Jing Yan et al. (+5 more)
MRI preprocessing defines the input distribution seen by
brain MRI foundation models, yet it is usually treated as routine data cleaning rather than a modeling choice. We ask how much preprocessing is worth its computational cost for self-supervised 3D MRI pretraining. Keeping the corpus, 3D ViT backbone, masking protocol, and downstream evaluations fixed, we compare a graded P0-P7 preprocessing spectrum for masked autoencoding (MAE) and joint-
embedding predictive learning (JEPA) on 20,000 heterogeneous
brain MRI volumes, then transfer the encoders to IDH prediction, MCI classification,
brain age regression, and GLI/PED tumor
segmentation. The results do not support a simple "more is better" rule. P0/P1 are numerically unstable, making P2 the lowest-cost feasible level; beyond P2, choosing the best feasible preprocessing level improves aggregate utility by only 3.4 percentage points for MAE and 1.8 percentage points for JEPA, with most paired gains statistically unresolved. Stronger preprocessing is beneficial only in selected regimes: IDH improves modestly, AGE and GLI/PED are often near or best at P2, and MCI shows the clearest empirical P7 gain. Cross-level MCI transfer further shows that much of the P7 advantage can be recovered by applying stronger preprocessing downstream, without requiring P7 throughout pretraining. These findings recast MRI preprocessing as a downstream-aware cost-utility decision rather than a default escalation pipeline. Code is available at https://github.com/PangJiangShuan/PreBrain.
Read the full abstract ▾
All authors Jiangshuan Pang, Wangyang Tang, Jing Yan, Zhixuan Cheng, Youzhe He, Yiting Deng, Zhenkun Zhuang, Shiping Liu
Computer Vision
RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
Kyumin Choi and Ikbeom Jang
Vision Transformers (ViTs) achieve strong performance but suffer from high computational costs due to quadratic self-
attention complexity. Although
token reduction techniques such as
pruning and merging mitigate this, they typically overlook how representations evolve across network depth. We propose RAPID, a depth-aware
token reduction framework that adapts reduction strategies to the layer-wise characteristics of
token representations. The primary methodological contribution is a bifurcated strategy: in shallow-to-middle layers, RAPID employs a redundancy-similarity aware
pruning metric to eliminate over-represented local patterns. As features transition to global semantic concepts in deeper layers, the framework shifts to an importance-similarity aware merging mechanism. This stage leverages classification (CLS)
token attention weights to protect semantically critical
tokens while fusing less important but similar neighbors. Empirical validation on ImageNet-1K using ViT and DeiT architectures demonstrates that RAPID establishes a superior accuracy-compression Pareto frontier compared to plug-and-play baselines such as ToMe and ToFu. RAPID is particularly robust in aggressive compression regimes, achieving up to 4.29% higher accuracy than ToMe at extreme reduction rates. Our framework provides a training-free template for optimizing
vision models by aligning reduction strategies with hierarchical feature
evolution.
Read the full abstract ▾
Computer Vision
Property-Informed Diffusion-Based Text-to-Microstructure Generation
Bingxuan Dai, Hongsong Wang and Jie Gui
Designing 3D metamaterial microstructures that meet the intended functions remains a major challenge, as it typically requires domain expertise, iterative simulations, and extensive manual tuning. Existing work on inverse design that automatically generates microstructures based on desired target properties often suffers from limited design diversity and faces challenges in ensuring the physical feasibility of the generated structures. To address this issue, a property-informed
diffusion-based network is proposed that enables the generation of 3D microstructures directly from textual descriptions. Unlike traditional property conditioning methods, our approach leverages rich guidance in terms of semantics and physical properties in the text input to support diverse structure synthesis. To enforce consistency between the generated structures and the target textual
prompts, a dual
alignment strategy is adopted, including contrastive text-structure
alignment and test-time reward-guided
alignment. Experimental results show that the model is capable of generating semantically meaningful and physically plausible structures across a wide range of material categories. Our approach has good potential for interactive microstructure design and opens up new directions for combining
language-based interfaces with inverse material discovery. Code is available at: https://github.com/hongsong-wang/PropDiff-TMG
Read the full abstract ▾
Computer Vision
IMAGINE: Adaptive Schema-Imagery Enhanced Composition for Composed Video Retrieval
Jiale Huang, Zixu Li, Zhiwei Chen et al. (+3 more)
Composed
Video Retrieval (CVR) is designed to retrieve a target
video that matches a reference
video modified by a modification text. While existing methods explore cross-modal correspondences, they often assume modified objects appear directly in
videos. However, modification texts frequently describe concepts not explicitly presented but implicitly expressed through semantically related visual cues (e.g., "cake" implying "birthday party"). Current approaches typically rely on aligning explicit feature representations within the concrete space, neglecting critical latent associations. To address this, we propose an adaptIve scheMa-ImAGery enhanced composItional NEtwork (IMAGINE). Unlike standard explicit matching, IMAGINE materializes implicit semantics (termed schema imagery) via dynamic
multimodal prototypes. These prototypes capture shared latent concepts to adaptively modulate visual features, effectively injecting implicit guidance into the
retrieval process. By bridging the gap between explicit visual contents and implicit
retrieval intentions, IMAGINE achieves state-of-the-art performance in both CVR and Composed Image
Retrieval (CIR) across three widely used
benchmarks.
Read the full abstract ▾
All authors Jiale Huang, Zixu Li, Zhiwei Chen, Zhiheng Fu, Chunxiao Wang, Yupeng Hu
Computer Vision
Gravity-guided Contact Dynamics Estimation from 3D Human Motions
Cuong Le, Urs Waldmann, Bastian Wandt et al. (+1 more)
Ground contact forces acting on the human body, are crucial for biomechanics studies or sport performance analysis. Prior methods rely on force plates or pressure mats to collect ground contact dynamics, limiting their applicability to carefully controlled settings. A more scalable solution is to estimate the dynamics directly from motion capture data. Recent approaches only roughly estimate the ground contact dynamics from the vertical distance between the body and the ground plane, which cannot capture the complex pressure distribution of all contact points. To this end, we propose GraCE –
Gravity-guided Contact Dynamics Estimation, a novel full-body contact dynamics model for human motions using a realistic influence of body mass distribution and
gravity. We use the human's center of
gravity to estimate the ground contacts based on its relative distance to the human body. The applied force on each contact is estimated via the product of predicted contact probabilities and the total exterior force computed from the center of mass trajectory. We outperform related work on the GroundLink
dataset for ground reaction force estimation, and on the MOYO
dataset for detailed contact pressure prediction. The code is published at https://github.com/cuongle1206/GraCE
Read the full abstract ▾
All authors Cuong Le, Urs Waldmann, Bastian Wandt, Mårten Wadenbäck
Computer Vision
Phase Marginalization for Patch-Grid Instability in Vision Transformers
Oğuzhan Ercan
Vision Transformers operate on fixed patch grids, which can introduce phase-dependent instability for dense prediction: changing the patch partition can change the
token evidence available to a pixel, especially near boundaries. We formalize patch-grid phase as a nuisance variable and propose Phase Marginalization, a post-hoc marginalization method that evaluates structured patch-grid phases, inverse-aligns dense outputs, and aggregates them in the original image coordinate system. The central variant, Uniform Phase Marginalization with K = 4, is training-free and improves over the canonical K = 1 baseline across measured
segmentation, depth, and local matching settings. In a controlled Cityscapes experiment, Uniform Phase Marginalization provides a modest compute-matched advantage over generic shift-based four-forward test-time augmentation (TTA) (+0.31 mean Intersection-over-Union over the strongest tested generic row). A scaling study further shows that K = 4 is a practical cost-accuracy trade-off: K = 8 is essentially unchanged and K = 16 adds little accuracy at much higher latency. These results position patch-grid phase as a measurable nuisance variable and Phase Marginalization as a simple diagnostic and post-hoc marginalization baseline for dense ViT prediction.
Read the full abstract ▾
Computer Vision
One Stone, Three Birds: Self-adaptive Optimal Transport for Multi-VLM Selection, Adaptation, and Ensembling
Qiyu Xu, Zhanxuan Hu, Yu Duan et al. (+4 more)
Vision-
language models (VLMs) enable visual recognition from semantic class descriptions, which makes them attractive when target annotations are scarce or unavailable. Most deployment pipelines, however, first choose a single VLM and then adapt that model to the unlabeled target set. This single-backbone paradigm hides a critical assumption: the selected VLM is already compatible with the target domain. In realistic cross-domain deployment, several general-purpose and domain-specialized VLMs may be plausible, yet no instance-level target labels are available to identify the reliable ones. Deployment therefore requires a coupled solution for model selection, target adaptation, and prediction integration. We revisit this problem from a system-level multi-VLM perspective. Our central observation is that the three decisions above depend on the same latent object: a trustworthy sample-class structure in the target set. Different VLMs may encode different transfer biases and produce conflicting predictions, but their outputs can still provide complementary evidence for estimating this structure. We propose One Stone, Three Birds, a training-free framework based on self-adaptive optimal transport. Given a pool of frozen candidate VLMs, OSTB estimates a consensus sample-to-class transport plan without updating VLM parameters. The learned transport structure is then reused for all deployment objectives: model selection is performed by ranking the combined semantic and visual reliability induced by the consensus plan; target adaptation is obtained by fitting transport-conditioned visual classifiers; and ensembling is implemented through reliability-aware probabilistic integration. Extensive experiments on natural-image, remote-sensing, and medical-pathology
benchmarks show that OSTB improves model ranking, adaptation stability, and ensemble robustness under heterogeneous candidate pools.
Read the full abstract ▾
All authors Qiyu Xu, Zhanxuan Hu, Yu Duan, Yonghang Tai, Huafeng Li, Quanxue Gao, Xiangyong Cao
Computer Vision
Beyond aggregate scores: Deployment-aware and non-compensatory benchmarking of vision-based eye-state recognition models for driver monitoring
Ruben Dario Florez-Zela
Model selection for safety-relevant visual recognition is often based on clean aggregate performance, although robustness, transfer, embedded latency, and explanation faithfulness may produce different preferences. This study presents a Human-Centered Benchmarking Framework (HCBF) that separates multidimensional evidence from non-compensatory operational eligibility. Six compact convolutional and
transformer-oriented eye-state recognition models were evaluated using a subject-disjoint MRL Eye protocol, deterministic image corruptions, zero-shot transfer and participant-safe target-domain training with out-of-fold evaluation on RT-BENE, TensorRT FP32 inference on an NVIDIA Jetson Nano, and black-box RISE faithfulness. Clean MRL Macro-F1 ranged from 0.9566 to 0.9794, whereas zero-shot RT-BENE Macro-F1 ranged from 0.2066 to 0.7771. Matched target-domain effects varied from -0.0406 to 0.4807 and redistributed the two directional errors differently across architectures. Only MobileNetV3-Large and ShuffleNetV2 met the 33.333-ms binocular-pair latency deadline, while neither passed the predefined safety-related screen. The remaining four models failed both requirements, yielding an empty eligible set. Normalized deletion AUC ranged from 0.5826 to 0.9113, while normalized insertion coverage varied from 17.6% to 88.6%. Model ordering changed across clean prediction, corruption robustness, transfer, deployment, faithfulness, and historical score sensitivity. These findings show that relative ranking, multidimensional preference, and operational eligibility are distinct decisions. Deployment-aware benchmarking should preserve directional failures and uncertainty and should allow no model to be selected when mandatory requirements are unmet.
Read the full abstract ▾
Computer Vision
Trustworthy Visual Predicates for Robust Manipulation Understanding under Degradation
Fatemeh Ziaeetabar
Manipulation understanding requires reliable relational evidence, such as contact, support, containment, motion coupling, grasp, release, and active-hand involvement. Although these visual predicates are widely used in event-chain, graph-based, and neuro-symbolic models, their reliability under visual degradation is rarely analyzed directly. This paper introduces a predicate-level reliability framework for robust manipulation understanding under blur, occlusion, illumination change, low resolution, frame dropping, and detection noise. The framework defines a structured predicate vocabulary, confidence-aware predicate estimation, and reliability metrics for predicate preservation, degradation sensitivity, temporal consistency, confidence-weighted stability, and downstream impact. Experiments on controlled manipulation
videos and public
egocentric or bimanual
datasets, including VISOR/EPIC-KITCHENS, H2O, and ARCTIC, show that predicate failures are structured rather than uniform. Static spatial predicates remain comparatively robust, whereas contact-sensitive, dynamic, and derived predicates such as grasp and release are more fragile. Under severe degradation, detection noise, occlusion, and frame dropping cause the strongest reliability losses. Downstream analysis shows that degraded predicates reduce manipulation-understanding accuracy from 0.89 to 0.58, while removing confidence weighting under moderate degradation reduces accuracy from 0.74 to 0.64. These results show that predicate reliability provides a diagnostic layer between visual perception and structured manipulation
reasoning.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Revisiting Articulated Parts Perception in Robot Manipulation
Xiaoqian Wu, Yejie Guo, Xiaoyang Chen et al. (+3 more)
We are surrounded by various objects with movable, articulated parts, e.g., box, handle, door. An accurate and generalizable perception of articulated parts is essential to enhance robotic manipulation capabilities. Building on this need, recent efforts in articulated parts perception have followed two main directions: One line of work uses pose-based representation, which requires high manual cost; in parallel, affordance-based methods extract future object motion from point tracking without additional manual efforts, but suffer from low-quality data. In this paper, we propose a new representation of articulated parts,
Geometric Primary Structure (GPS), an abstraction of the part geometry structure to balance scalability and quality. For efficient and scalable data collection, GPS is integrated with a portable Virtual Reality (VR) device and requires only one minute to annotate one object sequence. This direct human annotation provides higher quality than the estimated affordance. With this efficient VR-GPS system, we collect 41K frames for 234 objects across six part classes, and train a generalizable GPS model with a single RGB-D object image as input. For object manipulation, we deploy a heuristic policy based on GPS prediction. Without any in-domain fine-tuning, our method achieves an 73% success rate, covering 270 initial states for 9 objects. Our code, data and reusable tool are available at https://enlighten0707.github.io/gps.
Read the full abstract ▾
All authors Xiaoqian Wu, Yejie Guo, Xiaoyang Chen, Lixin Yang, Cewu Lu, Yong-Lu Li
Computer Vision
VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation
Jianhui Wei, Yan Zhang, Jie Tan et al. (+7 more)
Agentic long
video generation requires planning, tool orchestration, and cross-clip coordination over a long horizon. Most existing
video agents either rely on static, human-crafted workflows, which require substantial manual effort and poorly adapt across tasks, or iteratively refine the output of the current task without persistently distilling execution experience into reusable skills for future tasks. We introduce VideoWeaver, an
agent harness and
benchmark that evaluates and evolves skills for long
video generation. Given a single high-level instruction, an
agent dynamically composes foundation skills into its own workflow rather than following a predefined pipeline. We construct a
benchmark of 16 task categories and 285 cases, with references spanning text, image, audio,
video, and their combinations. We further propose an evidence-grounded
agent-as-judge that inspects both the execution trace and the final
video to diagnose process and output failures. Based on this feedback, our
evolution algorithm progressively refines category-level composition and creator skills, allowing recurring experience to guide dynamically constructed workflows for unseen cases. Experiments show that explicit composition skills improve the generation process over foundation skills alone, while skill
evolution further improves output quality and generalizes to unseen cases. Incorporating judge feedback yields additional gains, especially on output metrics, and the
agent-as-judge aligns well with human, particularly on process metrics. Code is available at https://github.com/JianhuiWei7/VideoWeaver.
Read the full abstract ▾
All authors Jianhui Wei, Yan Zhang, Jie Tan, Hengchuan Zhu, Xiaotian Zhang, Ziyi Chen, Daoan Zhang, Wei Xu, Yeying Jin, Zuozhu Liu
Computer Vision
Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?
Jiaqi Tang, Jianmin Chen, Youyang Zhai et al. (+6 more)
Multimodal Large
Language Models (MLLMs) have demonstrated remarkable success in visual understanding, yet their performance degrades significantly under real-world visual corruptions. While existing robustness enhancement approaches exist, they are limited: black-box feature
alignment lacks interpretability, and white-box text-based
reasoning cannot restore lost pixel-level details. This work investigates a fundamental research question: Can MLLMs recover corrupted visual content by themselves? To address this, we propose Robust-U1, a novel framework that equips MLLMs with explicit visual self-recovery capability for robust understanding. The approach comprises three core stages: supervised fine-tuning for initial reconstruction, reinforcement learning with dual rewards (pixel-level SSIM and semantic-level CLIP similarity) for aligning high visual quality, and
multimodal reasoning that jointly considers both the corrupted input and the recovered image. Extensive experiments demonstrate that Robust-U1 achieves state-of-the-art robustness on the real-world corruption
benchmark and maintains superior performance under
adversarial corruptions on general VQA
benchmarks. Analysis confirms that high-quality visual recovery directly enhances
reasoning performance, establishing self-recovery as a critical mechanism for robust visual understanding. The source code is available at https://github.com/jqtangust/Robust-U1.
Read the full abstract ▾
All authors Jiaqi Tang, Jianmin Chen, Youyang Zhai, Wei Wei, Runtao Liu, Mengjie Zhao, Xiangyu Wu, Qingfa Xiao, Qifeng Chen
Computer Vision · Cross-listed from cs.AI
OSMGraphCLIP: Learning Global Location Representations from OpenStreetMap Graphs
Dimitrios Michail, Eleni Saka, Ioannis Giannopoulos et al. (+1 more)
We present OSMGraphCLIP, a CLIP-style geospatial representation model that learns global location
embeddings from freely available OpenStreetMap (OSM) data. OSMGraphCLIP represents geographic environments as heterogeneous graphs of typed OSM features, preserving the topological and semantic relationships among roads, buildings, land-use regions, and points of interest. A multi-scale graph encoder captures both fine-grained local structure and broader landscape composition, and supervises a spherical-harmonics location encoder through a contrastive
alignment objective. We evaluate OSMGraphCLIP across a diverse suite of downstream geospatial regression and classification tasks spanning
climate, ecology, socioeconomic indicators, public health, land cover, biodiversity, and wildfire
forecasting, and show that structured OSM data alone supports strong global location representations across domains. OSMGraphCLIP matches or exceeds satellite-based baselines on the majority of
benchmarks, with the most pronounced advantage on socioeconomic and public-health tasks, where OSM's explicit semantic annotation of the built environment encodes patterns of human activity that satellite pixels can only capture indirectly. On ecological and environmental tasks, the model remains closely competitive with imagery-based methods despite using no Earth observation data. Qualitative analysis confirms that the learned
embeddings organize geographic space coherently, recovering biome boundaries, urban gradients, and tropical–temperate distinctions from map
topology alone.
Read the full abstract ▾
All authors Dimitrios Michail, Eleni Saka, Ioannis Giannopoulos, Ioannis Papoutsis
Computer Vision · Cross-listed from cs.GR
OmniFaceRig: Fully Automatic Inner-Mouth-Aware Face Rigging Across Diverse 3D Character Topologies
Chao Wang, Guangyao Ma, John Doublestein et al. (+11 more)
Facial rigging–creating Facial Action Coding System (FACS) blendshapes and inner-mouth geometry (teeth, gums, and tongue)–remains a major bottleneck in 3D character production. We present OmniFaceRig, a fully automatic end-to-end pipeline that converts a static surface-only 3D character mesh, with no pre-modeled oral cavity, into an inner-mouth-aware FACS rig with up to 155 blendshapes, procedurally fitted teeth, gums, and tongue, and re-packed UV/texture. It supports humans,
humanoids, and long- and short-muzzled animals with no manual landmarks, user-provided templates, or per-asset setup. OmniFaceRig combines a hybrid
vision-
language and computer
vision riggability checker, a four-model face
segmentation ensemble, keypoint-driven template registration, procedural inner-mouth construction, and collision-aware blendshape transfer. For non-human characters, it selects
topology-specific templates with a nose-landmark-free design and uses as-rigid-as-possible deformation with signed-distance-field constraints to reduce teeth-face intersections. We also introduce Omni-Bench, an open-source
benchmark of more than 1,000 biped 3D characters rigged with FACS blendshapes and inner-mouth geometry, spanning humans,
humanoids, cats, dogs, and other animals. Experiments show high rigging success, nearly complete face detection recall, low inner-mouth penetration, and 20-30 s end-to-end latency per asset on a single A100 GPU.
Read the full abstract ▾
All authors Chao Wang, Guangyao Ma, John Doublestein, Junming Chen, Yiming Lin, Zhaoen Su, Xiaomin Luo, Yushan Yin, Shiyang Cheng, Jie Shen, Doug Roble, Dilin Wang, Yilei Li, Rakesh Ranjan
Computer Vision · Cross-listed from cs.GR
Wispy to Voluminous: Prior-free Multi-view Capture of Strand-level Facial Hair
Jaeseong Lee, Giljoo Nam, Adrian Jarabo et al. (+1 more)
Facial hair is a defining trait of personal identity, yet remains a critical bottleneck for digital avatars. Recent volumetric methods achieve photorealism but bake hair into the underlying face geometry, preventing editability and failing to resolve sparse, strand-like structures. Meanwhile, scalp-hair reconstruction methods target dense hair volumes and do not transfer to the sparse, spatially-varying nature of facial hair. We present a pipeline that automatically reconstructs facial hair – beard, mustache, lashes, and brows – from multi-view images, converting an unstructured 3D Gaussian representation into an explicit curve-based strand representation. We resolve
geometric ambiguities in four stages: (i) optimizing 3D Gaussians constrained by tracked head geometry to enforce early ray termination and suppress sub-surface noise; (ii) tracing continuous strands robust to frequent crossings and extreme
curvature; (iii) grounding strands to the surface and resolving root-tip ambiguity via a physically-motivated prior; and (iv) refining the reconstruction through opacity-driven density control under photometric optimization. To our knowledge, this is the first method to reconstruct high-fidelity facial hair strands from a 3D Gaussian representation. The recovered strands faithfully preserve the orientation and sparsity patterns characteristic of facial hair, and yield assets immediately suitable for downstream production tasks, including facial animation and physical simulation,
geometric grooming and transfer, appearance editing, and physics-based rendering.
Read the full abstract ▾
All authors Jaeseong Lee, Giljoo Nam, Adrian Jarabo, Carlos Aliaga
Computer Vision
DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning
Hangui Lin, Yan Shu, Zhengyang Liang et al. (+8 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a leading paradigm for enhancing visual
reasoning in
Multimodal Large
Language Models (MLLMs). However, existing RLVR methods optimize primarily for the
reasoning outcome and overlook how that outcome is produced: as a chain-of-thought unfolds, some
tokens must extract evidence from the image while others must anchor to the preceding textual context, yet a single sequence-level reward provides no signal about whether each
token draws on the modality its role demands.
Token-level analyses and controlled interventions reveal that
tokens frequently under-attend to their designated modality, and that this coordination breakdown may contribute to
reasoning failures. Motivated by these findings, we propose DyCo-RL, which integrates dynamic cross-modal coordination into RLVR optimization. Specifically, DyCo-RL uses the Fisher-Rao geodesic distance to measure within-modality
attention shifts, assigning
tokens to either visually-oriented or text-oriented functional roles. It then evaluates the
alignment between a
token's actual
attention allocation and its assigned role, leveraging this score for
alignment-guided advantage reweighting during policy optimization. Extensive experiments using 11K examples for RL training demonstrate that the
algorithm-agnostic DyCo-RL, when applied to Qwen2.5-VL-3B/7B and Qwen3.5-27B, improves the average performance of all four evaluated RLVR
algorithms across seven
benchmarks spanning visual-centric and mathematical
reasoning.
Read the full abstract ▾
All authors Hangui Lin, Yan Shu, Zhengyang Liang, Chi Liu, Xiangrui Liu, Minghao Qin, Derek Li, Bryan Dai, Teng Long, Zheng Liu, Nicu Sebe
Computer Vision
Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems
Muhammad Falensi Azmi, Ikhlasul Akmal Hanif, Vallerie Alexandra Putra et al. (+3 more)
Symbolic
benchmarks have emerged as a key approach to assess model robustness under minor modifications to STEM-related questions. However, existing symbolic
benchmarks mostly remain limited to mathematical
reasoning, lack visual grounding, and are predominantly in English. In this work, we introduce Sci-Rho (Science Rhobustness), a dynamic
benchmark for visually-grounded STEM problems spanning five subjects and seven
languages, comprising 4,242 problem templates (606 per
language) crafted by domain experts, including Olympiad medalists. Each template is implemented as executable Python code that generates diverse but equivalent problem instances by varying numerical values, visual patterns,
geometric shapes, color schemes, and function types, resulting in 42,420 instances in total, each paired with
reasoning steps and ground-truth solutions. We evaluated 17 state-of-the-art VLMs and discovered a noticeable gap between worst-case accuracy (defined as the proportion of problem templates that a model answers correctly across every generated variation) and average accuracy. We also discovered that smaller models show noticeable performance degradation across
languages, whereas proprietary and larger models remain robust. Step-level evaluation reflects this same trend, revealing a significant gap between average F1 and worst-case F1 scores. Finally, our inspection of
attention heads of a VLM reveals substantial cross-lingual variation in the relative
attention allocated to image
tokens compared to text
tokens. Our work highlights the importance of evaluation beyond static
benchmarks as a metric to measure the quality of VLMs.
Read the full abstract ▾
All authors Muhammad Falensi Azmi, Ikhlasul Akmal Hanif, Vallerie Alexandra Putra, Adi Yeltay, Abdullah Mubarak, Fajri Koto
Computer Vision
Balancing Real and Synthetic Data for CNN-based Masonry Crack Detection
Mattia Forlesi, Alfonso Esposito, Ivan Zyrianoff et al. (+2 more)
Cracks are a critical indicator of building health, and early stage identification is fundamental to prevent harmful damages. Advances in deep learning (DL), particularly convolutional neural networks (CNNs), have enabled scalable solutions for automated crack detection. However, CNN performance strongly depends on the availability of large and diverse
datasets, which is particularly challenging for complex surfaces such as masonry. Collecting sufficient real data is time-consuming, while publicly available
datasets may not be adequate. To address this limitation, we explored generating synthetic crack data, which complements real data and improves training effectiveness. The real
dataset consists of masonry crack images collected from buildings in Bologna and surrounding areas. In contrast, the synthetic
dataset was generated using a crack overlay tool that adds cracks to background images in a controlled orientation and placement. The real
dataset was used to train several DL architectures, to identify the best-performing model (InceptionV4) employed for experiments with generated data. Six training scenarios were tested in InceptionV4 by varying the ratio of real and synthetic data, with evaluation performed on a test set composed of real images using the F1-score and mean Intersection over Union (mIoU) metrics. Results show that training on synthetic data plus a modest addition of 20% real data achieves results comparable to training on real data only. Moreover, the 20/80 scenario (synthetic/real) achieved an 76% F1-score and 80% mean IoU, outperforming the real-only case. As can be seen, the method demonstrates the potential of synthetic data to reduce collection efforts while enhancing crack detection accuracy.
Read the full abstract ▾
All authors Mattia Forlesi, Alfonso Esposito, Ivan Zyrianoff, Alessandro Marzani, Marco Di Felice
Computer Vision
Vision-Language Asymmetry in Bistable Image Captioning
Arohan Agate
Wittgenstein's duck-rabbit poses a question for
vision-
language models: when a model captions an ambiguous image, where in the model is the commitment to one aspect made? We address this with a 3,320-generation behavioral baseline over 83 bistable stimuli that surfaces three regimes (default-dominant, force-dominant, force-balanced) under neutral vs forced-choice prompting, then probe the underlying representations using a TopK sparse autoencoder we train on the CLIP layer that LLaVA-1.6-7B actually consumes (validation EV 0.93). Across 69 bistable stimuli with both per-aspect feature pools available, 72% (50/69) show simultaneous activation of both pools at the
vision tower, including 12/12 default-dominant duck/rabbit and 7/8 force-balanced young/old. Causal steering at CLIP layer 22 flips captions on default-dominant stimuli (33% rabbit-flip rate under a fluency guard) but cannot flip captions on force-balanced young/old at any tested coefficient, despite their
vision-side superposition. The dominance bottleneck lives downstream of the
vision tower; the gap between
vision-side representation and
language-side commitment is an empirical handle on the seeing/seeing-as distinction. We also flag a methodological note: rank-based statistics on TopK SAE outputs require tie-corrected ranking to avoid silent row-order bias.
Read the full abstract ▾
Computer Vision
IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment
Zichen Zhu, Yuheng Sun, Mingxuan Zhu et al. (+11 more)
Current image editing software often hinges on fixed filters or expert tuning, leaving a gap between amateur users' intent and outcomes. Creations by generative models may contain artifacts, implausible details, or stylistic drift away from photorealism and offer little insight into why an edit was made. We propose IEA, a conversational Image Editing
Agent that learns to operate parameterized tools in an explicit, interpretable action space. IEA is trained via a three-stage multitask pipeline: (1) SFT on distilled expert edits, (2) GRPO with rewards for likeness improvement, tool usefulness, and intent summarization, and (3) large-scale synthetic fine-tuning to jointly master image editing, refinement, and user intent summarization. By manipulating 16 editing tools step by step, IEA produces transparent edit traces that can be inspected and debugged. In quantitative experiments, it attains a lower pixel distance on the edit task and a higher ROUGE-L on the summary task than strong baselines. In user studies, it ranks best among tool-calling methods for instruction following while surpassing generative methods in overall perceptual quality. Our results validate interpretable, tool-centric VLMs as a reliable path to human instruction-guided image retouching.
Read the full abstract ▾
All authors Zichen Zhu, Yuheng Sun, Mingxuan Zhu, Wenjie Ma, Situo Zhang, Zhexiang Wang, Ziyue Yang, Danyang Zhang, Kunyao Lan, Zihan Zhao, Dingye Liu, Siqi Xiang, Lu Chen, Kai Yu
Computer Vision
GVC-Seg: Training-Free 3D Instance Segmentation via Geometric Visual Correspondence
Liang Xu, Fangjing Wang, Jinyu Yang et al. (+1 more)
Accurate 3D instance
segmentation in point cloud data is critical for machine
vision applications. Recent advancements leverage multiple pre-trained foundation models to generate 3D proposals, followed by the application of proposal aggregation methods, which significantly enhance performance. However, they often produce sub-optimal results due to inherent variations in confidence levels across different
segmentation models, resulting in a bias toward the model with higher confidence. This bias is inherently model-dependent and is influenced by factors such as data preprocessing techniques and training strategies. To address this bias, we propose a novel, training-free 3D instance
segmentation approach via
Geometric Visual Correspondence (GVC-Seg), which exploits the correspondence between 3D
geometric cues and 2D visual cues to mitigate the confidence bias. Additionally, a 3D proposal generation module and a mask-aware CLIP feature extraction module are introduced during the instance mask generation and instance semantic
reasoning, respectively. In this way, GVC-Seg enhances proposal quality assessment, ensuring unbiased ensemble learning across different models. Extensive experiments demonstrate that our method achieves state-of-the-art performance on several challenging
benchmarks, while also exhibiting strong potential in open-vocabulary semantic
segmentation settings.
Read the full abstract ▾
All authors Liang Xu, Fangjing Wang, Jinyu Yang, Feng Zheng
Computer Vision
CoRef-DETR: Decoupled Global Context and Selective Boundary-Category Refinement for Underwater Instance Segmentation
M. Fazri Nizar, Julian Supardi and Muhammad Naufal Rachmatullah
Underwater instance
segmentation supports marine surveys, aquaculture, and
underwater robotics. Haze, colour casts, and low contrast blur object boundaries, and recent
benchmarks distinguish up to 158 fine-grained categories. The strongest published methods mostly have 219 to 632 million parameters. This paper proposes CoRef-DETR, a compact detection
transformer that targets three error sources of a real-time detector: context, boundaries, and categories. First, a decoupled global context (DGC) detector, with the base detector's 33.4 M parameters, applies global self-
attention to a 384-pixel
token grid but supervises and decodes masks in a 576-pixel frame. Second, a posterior-conditioned boundary refiner with 47.7 k parameters corrects mask logits on predicted boundaries, and a quality score on the corrected masks re-ranks the detections. Third, a crop-probe category branch on frozen DINOv2 features revises the categories of at most three masks per image, where a rule fixed on validation data chooses soft fusion or a confident-disagreement gate. On five protocols of UIIS, UIIS10K, USIS10K, and USIS16K, CoRef-DETR achieves state-of-the-art mask average precision (AP), exceeding the strongest published results by 2.3 to 5.2 points, and outperforms three reproduced recent methods on the same test images (on USIS16K, with its category branch). In half precision with TensorRT, it processes an image in a median of 34 to 40 ms, JPEG decoding included, on a T4 GPU, 1.4 to 11 times faster than these methods. Ablations attribute most of DGC's gain to the larger frame; an error analysis locates the remaining error in crowds of small objects.
Read the full abstract ▾
Computer Vision
Learning a Semantic Calibration Network for Open-Vocabulary Semantic Segmentation
Yang Sun, Tao Wang, Anastasia Ioannou et al. (+1 more)
Semantic image
segmentation assigns a predefined category label to each pixel, has achieved significant progress lately. Open-Vocabulary
Segmentation (OVS) extends the
segmentation task from a fixed set to an open set, enabling the identification and
segmentation of novel concepts based on arbitrary text inputs, such as category names or descriptions. In this paper, we propose a novel Semantic Calibration Network (SCN) for open-vocabulary semantic
segmentation. Different from prior approaches that focus on feature aggregation or simple fine-tuning of pre-trained models, SCN refines the mask classification process by explicitly modeling the semantic correlations between classes, aiming to enhance the model's discriminative power while effectively preserving the generalization abilities of the pre-trained CLIP model. Specifically, SCN comprises two core components: Class Disambiguation (CD) and Logits Fusion (LF). First, a cross-
attention mechanism is utilized to transform the text
embeddings into visually aware pseudo-text
embeddings, in order to derive an enhanced similarity score that complements the original mask-text similarity score. Subsequently, the Class Disambiguation module captures implicit inter-class dependencies through a residual architecture to effectively resolve semantic ambiguities. Finally, the Logits Fusion module dynamically integrates multifaceted semantic evidence to ensure that the model achieves a robust semantic consensus while maintaining CLIP's inherent generalization capability. Comprehensive experimental results on mainstream
benchmarks demonstrate that the proposed method achieves significant performance improvements compared to state-of-the-art
algorithms.
Read the full abstract ▾
All authors Yang Sun, Tao Wang, Anastasia Ioannou, Ge Xu
Computer Vision
FMRFusion: Frequency-Aware Multi-View Representation Learning for Heterogeneous Image Fusion
Tao Zhoua, Yunlong Liu, Qinghui Chen et al. (+6 more)
Infrared and visible image fusion aims to generate a composite image that retains significant target information and preserves detailed textures, integrating two heterogeneous modalities. Previous image fusion methods typically adopt a single-module stacking approach to extract features from the two modalities. However, these approaches may result in incomplete learning of their distinct characteristics, thereby limiting the fusion effectiveness and constrain ing robustness in real-world heterogeneous data scenarios. To address these challenges, we propose FMRFusion, a frequency-aware multi-view representation learning network for Heterogeneous Image Fusion. A Multi-Scale Struc tural Perception Module is introduced to effectively capture discriminative structures, extracting fine-grained local structures and essential contextual information. A bilinear frequency decomposition mechanism is employed to sepa rate features into high-frequency and low-frequency components, enabling joint modeling of local details and global representations across different frequency domains. Moreover, a Cross-View Complementary Interaction is incorpo rated to explicitly model and fuse the complementary characteristics between reflected light information and radiative intensity responses, facilitating effective cross-view interaction. We further improve the Performance of the fused results by flow matching, which progressively refines the fused features by learning the transformation from coarse data to high-quality representations. Extensive experiments conducted on multiple
benchmark datasets demonstrate that FMRFusion achieves superior and consistent performance across a range of fusion tasks, especially in nighttime scenarios
Read the full abstract ▾
All authors Tao Zhoua, Yunlong Liu, Qinghui Chen, Zekai Zhang, Minlong Sun, Changlin Biana, Dagang Li, Wenmin Wang, Jinglin Zhang
Computer Vision
DisCo: World Models with Discrete Camera Motion Control
Hongrui Huang, Junke Wang, Quanhao Li et al. (+2 more)
Controllable
video world models target interactive world exploration, where models must faithfully execute explicit action commands while preserving visual quality and temporal coherence. However, most existing approaches rely on continuous camera trajectories as action conditions, which often lead to unreliable action following, especially under complex motion sequences. In this work, we identify action representation
entanglement as a key bottleneck in controllable
video generation, and show that continuous camera representations lead to high feature similarity across distinct motion patterns, degrading action controllability. Based on this insight, we propose DisCo, a controllable
video world model that conditions generation on a compact set of discrete action primitives to improve action separability. We further introduce DisCoBench, a comprehensive
benchmark for evaluating the ability of models in short-term, long-horizon, and highly dynamic exploration scenarios. Extensive experiments demonstrate that DisCo achieves significantly more reliable action following while preserving visual quality.
Read the full abstract ▾
All authors Hongrui Huang, Junke Wang, Quanhao Li, Yu-Gang Jiang, Zuxuan Wu
Computer Vision
ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?
Bin Zhu, Yanhao Jia, Kexin Zhao et al. (+12 more)
Recent advancements in
Multimodal Large
Language Models (MLLMs) have demonstrated remarkable proficiency in open-world
reasoning and understanding. However, a critical ambiguity persists: it remains unclear whether these models genuinely synthesize cross-modal information to construct physically grounded
reasoning chains, or if they merely exploit strong
language priors to mask single-modality reliance, thereby hallucinating advanced
multimodal capabilities. Motivated by this, and to rigorously mitigate
language modality bias and shortcuts, we propose a novel
multimodal Chrono}logical Physical Dynamics
Reasoning Benchmark ChronoPhyBench, which unifies next state prediction with Visual Question Answering (VQA) paradigms by conditioning on historical
video context and textual captions to enforce models to deduce subsequent physical states through both single image selection and the inherently more complex task of multiple frame chronological sorting. Concurrently, we construct a large-scale
multimodal reasoning dataset curated using the ChronoPhyBench criteria, comprising over 10,000 long-form
videos paired with meticulously annotated captions, totaling 5M
tokens. Our experimental evaluations reveal a stark contrast to conclusions drawn by previous
benchmarks. The capacity of current open-source models to perform physically grounded
multimodal reasoning remains in its infancy. Ultimately, this work seeks to systematically stress-test the
reasoning capabilities of
multimodal models, quantify
hallucination rates, and advance the development of Physical AI, thereby providing the community with a robust and transparent evaluation framework toward Artificial General Intelligence (AGI).
Read the full abstract ▾
All authors Bin Zhu, Yanhao Jia, Kexin Zhao, Jie Wang, Munan Ning, Hao Li, Yuwei Niu, Tanqing Sun, Huangchong Yan, Mingjun Pan, Xinyi Wu, Qishen Yin, Yunyang Ge, Shuai Zhao, Li Yuan