Computer Vision · Cross-listed from cs.LG
Adjoint Guidance Flow: Amortized Critic Guidance for VLA Policies
Jeongsol Kim, Youngjun Jun, Kyumin Choi et al. (+6 more)
Flow-based
Vision-
Language-Action (VLA) policies are typically trained by behavior cloning and thus do not explicitly optimize long-term task return. Critic guidance steers generation toward higher-value actions, but existing methods differentiate the critic through a one-step surrogate of the sampler and back-propagate a critic ensemble at every flow step. In contrast, here we propose Adjoint Guidance Flow (AGF), which amortizes trajectory-aware critic guidance into a lightweight guidance network while preserving the pretrained VLA policy. Specifically, we formulate critic-guided flow generation as a deterministic optimal control problem, whose optimal guidance is a costate that carries the terminal critic gradient back through the remaining flow, and regress the guidance network onto this costate while keeping both the VLA and critic frozen. This design provides favorable
memory and throughput scaling during training, and inference needs one guidance-network forward pass per step, without the critic ensemble, back-propagation, or adjoint computation. Across LIBERO, RoboCasa, and LIBERO-Pro, AGF consistently improves pretrained VLAs, remains competitive with critic-guidance and policy-fine-tuning baselines, and is the most robust method when a single guidance strength is deployed across tasks. Compared with QGF, AGF runs $3.6\times$ faster per guidance step with $7.0\times$ fewer parameters, with comparable and even better performance, showing that critic guidance can be trajectory-aware and lightweight.
Read the full abstract ▾
All authors Jeongsol Kim, Youngjun Jun, Kyumin Choi, Youngmin Kim, Seonghyun Jin, Sunwoo Park, Jangho Park, Kwanyoung Kim, Jong Chul Ye
Computer Vision
Resolution as a First-Class Decision: Task-Conditioned Routing for Efficient Multimodal Large Language Models
Zhiqiang Xia, Yang Li, Xinyuan Zhang et al. (+5 more)
The inference efficiency of
Multimodal Large
Language Models (MLLMs) is severely constrained by massive visual
token sequences induced by high-resolution inputs, with computational cost scaling quadratically. Existing approaches primarily focus on downstream
token compression, while overlooking a fundamental upstream inefficiency: input resolution is treated as a static, task-agnostic hyperparameter. We propose Task-Conditioned Resolution Routing (TCRR), which formulates visual compression as a task-conditioned decision and employs a lightweight cross-modal router that conditions backbone visual representations on textual semantics via feature-wise modulation and cross-
attention to predict the minimal sufficient compression level per query. To support this, we curate a
dataset of 500k samples across 12 task categories, labeled via a teacher-oracle pipeline to approximate Pareto-optimal compression scales. Extensive experiments across diverse architectures show that TCRR achieves a superior efficiency frontier, specifically reducing visual FLOPs by 40.9% and latency by 53.7% on Qwen3-VL-8B while preserving competitive performance. Further analysis of scaling behavior confirms that dynamically routing visual compression enables optimal resource allocation without modifying the MLLM backbone.
Read the full abstract ▾
All authors Zhiqiang Xia, Yang Li, Xinyuan Zhang, Yuchen Liu, Haoyu Lu, Jiaming Xu, Runyu Shi, Ying Huang
Computer Vision
TaoTex: Boosting Texture Detail Fidelity for Native 3D Material Generation
Xiuchao Wu, Shuichang Lai, Jiangjing Lyu et al. (+1 more)
Recent 3D generation models can produce accurate geometries while still struggling to reconstruct detailed textures. We propose a
diffusion-based native 3D material generation model TaoTex, which faithfully recovers intricate textures through tailored strategies and improvements. First, we develop a data construction
agent to create high-frequency textured 3D assets to bridge the data gap in public
datasets. Training with these data significantly enhances the ability of TaoTex to recover challenging details such as text and patterns. Second, we design a multi-level feature fusion (MLFF) module to adaptively integrate local and global features of the conditional input, providing more complete texture cues for the
diffusion model and thereby enhancing reconstruction fidelity. To alleviate VAE reconstruction errors, we adopt a latent-to-pixel space loss transition, further improving the pixel-level details and generation quality. Finally, we scale TaoTex to multi-view inputs by incorporating learnable viewpoint
embeddings, achieving accurate and consistent material reconstruction across views. Extensive experiments demonstrate that our method significantly outperforms existing approaches in preserving texture details in both single- and multi-view settings.
Read the full abstract ▾
All authors Xiuchao Wu, Shuichang Lai, Jiangjing Lyu, Chengfei Lyu
Computer Vision · Cross-listed from cs.RO
Don't Throw Away the Tail: Action Upcycling for Policy Acceleration
Taesung Kwon, Jangho Park, Sunwoo Park et al. (+5 more)
Modern
robot policies predict a chunk of future actions from a single observation, execute only a prefix, and discard the rest before replanning. Choosing the length of this prefix, the execution horizon, poses a trade-off between reactivity and efficiency. A short horizon keeps the policy reactive to the environment, but requires frequent policy calls. Recent test-time methods adaptively select the horizon for each chunk, but they either read model internals, where the signal must be chosen for each architecture, or draw extra samples, which adds cost. We propose Action Upcycling, a training-free
algorithm that reuses actions the policy would otherwise discard, without accessing model internals or drawing extra samples. We find that discarded actions stay close to their replanned versions as long as the action velocity remains smooth. Action Upcycling therefore extends the execution horizon up to the point where the velocity begins to fluctuate. Extensive experiments on simulated and real-world manipulation tasks show that Action Upcycling reduces policy calls by 1.2-1.7x with no loss in success rate, across multiple
Vision-
Language-Action Models (VLAs) and even a World Action Model (WAM). It applies to any chunked policy at negligible cost and is orthogonal to other policy acceleration methods such as few-step sampling and streaming action decoding, opening a new axis for policy acceleration.
Read the full abstract ▾
All authors Taesung Kwon, Jangho Park, Sunwoo Park, Youngmin Kim, Seonghyun Jin, Youngjun Jun, Kyumin Choi, Jong Chul Ye
Computer Vision
ReSight-SMC: Two-Stage Power Sampling via Island SMC with Visual Scouts
Yaowen Zhang, Xiangyu Qiu, Junyi Hu et al. (+5 more)
Power sampling has emerged as a training-free approach to LLM
reasoning, eliciting capabilities comparable to reinforcement learning by sharpening the model distribution over complete responses. Despite this success, power sampling remains underexplored in large
vision-
language models (LVLMs). We transfer Power-SMC to LVLM decoding by defining a sequence-power target conditioned on both the image and the
prompt. This direct transfer provides a strong training-free baseline, but leaves two aspects of finite-particle
multimodal inference unaddressed. At the particle level, global resampling can collapse genealogies, while particle-based power sampling does not diversify trajectories through distinct visual cues in
multimodal decoding, limiting exploration under a finite particle budget. At the answer level, sequence-level sharpening makes distinct
reasoning trajectories compete even when they support the same answer. We introduce ReSight-SMC, a verifier-free two-stage power sampler for LVLM inference. Its first stage uses ancestry-isolated SMC islands to preserve independent trajectory families and routes a bounded set of prefix-conditioned visual scouts to prefix-relevant image regions while discouraging redundant overlap. Each scout temporarily increases
attention to the image
tokens and emphasizes its routed region. Exact importance correction preserves the base LVLM sequence-power target. The second stage aggregates terminal importance mass by canonical answer, powers the answer marginal, and samples an answer together with a supporting trajectory. Across four LVLM backbones and five
benchmarks, ReSight-SMC achieves stronger aggregate performance than Power-SMC over both the
reasoning and perception
benchmark groups. Without post-training, it remains competitive in aggregate with backbone-matched models trained using reinforcement learning.
Read the full abstract ▾
All authors Yaowen Zhang, Xiangyu Qiu, Junyi Hu, Zhi Lu, Wenwen Tian, Aoqin Wang, Junhai Luo, Zhenming Peng
Computer Vision
FILIGREE3D: Scaling Sparse Latent Flow Matching for Ultra-High-Resolution Image-to-3D Generation
Hongjie Li, Xinran Yang, Xiuchao Wu et al. (+2 more)
Scaling image-to-3D generation to ultra-high resolutions requires controlling rapidly growing computational costs without sacrificing fine
geometric detail. We present Filigree3D, a sparse latent flow-matching framework that generates 3D geometry from a single image at voxel resolutions up to $2048^3$, with straightforward extensibility to $4096^3$. To make training tractable, we introduce Structure-Aware Sparse Scaling, which combines spatial bounding with alternating local-global
attention to constrain
token growth while preserving both fine-scale details and long-range structural context. To enhance detail reconstruction, we curate training samples based on their high-resolution
geometric gains and inject multi-scale image features into a sparse 3D DiT, effectively coupling structural semantics with fine-grained visual cues. Furthermore, a visibility-aware voxel regularization strategy improves robustness against sparse perturbations and facilitates the completion of unobserved geometry. Under our default configuration, Filigree3D maintains peak GPU
memory consumption within practical limits for contemporary hardware, enabling the generation of highly intricate 3D geometry in approximately one minute. Extensive experiments demonstrate that our method yields substantial improvements in overall
geometric fidelity and fine-detail preservation compared to existing baselines, validating practical, detail-preserving 3D generation at unprecedented resolutions.
Read the full abstract ▾
All authors Hongjie Li, Xinran Yang, Xiuchao Wu, Jiangjing Lyu, Chengfei Lv
Computer Vision · Cross-listed from cs.IR
ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport
Zhuchenyang Liu, Ziyi Wang, Yao Zhang et al. (+1 more)
Multi-vector retrievers built on
vision-
language models lead visual document
retrieval (VDR), but they run a multi-billion-parameter query encoder on every search. Distilling this encoder into a small student that queries the teacher's existing index would remove the bottleneck. The standard recipe, however, matches the teacher's MaxSim scores and so requires encoding and caching every training page, which can reach terabytes of page
tokens. NanoVDR avoids pages entirely by training on the teacher's query
embeddings alone, but only for single-vector retrievers. We present ColNanoVDR, to our knowledge the first framework to bring this document-free
distillation to multi-vector VDR. Its objective, OTW (Optimal Transport with Learned Weights), aligns the student's query
tokens with the teacher's by entropic optimal transport, with a learned weight for each student
token, and needs no correspondence between the two tokenizations. We prove that the resulting
alignment cost bounds the MaxSim score difference on every page. Distilled from five state-of-the-art teachers, the 149M text-only students retain about 95% of their teachers' NDCG@5 on ViDoRe v1-v3 while encoding queries up to 26x faster. Under identical training, OTW matches score
distillation while encoding no page and reading 12.6x less cached teacher data.
Read the full abstract ▾
All authors Zhuchenyang Liu, Ziyi Wang, Yao Zhang, Yu Xiao
Computer Vision
Role-Guided MOE for Encoder-Level Pathology Representation Learning in WSI Classification
Xinyu Ma, Xing Yang, Hongtao Jin et al. (+4 more)
Whole slide image classification is a fundamental task in computational pathology, where patch representation quality directly affects downstream aggregation and slide-level discriminability. Pathology foundation models are widely adopted as frozen feature extractors for WSI classification; however, their fixed encoders may produce representations insufficiently adapted to target-specific tissue patterns and discriminative cues. Fine-tuning can improve target adaptation, but introduces a trade-off between pathology-specific representation capacity and adaptation efficiency, particularly in data-scarce settings. To address this, we propose a pathology role-guided mixture-of-experts feed-forward network (MoE-FFN) framework for efficient encoder-level representation learning. We design a two-stage training paradigm to establish and adapt pathology-aware expert specialization. In source-domain expert initialization, pathology-specific priors are distilled from a frozen Virchow2 teacher into a lightweight DINOv2-small student, while role prototypes serve as weak pathological anchors to encourage distinct expert functions. MoE-FFN blocks are introduced into selected high-level
transformer layers to provide transformation diversity for heterogeneous pathological patterns. In target-domain adaptation, the initialized experts are refined through asymmetric prototype-guided optimization, enhancing task-relevant positive evidence and separating confusable hard negatives. The resulting encoder extracts offline patch representations that can be directly integrated with standard MIL aggregators. Experiments on the public BRACS
dataset and a private PAROTID WSI
dataset across five representative backbones demonstrate consistent improvements over the strongest baseline.
Read the full abstract ▾
All authors Xinyu Ma, Xing Yang, Hongtao Jin, Guoquan Zhang, Shijie Zhang, Yu Zhang, Xitong Li
Computer Vision
Narges Norouzi, Niccolò Cavagnero, Idil Esen Zulfikar et al. (+3 more)
Existing online
video segmentation methods struggle to track objects in long, complex
videos with long-term occlusions. We hypothesize that this limitation is caused by (i) the inability of their temporal propagation mechanism to adaptively select the object information that is propagated across time, and (ii) their inability to be trained on long
videos due to
memory requirements and vanishing gradients. To address the first limitation, we propose to use a lightweight GRU-based temporal propagation module that can learn to select which information it keeps in
memory and propagates across time. Second, to allow training on long
videos, we introduce Truncated Query Propagation (TQP), a training strategy in which the model processes a
video in chunks of frames, where information about tracked objects is propagated between chunks but backpropagation is only conducted in individual chunks, enabling longer temporal supervision without out-of-
memory issues, inference overhead, or vanishing gradients. The resulting model is called the Long-term
Video Mask
Transformer (LVMT). Extensive experiments on six
benchmarks show that LVMT sets a new state of the art across a range of
video segmentation tasks, while retaining the speed of the highly efficient model it is based on, making it 10X faster than the prior state of the art. Code: https://github.com/tue-mps/lvmt
Read the full abstract ▾
All authors Narges Norouzi, Niccolò Cavagnero, Idil Esen Zulfikar, Bastian Leibe, Gijs Dubbelman, Daan de Geus
Computer Vision
ECHO: Event-Augmented Context with Hindsight and Outlook for Wrist-Only Manipulation
Xinyue Wang, Yicheng Jiang, Zesen Gan et al. (+8 more)
Learning-based manipulation policies relying on RGB cameras often suffer from degraded observations under extreme exposure. Event cameras mitigate this degradation by asynchronously detecting pixel-level intensity changes to offer a high dynamic range. However, their observations heavily depend on camera placement, as fixed cameras miss static scene content while wrist-mounted camera motion causes previously visited regions to leave the field of view. To address these spatial-temporal limitations, we present ECHO (Event-augmented Context with Hindsight and Outlook), a wrist-only latent world action model that encodes wrist events into compact motion representations to provide temporal and spatial context for policy
reasoning. Specifically, ECHO utilizes a pretrained event encoder to explain visual-feature changes between frames. Its hindsight module preserves the gripper trajectory with past event stream as addressable off-camera context. Concurrently, the outlook module introduces learnable event foresight queries supervised to anticipate the event window for future actions, enabling the policy to predict upcoming scene changes. Evaluated on wrist-only RLBench tasks, ECHO outperforms RGB and RGB+event baselines by 20.6 and 12.0 percentage points under normal lighting, and by 14.6 and 11.3 points under severe exposure drops, respectively, while also surpassing RGB references using a third-person camera. Real-world experiments with a wrist-mounted event camera validate that ECHO outperforms RGB-only and RGB+event baselines across multiple tasks under both nominal and severely dark lighting. Project page is at https://echo-wam.github.io/.
Read the full abstract ▾
All authors Xinyue Wang, Yicheng Jiang, Zesen Gan, Junhao He, Jiaxu Wang, Junhao Li, Jingtao Zhang, Tianlun He, Jianan Wang, Isabel Guan, Qiming Shao
Computer Vision
SubRot: Signed Gradient Subspace Calibration for VLM Rotation Quantization
Zhenhao Shang, Haizhao Jing, Haokui Zhang et al. (+4 more)
Post-training quantization reduces the deployment cost of
vision-
language models (VLMs), but preserving
multimodal capabilities at low bit widths remains challenging. Existing methods rely on modality- or
token-level gradient statistics, which are susceptible to cross-sample variations in visual-to-textual
token ratios and the positions of visual information, limiting statistical stability. Moreover, overly coarse aggregation through absolute values and averaging discards gradient signs and channel-wise differences, limiting the separation of modality-specific sensitivities. In contrast, the channel space provides a shared coordinate system across samples, making it a more natural basis for capturing stable task-sensitive structures. We therefore propose SubRot, a signed gradient subspace calibration method for VLM rotation quantization. Through eigendecomposition of the empirical Fisher
matrix of activation gradients, SubRot identifies a sensitive channel subspace with three properties: cross-sample stability, clear sensitivity separation, and consistent signed effects on the autoregressive loss along certain directions. Guided by a local Taylor expansion, SubRot combines signed first-order guidance along sign-stable directions with second-order constraints along the remaining sensitive directions, while retaining MSE for overall reconstruction quality. This objective steers quantization errors toward loss-decreasing directions while controlling their magnitude. Experiments on five VLMs across five
benchmarks show consistent average-score improvements over FlatQuant under W4A6 and W4A4, reaching 1.4 percentage points on LLaVA-NeXT-7B. Under W4A4, average accuracy degradation from FP16 remains within 1.4 percentage points across all evaluated models, while LLaVA-v1.5-13B exceeds its FP16 average score by 0.4 percentage points.
Read the full abstract ▾
All authors Zhenhao Shang, Haizhao Jing, Haokui Zhang, Guoting Wei, Rong Xiao, Jianqing Gao, Peng Wang
Computer Vision
SPOC-Net: Single-Primitive Online Composition Network for GNSS Jamming Set Recognition
Zhihan Zeng, Kaihe Wang, José A. López-Salcedo et al. (+2 more)
Reliable positioning, navigation, and timing support intelligent transportation, autonomous systems, and space-air-ground integrated networks. However, global navigation satellite system (GNSS) jamming recognizers that treat each mixture as a separate class are difficult to extend to new combinations. Therefore, this paper proposes SPOC-Net, which decomposes the recognition problem into identifying a set of basic jamming components. Multi-resolution time-frequency features and learned component queries provide evidence for each component type. A high-resolution branch estimates the number of active types, and a structured decoder combines this estimate with component evidence to select a valid set. For training, measured single-component records are the only physical samples used in gradient optimization. Their associated clean in-phase and quadrature (IQ) sequences are combined on demand during training to produce labeled mixtures with different relative powers and jamming-to-noise ratios. Separate measured mixtures from ten training-listed compositions support model selection and decoder calibration; six other compositions are reserved for final testing. Evaluation on 14,220 independently generated, conductively combined, and recorded
radio frequency mixtures yields 80.69% exact-set accuracy and a 92.84% micro-averaged F1 score. On combinations excluded from model development, SPOC-Net achieves 80.89% exact-set accuracy, exceeding the strongest comparison method by 18.77 percentage points under the reported protocols.
Read the full abstract ▾
All authors Zhihan Zeng, Kaihe Wang, José A. López-Salcedo, Gonzalo Seco-Granados, Zhongpei Zhang
Computer Vision
P4Q: Co-designing Token Pruning and Quantization for Vision-Language Model Acceleration
Haizhao Jing, Zhenhao Shang, Haokui Zhang et al. (+2 more)
Vision language models have achieved strong performance across a wide range of
multimodal applications, yet their substantial computational and
memory costs hinder efficient deployment. Visual
token pruning and post-training quantization reduce inference overhead along two complementary dimensions, namely sequence length and numerical precision. Existing workflows typically optimize these techniques independently or apply them sequentially. Their distinct optimization objectives leave critical interactions unaddressed and constrain the achievable compression performance. We revisit these designs and present P4Q, a practical co-design framework that jointly optimizes visual
token pruning and low-bit quantization for efficient VLM inference. First, P4Q introduces a quantization-aware visual
token selection strategy before the LLM. It applies fake quantization to copies of the features produced by the projector and selects visual
tokens using statistics computed from these fake-quantized features, thereby conditioning the selector's feature-based decisions on simulated low-bit perturbations. Second, P4Q introduces a
pruning-aware quantization calibration strategy. It uses the same selection strategy as
pruning to calibrate the quantized model on the retained-
token distribution, thereby aligning the calibration process with the pruned execution path used during deployment. By coupling these two components, P4Q achieves substantial inference speedups while maintaining comparable task performance, resulting in a better efficiency-accuracy trade-off than independently optimized pipelines. For instance, on LLaVA-NeXT, P4Q achieves an average end-to-end inference speedup of 2.8x across eight distinct test sets, while retaining higher accuracy than prior compression and quantization methods.
Read the full abstract ▾
All authors Haizhao Jing, Zhenhao Shang, Haokui Zhang, Rong Xiao, Peng Wang
Computer Vision
Cilin Yan, Yilun Qiu, Wanyang Zhang et al. (+4 more)
Multimodal large
language models (MLLMs) have approached image segmentation by
reasoning about visual content and predicting target locations. Their generated responses contain
reasoning traces and localization proposals that can serve as working
memory when revisiting the same image and query. Our exploration reveals that MLLMs benefit from using this self-generated working
memory as context, leading to enhanced
reasoning segmentation. Motivated by this finding, we seek to strengthen the backbone model's
reasoning segmentation capabilities by distilling the guidance gained from revisiting prior attempts, enabling it to benefit with or without working
memory at inference time. To this end, we propose
Reasoning Segmenter with Working
Memory (SWiM), a working-
memory distillation framework for
reasoning segmentation. Specifically, SWiM selects rollouts based on segmentation quality to construct working
memory and uses the
memory-conditioned model as a teacher. The teacher provides
token-level distributional supervision along student-generated trajectories, while the student receives only the original image and query. Joint optimization of on-policy self-
distillation and outcome-based reinforcement learning combines working-
memory guidance with direct feedback on segmentation quality. Extensive experiments on
reasoning segmentation
benchmarks demonstrate that SWiM achieves state-of-the-art performance, validating the effectiveness of working-
memory distillation.
Read the full abstract ▾
All authors Cilin Yan, Yilun Qiu, Wanyang Zhang, Rui Zu, Xiaolong Jiang, Jiayin Cai, Yao Hu
Computer Vision
When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model
Minchan Kang, Kyeonghye Park, Seoyoung Cho et al. (+2 more)
Visual
token pruning has been widely studied as a practical approach to reducing the computational cost of large
vision-
language models. However, it struggles to preserve essential visual information, which can lead to substantial performance degradation. In particular, image-based
token selection can overlook task-relevant details, while text-guided
token selection may fail to capture the text–visual relationships needed for complex
reasoning. We find that applying textual guidance too early can limit its ability to identify answer-relevant visual regions, whereas text-to-visual
attention becomes more informative at intermediate decoder depths. This finding motivates our training-free method, which separates early
vision-guided
pruning from deferred text-guided reselection. We first prune visual
tokens using
vision-encoder
attention, retain additional candidates until the decoder midpoint, and then use text-to-visual
attention to determine the final visual-
token set. Across eight
benchmarks and three models, our method outperforms the best-performing baselines by an average of 11.10 and 16.84 percentage points in performance recovery at 80% and 90%
pruning, respectively, with comparable or lower LLM-prefill latency than most baselines. The source code is publicly available at https://github.com/kmc3661/DeFT
Read the full abstract ▾
All authors Minchan Kang, Kyeonghye Park, Seoyoung Cho, Daeshik Kim, Yucheol Cho
Computer Vision
EviSplat: Preserving Multi-View Evidence in 3D Gaussian Splatting for Open-Vocabulary Segmentation
Sungho Moon, Kota Shimomura, Junwoo Park et al. (+4 more)
Open-vocabulary 3D scene understanding enables object localization and segmentation from free-form text queries without a fixed category vocabulary. Many recent methods build on 3D Gaussian Splatting and consolidate multi-view observations, such as masked crops from individual views, into
language features or compact object descriptors before the query is known. However, observations of the same object vary across viewpoints and are not equally informative: some reveal cues relevant to a particular query, whereas others provide incomplete or misleading evidence. Pre-query consolidation can therefore suppress cues on which a later query depends. We introduce EviSplat, which preserves individual observation features as evidence for later text queries. EviSplat retains individual observation features within class-agnostic 3D instances that represent objects, object parts, or background regions. It also learns, for each Gaussian, a distribution describing which visual appearances its observations support. Given a text query, EviSplat scores each instance using its most relevant observations. It then computes a score for each Gaussian by combining instance-level relevance with locally supported evidence, weighted by how often and how unambiguously that Gaussian was observed. Different queries can thus draw on different visual cues from the same preserved evidence. Experiments across diverse
datasets and evaluation protocols demonstrate state-of-the-art performance, supporting the benefit of preserving multi-view evidence until query time and aggregating it according to the query.
Read the full abstract ▾
All authors Sungho Moon, Kota Shimomura, Junwoo Park, Wonhyeok Choi, Seunghun Lee, Takayoshi Yamashita, Sunghoon Im
Computer Vision
ORAV: Benchmarking Audio-Video Generation from Multimodal Contexts
Jiacheng Hua, Xiaokun Feng, Jiaqi Hua et al. (+3 more)
Audio-
video generation using heterogeneous
multimodal references has emerged as a new challenge, requiring both compositional control over generation and grounded understanding of
multimodal context. In this paper, we introduce ORAV Bench for Omni Reference Audio-
Video Generation, comprising 380 task instances with 2-10 references, 9 semantic roles, and 30 role compositions. Instructions specify the relationships among references; the media supply the identities, dynamics, and audio characteristics to be realized. To evaluate these open-ended outputs, we develop a reference-aware pairwise protocol that prepares visual and auditory evidence, compares the intended contribution of each reference, and checks the overall verdict in both presentation orders. On held-out instances, it achieves 86.08% effective agreement with human judgments. Across 5 frontier systems, overall rankings conceal distinct strengths across reference compositions. A recurring failure is to reproduce unintended source content in place of the requested result, despite closely resembling a reference. Reproducible pointwise diagnostics of quality, reference affinity, and
speech reveal distinct dimensions of model behavior. ORAV thus offers a
benchmark for tracking progress toward controllable, compositional, and reference-faithful audio-
video generation.
Read the full abstract ▾
All authors Jiacheng Hua, Xiaokun Feng, Jiaqi Hua, Chang Liu, Biao Wang, Miao Liu
Computer Vision
Transform-Aligned Learned Features for Lossy Point Cloud Attribute Compression
Yueru Chen, Pengpeng Yu, Dingquan Li et al. (+3 more)
Transform-based methods provide an effective framework for point cloud attribute compression by representing attributes as transform coefficients. Introducing learned spatial context into this framework requires mapping spatial representations to the transform domain, but this known basis change is often left for the network to learn implicitly. We propose Transform-Aligned Learned Features (TALF) by applying the attribute transform to learned spatial representations, explicitly aligning them with the coding targets. Our analysis shows that the resulting features exactly represent the first-order prediction term of a smooth nonlinear model, with a bounded Taylor remainder. We integrate TALF into a transform-based attribute codec with explicit coefficient prediction and conditional residual
entropy modeling under a unified coefficient-domain rate–distortion objective, while retaining explicit quantization-step control. Extensive experiments across three
benchmark datasets and multiple transform bases demonstrate that TALF improves rate–distortion performance over conventional and learned baselines.
Read the full abstract ▾
All authors Yueru Chen, Pengpeng Yu, Dingquan Li, Wei Gao, Wei Zhang, Fei Song
Computer Vision
Multi-Scale Semantic Mapping in Urban Environments via Observation Calibration and Policy Dependence Regularization
Runling Long, Junhao Feng and Jia Wan
Semantic mapping is fundamental to embodied navigation, yet existing methods are developed for indoor environments, where objects exhibit relatively limited scale variation and are observed from a restricted range of viewpoints. Urban environments pose substantially greater challenges:
agents must map objects ranging from pedestrians to buildings while navigating large spaces with highly diverse viewing distances. These conditions introduce two key difficulties that existing
datasets and methods fail to cover. First, object scale and observation distance can be severely mismatched. For example, small objects may be viewed from far away, whereas large objects may be observed at extremely close range, resulting in unreliable observation likelihoods. Second, objects with substantially different sizes and geometries require distinct mapping behaviors, which are difficult to capture with a single shared value
estimator. To investigate these challenges, we introduce a large-scale urban semantic mapping
dataset featuring realistic city layouts, high-fidelity rendering, and instance-level annotations spanning multiple object scales. We then propose a category-aware likelihood calibration policy that identifies and alleviates unreliable observations according to object category and viewing distance. Because the calibration and motion policies are optimized toward the same mapping objective, they may learn redundant shortcuts and become excessively coupled. We therefore introduce a mutual-information (MI) regularizer that penalizes their estimated representation dependence and encourages complementary behaviors. To better model heterogeneous mapping strategies across object scales, we further employ category-wise value
estimators. We formulate their joint optimization as a Pareto optimization problem to mitigate conflicting gradients across categories.
Read the full abstract ▾
Computer Vision
WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoning
Yuheng Zha, Yilei Wang, Qiyue Gao et al. (+5 more)
Humans often solve spatial problems by mentally simulating visual transformations. In contrast, conventional
vision-
language models (VLMs) reason primarily through
language. We investigate whether VLMs can solve spatial problems by
reasoning with both text and generated visual states. To this end, we introduce WM-VLM, which equips a pretrained VLM with a lightweight world model branch for generating intermediate visual states. Our two-stage training first teaches the model to generate the next visual state and then to use that state for
reasoning. We programmatically construct spatial
reasoning tasks with verifiable intermediate visual states. These tasks allow us to evaluate how well the model generates visual states and how much it relies on them to answer the question. On 2D and 3D mental rotation tasks, WM-VLM consistently outperforms the supervised fine-tuned backbone, with gains of up to 39.25 percentage points. Ablations suggest that these gains depend on the generated visual states, as removing or corrupting them sharply reduces performance. Together, these results suggest that internal world models offer a promising path toward VLMs that reason in both
language and visual space.
Read the full abstract ▾
All authors Yuheng Zha, Yilei Wang, Qiyue Gao, Junrong Chen, Yujia Wu, Zhengfeng Lai, Zhengzhong Liu, Eric P. Xing
Computer Vision
Generative Residual Factorization
Letian Gong and Yuzhou Hong
Under a shared-factor model, the conditional law of the next image patch factors into a posterior over the shared scene factor and a residual kernel given that factor. A sufficient statistic of the past replaces the raw past in the posterior and does not replace the kernel. The conditional
entropy splits into residual
entropy, which no observation of the factor can remove, and a posterior term, which a better representation of the past can remove. Next-
embedding prediction is a directional likelihood on a shallow map, so the fiber of that map is unidentified and a constant
embedding remains a minimizer. The same split is an equality in a scalar Gaussian model, evaluated in closed form.
Read the full abstract ▾
Computer Vision
ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wild
Hongyu Ma, Hairong Qu, Shiqi Zhao et al. (+2 more)
Human dexterity is guided by two eyes watching two hands: binocular
vision supplies the metric 3D structure that fine-grained manipulation consumes. Egocentric stereo is therefore the natural perceptual interface for
robots, AR, and VR-yet metric 3D hand reconstruction from this very signal still has neither an end-to-end model nor an in-the-wild
benchmark. We propose ESTHER, a model whose stereo geometry, temporal
reasoning, and output representation are designed for wearable egocentric stereo. It is trained on pseudo-labels from a calibrated labeling pipeline and in turn assembles our
benchmark ESTHER3D, an egocentric stereo hand
dataset pairing a large in-the-wild training set of model-generated labels with a motion capture test set of true metric ground truth. Experiments show state-of-the-art accu?racy, superior external generalization, and robustness to the missing views, dropped frames, and lighting and motion blur extremes of real egocentric capture that break existing meth?ods. This robustness runs deeper than graceful degradation: stereo guidance teaches the model to bind apparent hand scale to metric depth, so it not only adapts to different stereo rigs and modalities with minimal fine-tuning, but more strikingly preserves true metric scale even after collapsing to a single monocular view.
Read the full abstract ▾
All authors Hongyu Ma, Hairong Qu, Shiqi Zhao, Yongsong Yang, Peng Yin
Computer Vision
From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
Rong Yu Xu, Prayag Tiwari and Shaolei Zhang
Vision-
language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit
reasoning and that intervening on the corresponding states can affect the answer. During
reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop
reasoning at that point. On MMStar and RealWorldQA, this reduces mean
reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating
reasoning computation.
Read the full abstract ▾
Computer Vision
ControlTrace: Recovering Control Fields for Hidden-Content Recognition
Zijian Liu, Yaoguang Chen, Liwei Liu et al. (+4 more)
Spatially conditioned
diffusion models can embed words and contours in natural-looking images, but
vision-
language models (VLMs) may fail to recognize the hidden content. Transformation-based recovery depends on parameter and view selection. To evaluate hidden-content recovery and recognition, we construct FreqBlind, a 6,000-image
benchmark spanning contours, real words and non-words across three conditioning strengths. The evaluated transformation-based methods show limited recognition of contour patterns and weakly conditioned hidden content. To address this limitation, we propose ControlTrace to recover the grayscale control field used during generation. An 8.4M-parameter U-Net predicts this field from the carrier image, and a VLM then identifies its content. With Qwen2.5-VL-7B-Instruct, ControlTrace achieves 60.2% open-ended contour recognition accuracy across the three conditioning strengths, exceeding the best of the three evaluated prior methods by 26.9 percentage points. On an A100 GPU, the complete pipeline adds only 7.4 ms (5.3%) to direct VLM inference. Recovered fields have lower pixel errors and higher structural similarity than the evaluated transformation views. Across four evaluated VLMs, ControlTrace retains its overall contour recognition advantage. Recognition remains stable under the tested JPEG compression, Gaussian noise and downsampling. These results support control-field recovery for hidden-content recognition in the evaluated setting.
Read the full abstract ▾
All authors Zijian Liu, Yaoguang Chen, Liwei Liu, Weixi Wu, Hanming Zhang, Jiashui Wang, Na Ruan
Computer Vision · Cross-listed from cs.CL
InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision
Guanghao Zhu, Zeyu Liu, Zhitian Hou et al. (+9 more)
Recent medical
multimodal models have benefited from larger corpora, broader modality coverage, and stronger
reasoning-oriented training, yet effective data design across continued pretraining (CPT) and post-training remains challenging. Medical sources vary substantially in structure, granularity, and information density, and their utility shifts as training progresses from broad knowledge acquisition to late-stage consolidation. Meanwhile, post-training is often dominated by short-form visual question answering, providing limited supervision for informative and answer-consistent explanations. We introduce InfiMed2, a family of 4B and 27B generalist medical
multimodal foundation models built around stage-aware data design. We curate a 55.68B-
token corpus that combines broad clinical knowledge with context-rich biomedical visual evidence through source-specific processing. Our CPT pipeline first adapts the
vision encoder, then builds broad medical knowledge, and finally transitions to an evidence-focused data mixture during learning-rate decay. For supervised fine-tuning (SFT), we regenerate visual question-answering responses using answer stability, answer-masked reconstruction, and correctness-constrained selection to produce more informative and answer-consistent supervision. The 4B model is further optimized with reinforcement learning with verifiable rewards (RLVR). Across five medical
multimodal benchmarks, InfiMed2-4B achieves 66.73% mean accuracy after RLVR, surpassing the larger Qwen3.5-9B, while InfiMed2-27B reaches 73.72%, the highest among the evaluated open-weight models.
Read the full abstract ▾
All authors Guanghao Zhu, Zeyu Liu, Zhitian Hou, Pengkai Wang, Zhijie Sang, Shuo Cai, Yang Yu, Yuanyi Wang, Yanggan Gu, Congkai Xie, Jianmin Wu, Hongxia Yang
Computer Vision
Physics-Guided Spectral Distillation for Underwater Image Enhancement on Resource-Constrained Devices
Yifan Chen, Kai He, Ye Zheng et al. (+3 more)
Underwater image enhancement is crucial for improving visual perception in marine applications. Existing underwater image enhancement studies mainly focus on enhancement quality and visual fidelity, while rarely considering real-time deployment capability, which is essential for resource-constrained underwater
robots. To this end, we introduce a physics-guided spectral
distillation (PSD) method, which reduces model capacity for real-time applications while maintaining the high performance of underwater image enhancement models. To decompose the outputs of teacher and student models, PSD adopts a multilevel Haar discrete wavelet transform. It transfers low-frequency color and illumination information as well as high-frequency structural details through band-specific objectives. Moreover, the
distillation process of PSD is degradation-aware. We estimate degradation-aware weights through a physical head and combine them with ground-truth-guided reliability masks to selectively retain valuable teacher guidance. Experiments on the UIEB, LSUI, and EUVP
datasets validate the effectiveness of the proposed method. Furthermore, we demonstrate the benefits of enhanced images for downstream perception tasks, including object detection. Deployment on a self-developed ROV further demonstrates its practical applicability in real-world underwater scenarios.
Read the full abstract ▾
All authors Yifan Chen, Kai He, Ye Zheng, Jijun Lu, Zhe Sun, Tao Chen
Computer Vision
D$^2$-VLA: Dual-Memory Dual-Frequency Vision-Language-Action Model For Long Dynamic Manipulation
Zijian Ye, Chengqi Wei, Wei Huang et al. (+9 more)
Long-horizon manipulation requires
robots to remember cues that are no longer in view while responding to moving objects. Yet
vision-
language-action (VLA) policies often rely on the latest observation, and refreshing their visual context typically requires another costly
vision-
language model (VLM) pass. We present D$^2$-VLA, which combines dual
memory and dual-frequency control at the KV-cache interface of a pretrained VLA. D$^2$-VLA uses block-wise causal KV caching to encode observations incrementally and, guided by distinct temporal
attention patterns, constructs separate historical KV read views for the VLM and action expert. Between periodic VLM updates, a gated adapter incorporates fresh visual features into the latest history-conditioned KV block, while a short fast-
memory queue supports action replanning. We introduce DOMINO-Long, a ten-task
benchmark requiring
robots to use earlier visual cues when manipulating moving objects. D$^2$-VLA achieves complete-task success rates of 29.3% on DOMINO, compared with 9.6% for $\pi_{0.5}$ and 17.2% for PUMA, and 60.0% on DOMINO-Long, compared with 35.4% and 20.6%, respectively. It improves success rates on eight real-
robot tasks and reaches 97.5% on LIBERO-Long and 74.3% on RoboTwin 2.0.
Read the full abstract ▾
All authors Zijian Ye, Chengqi Wei, Wei Huang, Anlin Zheng, Chunyu Zou, Liangyu Wu, Zikang Zhao, Zhenjie Peng, Yushuo Yang, Shuman Zhao, Zhongrui Wang, Xiaojuan Qi
Computer Vision
Projective Normal Fields: A Convex Optimization Method for Constructing Smooth UDFs
Jiayi Kong, Chen Zong, Fei Hou et al. (+3 more)
Constructing a smooth approximation of an unsigned distance field (UDF) from a raw point cloud is challenging because the input provides neither surface connectivity nor consistently oriented normals. Methods that directly learn a scalar UDF must also handle its non-differentiability on the zero level set and weak supervision away from the samples, which can lead to unstable optimization and spatial artifacts. We introduce Projective Normal Fields (PNFs), an orientation-free representation and convex optimization framework for estimating bidirectional normals from point positions alone. Each normal axis is encoded by a rank-one projector, which is invariant to normal reversal. We relax the non-convex set of hard projectors to its convex hull: the symmetric positive-semidefinite
matrices with unit trace. Each soft tensor defines a local quadratic distance model and retains the relative weights of candidate normal axes. We estimate a coherent PNF by combining local tangent-plane fitting, soft-PCA anchoring, and overlap regularization on a fixed neighborhood graph. With positive anchoring weights, the objective is strongly convex and admits a unique global minimizer. Principal eigenvectors provide bidirectional normals, while the corresponding eigengaps provide spectral confidence indicators. We use these indicators to select and weight directional sources for heat
diffusion, followed by Poisson integration to construct a regularized UDF approximation. By separating local geometry estimation from scalar-field construction, PNF avoids directly fitting the non-differentiable UDF. Experiments demonstrate reduced sensitivity to neighborhood size, competitive reconstruction under noise and outliers, and improved accuracy near non-
manifold junctions. The project page is available at https://anonymous17777367.github.io/PNF-page/
Read the full abstract ▾
All authors Jiayi Kong, Chen Zong, Fei Hou, Junhui Hou, Wenping Wang, Ying He
Computer Vision · Cross-listed from cs.RO
CoHuB: A Simulation Benchmark for Multi-Humanoid Collaboration
Hyunjin Park, Jebeom Chae, Minwoo Park et al. (+11 more)
Many physical tasks in human environments require collaboration, from assisting a partner to jointly manipulating an object. Yet, existing humanoid
benchmarks largely focus on single-humanoid skills and lack evaluation of multi-humanoid collaboration under egocentric visual observations. We introduce CoHuB (Collaborative Multi-Humanoid
Benchmark), a simulation
benchmark for multi-humanoid collaboration under egocentric visual observations. CoHuB provides 10 tasks, eight with two humanoids and two with three humanoids, spanning diverse collaboration patterns. We also provide synchronized demonstrations collected through a multi-operator VR teleoperation pipeline, in which each operator controls one humanoid from its egocentric view. Experiments with representative visuomotor policies reveal substantial challenges across different forms of coordinated perception and control. CoHuB provides a foundation for developing and evaluating multi-humanoid collaboration policies.
Read the full abstract ▾
All authors Hyunjin Park, Jebeom Chae, Minwoo Park, Sunghyun Park, Hanjun Yoo, Seoyeon Choi, Soochul Yoo, Joohwan Seo, Sarmad Idrees, Jae-Sang Hyun, Jongmin Lee, Roberto Horowitz, Youngwoon Lee, Jongeun Choi
Computer Vision
When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Context
Yuxing Cheng, Yuan Wu and Yi Chang
Vision-
language models (VLMs) can read text in natural scenes, but their predictions may be influenced by the surrounding context. When the printed text conflicts with what the scene suggests, a model may return a more plausible word instead of the shown text. We introduce SceneFaith, a
benchmark of 781 generated scene images for studying this behavior. Each output is classified as Literal, Canonical, or Other, separating faithful transcription from context-consistent rewriting and ordinary recognition errors. Across 15 models from seven families, all models show rewriting on clear images, with rates ranging from 8.45% to 58.51%. Controlled experiments further show that surrounding context matters: removing surrounding scene information reduces rewriting and improves literal accuracy, while changing the scene around the same text patch can also change model outputs. Moreover, weakening the target text with blur increases rewriting. These results show that reliable scene-text recognition requires VLMs to balance visual character evidence with contextual information, preserving clear text while using context mainly when the visual evidence is uncertain.
Read the full abstract ▾