Computer Vision
Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs
Aimon Rahman, Joymallya Chakraborty, Vidit Bansal et al. (+1 more)
Multimodal large
language models have advanced rapidly, but most remain optimized for single-image inputs. Multi-image
reasoning is costly because each image adds many visual
tokens, increasing encoding and
attention demands. Existing techniques that address the above issue, including visual
token pruning,
token merging, visual
token compression, learned resampling, hierarchical compression,
retrieval-augmented visual
reasoning, and
memory-based architectures, have largely been designed for single-image scenarios, where redundancy is primarily spatial and local. Multi-image
reasoning introduces additional challenges: redundancy may occur across images, task-relevant evidence may be sparse or distributed, and aggressive per-image compression can remove details needed for comparison, counting, temporal
reasoning,
retrieval, or evidence grounding. Moreover, existing
token-reduction methods are often either (i) training-free but inference-inefficient, typically relying on accessing
attention matrices to get
token-importance that may be impractical to access with
memory-efficient
attention implementations such as FlashAttention, or (ii) training-dependent, requiring the base MLLM to be retrained or adapted to a new compressed-
token paradigm. In this work, we study scalable multi-image
reasoning and understanding without retraining the base model and without accessing
attention matrices at inference time. To this end, we propose mmVB, a lightweight plug-and-play module that assigns query-conditioned per-image
token budgets via cross-image context, adding only 1.6M trainable parameters while reducing visual input
tokens by 9x. This reduction yields 5x faster time-to-first-
token and 7x fewer prefill FLOPs, while maintaining 94.4% of full-resolution accuracy across eight multi-image
reasoning benchmarks.
Read the full abstract ▾
All authors Aimon Rahman, Joymallya Chakraborty, Vidit Bansal, Abhishek Dan
Computer Vision
PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts
Ruixiao Dong, Jianmin Bao, Bozhi Luan et al. (+6 more)
Pixel-space
diffusion models provide an appealing alternative to latent-space
diffusion models by directly modeling raw images without relying on a learned tokenizer. However, scaling pixel-space models has predominantly followed dense strategies, tightly coupling model capacity with active computation. In this work, we show that standard LLM-style Mixture-of-Experts (MoE) architectures can be effectively transferred to pixel-space
diffusion. Building on this finding, we introduce PixelMoE, a pixel-space generative framework that sparsely scales model capacity with standard LLM-style MoE, without
diffusion-specific routing mechanisms or auxiliary routing objectives. Specifically, PixelMoE combines standard
Token-Choice (TC) routing with fine-grained routed experts for input-adaptive sparse computation, a shared expert for capturing common representations, and a bias-based load-balancing strategy. To complement sparse capacity scaling, PixelMoE further introduces Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure formation to fine-grained detail refinement. At comparable active parameter counts, PixelMoE consistently outperforms its dense counterparts across model scales, demonstrating effective sparse capacity scaling in pixel space. On class-conditional ImageNet-1K, PixelMoE achieves FID scores of 1.47 at 256x256 and 1.53 at 512x512 without representation
alignment or perceptual losses, comparing favorably with leading pixel-space and latent-space generative models. Code and models will be released soon.
Read the full abstract ▾
All authors Ruixiao Dong, Jianmin Bao, Bozhi Luan, Zijie Meng, Xin Tao, Pengfei Wan, Kun Gai, Li Li, Houqiang Li
Computer Vision
WorldRecon: Generative Multi-View Mesh Reconstruction for Large-Scale 3D World
Mengchen Zhang, Yang Li, Chunshi Wang et al. (+2 more)
Reconstructing complete, high-quality meshes of large-scale scenes from unposed multi-view images requires faithful
geometric grounding while plausibly completing surfaces that are unobserved or only sparsely captured. We present WorldRecon, a generative multi-view reconstruction framework for recovering complete, large-scale 3D scene meshes from unposed RGB images. WorldRecon uses a coarse, noisy global point cloud and camera poses estimated from the input images by a reconstruction model as a global scaffold for whole-scene generation and partitions it into overlapping spatial blocks. To robustly leverage these imperfect
geometric priors, we equip the native 3D generative model TRELLIS.2 with a dedicated reconstruction-based conditioning architecture that generates each block from its local point-cloud crop and block-relevant image observations, producing sparse structure, detailed geometry, and PBR materials. To maintain scene-level coherence, we introduce neighbour-aware latent inpainting and a causal world-generation procedure that propagates structural,
geometric, and material context across overlapping blocks. Together, these designs scale multi-view generative reconstruction to large-scale scenes, yielding coherent, unified meshes. Extensive experiments demonstrate that WorldRecon achieves state-of-the-art reconstruction performance, reducing Chamfer distance by 14.9% and 43.1% while improving F-score by 17.3% and 12.0% over the strongest baselines on synthetic and real-world indoor scenes, respectively, thereby demonstrating faithful and complete generative reconstruction as well as strong sim-to-real generalization. Project page: https://kszpxxzmc.github.io/WorldRecon-website/.
Read the full abstract ▾
All authors Mengchen Zhang, Yang Li, Chunshi Wang, Tong Wu, Chunchao Guo
Computer Vision
The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement
Sijie Zhao, Zeqi Zhou, Heli Qi et al. (+6 more)
Earth observation (EO) models are still built by hand. For each new task, experts design a training recipe and tune it against a
benchmark. We show that this loop can be handed to an
agent. Any verifiable task can be cast as a harness and a verifier, where the harness defines what the
agent may use and the verifier defines what counts as success. Every EO task can be made verifiable, through labeled
benchmarks, independent sensors, physical models or the repeated observation of the same place. Given only these two components, a general-purpose coding
agent builds the EO model and improves it recursively on its own outputs. Settings long treated as separate research problems, from learning with a handful of labels to adapting across domains, become choices of harness and verifier. On six EO
benchmarks spanning optical, SAR and multispectral time-series imagery, with outputs that range from land-cover maps and ship instances to canopy heights and answers about large images,
agent-built models surpass the state of the art on every one, and on land-cover mapping, they do so from 11% of the training labels. This establishes a new paradigm for EO, in which people design harnesses and verifiers and
agents build the models, so EO models improve as
agents do.
Read the full abstract ▾
All authors Sijie Zhao, Zeqi Zhou, Heli Qi, Xuanchao Fu, Zhuo Zheng, Junshi Xia, Junjue Wang, Weihao Xuan, Naoto Yokoya
Computer Vision
RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark
Mahsa Khoshnoodi and Sarah Adel Bargal
Repetition counting requires identifying and enumerating action cycles over time. While state-of-the-art
video language models describe actions fluently, they fail to reliably count how many times an action repeats. To investigate this limitation, we introduce RepProbe, a
benchmark of 42K real and synthetic
video clips containing exactly k in [1, 20] complete repetitions, with controls for clip position, temporal coverage, and duration. These controls allow us to study the counting failure directly and locate where in the model it arises. Counting accuracy collapses at high counts across the
video language models we evaluate, and controlled experiments show this collapse is not explained by frame sampling alone. We compare model outputs with linear probes trained on the final-layer representations used to generate those outputs. We find a consistent representation-to-output dissociation: substantial count information remains linearly decodable even when models fail to report the correct count. Further analysis shows this failure arises primarily at the output stage, where answers concentrate on a small set of preferred numbers rather than the available count information. The dissociation holds on both real and synthetic
videos and is not explained by clip duration, position, or scene content. These results indicate that models encode repetition structure but do not reliably report it in their outputs. Project page: https://arenaa.github.io/RepProbe-webpage/
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Scaling Online Optimal Transport Minibatches for Flow Matching
Peter Chen and Sergey Levine
A central challenge in scaling flow models is maintaining stability in training and integration. In flow matching, this stability depends critically on how source samples are paired with data. Independent random pairing introduces substantial stochasticity, motivating minibatch optimal transport (OT) to construct online source-target couplings that reduce this randomness (Tong et al., 2024). However, as applications extend to larger and higher-dimensional
datasets, small minibatches can limit these benefits, while scaling the coupling to substantially larger minibatches at every iteration is computationally prohibitive. Moreover, how OT coupling affects the full
stochastic gradient and integration remains theoretically unresolved. In this work, we develop a scalable online entropic OT approximation that enables substantially larger source-target couplings while preserving the benefits of OT. We close the theoretical gap for both unregularized and entropic OT by proving strict reductions in full
stochastic-gradient variance and integration error relative to independent pairing, and further characterize how these improvements strengthen as the online coupling minibatch grows. Experiments on single-
cell gene interpolation-flow and image-flow training validate these predictions, showing that larger online couplings consistently improve training stability and final generation performance.
Read the full abstract ▾
Computer Vision
Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?
Juntong Feng, Qingan Liu, Jingru Dong et al. (+8 more)
Vision-
Language Models (VLMs) have achieved strong performance in visual perception and spatial
reasoning, leading to growing interest in using them as high-level planners for embodied AI. However, physical tasks require models to integrate spatial information, reason over temporal states, and generate multi-step plans under various constraints, raising the question of whether current VLMs can reliably support such
reasoning and planning independently of execution. We introduce Plan-Bench, a
benchmark for evaluating the high-level
reasoning and multi-step planning capabilities of VLMs relevant to embodied worlds, while isolating the influence of action execution. It covers three complementary dimensions, spanning 16 subclasses and 699 curated samples. Extensive evaluation of 28 VLMs reveals room for improvement, with even the strongest model achieving only 75.0% accuracy. We further find that model scaling generally improves these capabilities. Ablation studies also indicate larger models show increasing potential as high-level planners. Overall, Plan-Bench provides a lightweight probe for evaluating VLMs'
reasoning and planning capabilities relevant to embodied environments and assessing their potential as decision makers for physical tasks. Our
benchmark and code are available at https://github.com/MinneFeng/Plan-Bench.
Read the full abstract ▾
All authors Juntong Feng, Qingan Liu, Jingru Dong, Yue Zhu, Weiming Zheng, Yujie Cheng, Xinzhe Han, Shuai Hao, Yunzhi Zhuge, Haiwen Diao, Huchuan Lu
Computer Vision
Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?
Qingan Liu, Juntong Feng, Yue Zhu et al. (+6 more)
Visual
reasoning is an important capability for
vision-
language models (VLMs) to understand visual evidence, spatial structure, constraints, and state changes. Existing evaluations mainly rely on real-world data or procedurally generated tasks. Real-world data preserve natural complexity, while procedural games provide controllable generation and verifiable solutions. However, it remains unclear whether abilities measured in toy games correspond to real-world visual
reasoning and whether game examples can support transfer. We introduce Game2Real-Bench, a
benchmark that studies these connections across four capability groups, including fine-grained perception, spatial
reasoning, constraint
reasoning, and sequential planning. We construct 1,000 procedurally generated game samples with executable verification and curate 400 real-world samples aligned by shared
reasoning requirements rather than visual similarity. We evaluate 25 representative VLMs and observe strong game-to-real consistency, with Pearson and Spearman correlations of 0.935. We further study one-shot game-to-real transfer on four models. Constraint
reasoning improves for several models, while fine-grained perception declines with game demonstrations. We also construct Game2Real-
Dataset with nearly 24K verified game samples across 18 tasks and 65 QA families. Our
benchmark,
dataset, and code are available at https://github.com/QingAnLiu/Game2Real-Bench.
Read the full abstract ▾
All authors Qingan Liu, Juntong Feng, Yue Zhu, Yujie Cheng, Xinzhe Han, Shuai Hao, Yunzhi Zhuge, Haiwen Diao, Huchuan Lu
Computer Vision
OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding
Xuyang Liu, Shihao Wang, Xinyu Wei et al. (+5 more)
With the rapid development of
multimodal large
language models (MLLMs), evaluating their
video understanding capabilities has become increasingly important. Existing
video understanding
benchmarks cover only part of the evaluation dimensions, limiting systematic diagnosis of omni-modal
video understanding. We introduce OmniCompass, a holistic
benchmark for omni-modal
video grounding and understanding. OmniCompass defines two evaluation tracks: OmniCompass-VL assesses
vision-
language models on tasks requiring visual evidence, while OmniCompass-AV assesses omni-modal models on tasks requiring joint audio-visual evidence. Each track includes three complementary tasks: temporal grounding (TG), spatial grounding (SG), and question answering (QA), assessing when a referenced target appears, where it is, and what happens. QA covers five perception and three
reasoning categories in OmniCompass-VL, and three perception and two
reasoning categories in OmniCompass-AV. OmniCompass-VL contains 820
videos and 2,460 instances across domains and three duration types: short, medium, and long. A narrative- and audio-rich subset forms OmniCompass-AV with 240
videos and 720 instances. Each instance is evaluated under two paired settings: text-only and text+image, specifying the same target through text and visual-
language guidance. All instances are manually annotated with multi-round quality verification. Evaluations of 50 MLLMs show that open-source models still lag behind proprietary counterparts, particularly in audio-visual understanding. Precise temporal and spatial grounding remains challenging even for leading models, and their predictions still exhibit inconsistencies across the paired target specifications.
Read the full abstract ▾
All authors Xuyang Liu, Shihao Wang, Xinyu Wei, Minghan Li, Zeqing Wang, Zuhao Yang, Zichen Wen, Lei Zhang
Computer Vision · Cross-listed from cs.RO
WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions
Haoyi Duan, Liliang Chen, Nan Wang et al. (+13 more)
We introduce WorldOracle, a
benchmark specifically designed to evaluate action-conditioned world models (ACWMs) against paired real-world executions. Given the same initial observation and low-level action sequence, WorldOracle asks whether a predicted rollout reproduces what happens in the real world, including physical consequences and task-state transitions. We evaluate models along four complementary dimensions: physical state correctness, interaction spatial
alignment, key interaction area fidelity, and multi-view action coherence. To support this evaluation, we curate real-world training and evaluation suites that cover different interaction dynamics and embodiments. The evaluation suite contains long, multi-segment episodes with both successful and failed executions, enabling evaluation across varied action outcomes and behavior distributions. Experiments on seven representative ACWMs reveal distinct model strengths and failure modes. WorldOracle provides a systematic foundation for measuring and advancing action-conditioned world modeling for manipulation. We will release the
datasets, evaluation code, and leaderboard.
Read the full abstract ▾
All authors Haoyi Duan, Liliang Chen, Nan Wang, Yang Zhang, Yuhang Lu, Xiaotong Chen, Wei-Cheng Tseng, Shenyuan Gao, Zhaoshuo Li, Jingyi Jin, Pooya Jannaty, Dhruv Shah, Yuke Zhu, Ming-Yu Liu, Maoqing Yao, Guanghui Ren
Computer Vision
DuplexBridge: Coordinating Understanding and Generation for Unified Image Editing
Yidi Liu, Jie Xiao, Chaojie Mao et al. (+2 more)
Complex image editing requires faithful task interpretation and accurate execution, but a failed edit does not reveal which responsibility failed. We present DuplexBridge, which uses a unified model's own understanding capability to enhance
prompts for its own generation capability, without an external understanding module. A source-grounded bridge specifies targets, final-state changes, and preservation constraints. Understanding is supervised on fidelity to the original request without observing the generated image; generation is supervised on execution of the condition it actually receives. Reciprocal trust checks selectively admit global task feedback to each role while retaining independently valid local supervision. We introduce CE-Bench, a 1,200-task
benchmark measuring joint satisfaction of task-derived change and preservation requirements. Across three model configurations, the full system improves complete compliance over direct editors by 21.75-25.75 percentage points, with gains also observed on KRIS-Bench, RISEBench, ImgEdit, and GEdit-Bench. From a common supervised Wan-Image initialization, gated joint training reaches 77.08% complete compliance, compared with 70.92% without gates and 75.00% when generation-local supervision instead checks the original task with both gates retained. These results support distinct local responsibilities and selective shared feedback for joint understanding-generation adaptation.
Read the full abstract ▾
All authors Yidi Liu, Jie Xiao, Chaojie Mao, Jingfeng Zhang, Xueyang Fu
Computer Vision
Registration is a Representation Problem: Solving Whole-Body PET/CT Registration
Neel Dey and Adrian V. Dalca
Raw image intensities are a poor surrogate for similarity across images in correspondence problems. Further, while auxiliary segmentation labels can provide semantic identification and improve matching, they can assign only coarse identities to fine-grained anatomy and often cover only a subset of structures, particularly in whole-body settings. We present an approach that instead establishes correspondence using complementary representations of anatomy and demonstrate its utility for longitudinal whole-body PET/CT registration in the Learn2Reg 2026 PSMAReg Challenge. Our 3D foundation model, anatomix, is used to derive fine-grained, per-voxel semantic representations that are consistent across volumes, and we further employ off-the-shelf segmentation models to obtain coarse anatomical identities. Together, these representations drive a GPU-accelerated iterative registration solver (FireANTs) in a hybrid multistage framework: the first stage estimates
alignment from dense multi-channel features, after which a second refines the deformation using both anatomix- and segmentation-based losses. Lastly, because registration should not introduce non-biological tumor volume changes across time, we develop lesion-specific component-wise volume preservation schemes. Without any task-specific training, our system won first place in the PSMAReg Challenge leaderboard. Code and interactive visualizations are available at http://www.neeldey.com/reps-for-reg.
Read the full abstract ▾
Computer Vision
Trust Before Autonomy: Rethinking Agentic Remote Sensing for Crisis Response
Georgios Angelidis, Aggelos Psiris, Spyridon Evangelatos et al. (+4 more)
Agentic remote sensing is evaluated on how many questions it can answer, whereas crisis response depends on whether its answers can be relied on and how much of the decision can safely be handed to the
agent. This paper argues that autonomy should be earned, not assumed: An Earth-Observation (EO)
agent should be granted autonomy in proportion to demonstrated reliability, epistemic honesty and verifiability. The position is supported by an evidential meta-analysis of twelve agentic RS/EO/GIS
benchmark studies from peer-reviewed venues. Only one repeats runs, three contain unanswerable tasks and none measures calibration. Their own tables show that completed workflows are often not correct (task success exceeds tool-call correctness by up to 54pp on identical queries), that the gap to an expert trajectory widens overall with trajectory length (from 7 to 56pp), and that capability does not predict abstention (Spearman r from -0.25 to 0.50, all n.s.). The paper then contributes three EO-specific trust properties, an autonomy ladder for crisis EO tasks, and a readiness protocol that replays archived emergency activations before any live use. In this context, current evidence justifies assistive roles only.
Read the full abstract ▾
All authors Georgios Angelidis, Aggelos Psiris, Spyridon Evangelatos, Thomas Lagkas, Panagiotis Sarigiannidis, Iraklis Varlamis, Georgios Th. Papadopoulos
Computer Vision
Junfeng Ni, Ziyu Zhu, Fan Jiang et al. (+7 more)
Video diffusion models generate visually compelling short-term sequences but often struggle to preserve scene geometry and object structure over time. In this paper, we first demonstrate that current
video attention cannot identify reliable correspondences across frames, with responses dispersed across multiple regions even after recognizable scene structure has emerged. We argue that reliable cross-frame associations are essential for information exchange as scene structure forms in
video generation. Thus, we introduce Geometry
Attention Distillation (GAD), which transfers correspondences from a frozen geometry model's
attention to
video attention. Extracted from clean
videos, these correspondences link the same static surface or moving object across frames. Since structure formation begins in deeper layers at higher noise and shallower layers at lower noise, we apply
attention distillation at a student layer selected according to the noise timestep. After training,
video attention focuses more consistently on corresponding locations. Experiments across all three
video backbones demonstrate improved long-range scene consistency and dynamic temporal consistency, with
geometric gains across generation lengths. On Wan2.2 5B, GAD improves cross-frame matching accuracy by at least 30% over the Wan baseline on both long-range
benchmarks. Notably, GAD achieves these gains without additional inference modules, offering a promising approach to efficient and consistent
video-based world simulation. Project page: https://fantasy-amap.github.io/GAD/.
Read the full abstract ▾
All authors Junfeng Ni, Ziyu Zhu, Fan Jiang, Yu Liu, Chiyu Wang, Jiawei Zhang, Zhaoxu Sun, Mu Xu, Siyuan Huang, Yixin Chen
Computer Vision
The Federated Tumor Segmentation and Classification (FeTS++) Challenge
Wenyi Tang, Carla Pitarch-Abaigar, Siddhesh Thakur et al. (+5 more)
International challenges have become the standard for developing and benchmarking biomedical image analysis methods using centralized multi-institutional data.
Federated learning (FL) describes a paradigm-shift on multi-institutional collaborations, where models are trained across institutions while their data are always retained locally. The
Federated Tumor Segmentation (FeTS) challenge, first held in 2021, was the first challenge dedicated to FL in medical imaging and editions in 2022 and 2024 pushed the frontiers of real-world method validation. FeTS++ 2027 returns after a two-year gap with a substantially larger and more diverse cohort, and with the two "pluses" that name the challenge: an official classification task alongside segmentation, and a move of the infrastructure to NVIDIA
FLARE (NVFlare). Built on the clinically acquired, multi-institutional multi-parametric MRI data of the BraTS 2025 Lighthouse challenge, FeTS++ spans gliomas and meningiomas across patient cohorts from institutions in North America, Europe, Asia, and Africa, and focuses on ranking submitted methodological approaches according to their model utility and an efficiency-aware score. Task 1 asks participants to find the best strategy for aggregating knowledge from per-institution segmentation models, while Task 2 asks them to distinguish glioma from meningioma under the same
federated protocol, in which the entity label is heavily confounded with the site of origin.
Read the full abstract ▾
All authors Wenyi Tang, Carla Pitarch-Abaigar, Siddhesh Thakur, Verena Chung, Konstantinos Koukoutegos, Alexandros Karargyris, Holger R. Roth, Spyridon Bakas
Computer Vision
MIST: Multi-Scale Spatial Context for Gene Expression Prediction from Histology
Koushik Howlader, Tirtho Roy, Md Tauhidul Islam et al. (+1 more)
Predicting spatial gene expression from standard H&E images could extend molecular analysis to tissues without spatial transcriptomic measurements. A standard
Transformer applies self-
attention over spots but ignores where each spot sits in the tissue, whereas a spot's molecular state depends on tissue context at multiple spatial scales. Building on this insight, we introduce MIST, a Multi-context Spatial
Transformer that augments the self-
attention backbone with two context pathways applied in every layer: a coordinate-invariant local pathway that attends over spatial k-nearest-neighbor spots, and a slide pathway that broadcasts a whole-slide summary to every spot; the model predicts all spots in a single forward pass. Because coordinates enter only through pairwise distances, the local pathway is invariant to translations, rotations, and reflections when image features are fixed. In intra-cohort evaluation across six human organ cohorts from HEST-1k, MIST improves mean Pearson correlation over the strongest evaluated baseline by 9.7%, 11.6%, and 11.4% for 10-, 50-, and 100-gene panels, respectively, with the lowest average MSE and MAE. MIST further improves leave-one-organ-out transfer by 6.0% over the next strongest baseline, attains the best expression-derived domain recovery (highest mean ARI and NMI at every clustering resolution, and higher NMI than the closest baseline on 84% of held-out slides), and reproduces spatial expression patterns of selected cancer-marker genes. Together, these results show that adding local and slide context to a vanilla
Transformer backbone is an effective approach to spatial gene expression prediction from histology.
Read the full abstract ▾
All authors Koushik Howlader, Tirtho Roy, Md Tauhidul Islam, Wei Le
Computer Vision
LayerCanvas: Infographic Generation in AI Learning Assistants via Hybrid Layer-Set Diffusion
Keshav Gupta, Jeremy Owen, Christopher Lloyd et al. (+2 more)
Online learning platforms are increasingly deploying AI assistants to provide personalized, on-demand tutoring. These can leverage infographics to communicate complex information and improve the learning experience. However, automating effective infographic generation remains a challenging task, requiring a combination of visual richness with precise typography, structured layouts, and continued editability. Existing representation methods satisfy these goals only in part: Scalable Vector Graphics (SVG) representations preserve geometry, hierarchy, and text but provide limited aesthetic flexibility, while raster
diffusion models produce richer appearances by flattening content and layout into a single image. Conditioning a
diffusion model on a rasterized SVG does not resolve this conflict because the raster preserves approximate placement but discards element ownership, occlusion relationships, and the structure needed to modify and recompose the result. We introduce LayerCanvas, a hybrid approach that retains SVG as the semantic, structural, and typographic program while using
diffusion to aestheticize non-text content. Rather than generating one flattened raster, LayerCanvas partitions SVG elements into z-ordered sets of mutually non-overlapping elements and processes them through communicating parallel RGB generation streams. The resulting layers are independently editable and deterministically recomposed, while text is rendered from preserved vector metadata. Our approach combines rich visual synthesis with accurate typography and localized editing. LayerCanvas substantially improves text fidelity over state-of-the-art raster image generators while achieving the highest overall aesthetic quality. We further demonstrate non-destructive content editing, localization, asset reuse, and animation.
Read the full abstract ▾
All authors Keshav Gupta, Jeremy Owen, Christopher Lloyd, Shaunak Mishra, Manav Sharma
Computer Vision
Context-aware Bayesian Fusion for Multimodal Object Detection
Arunachalam Venkatachalam, Mhafuzul Islam, Marwan Yusuf et al. (+3 more)
Reliable detection of pedestrians and road users is essential for the safe operation of autonomous vehicles. However, this task becomes increasingly challenging under adverse environmental conditions, such as poor weather and low lighting, where the performance of state-of-the-art pre-trained object detectors and on-board sensors tends to degrade, which limits their effectiveness and reliability. Moreover, retraining models for each operating context imposes significant constraints due to high data requirements, computational resources, and time. Considering this, we propose a novel context-aware
multimodal late fusion approach that integrates a lightweight gating network within our probabilistic calibration framework. The gating network extracts environmental context and dynamically modulates the detection outputs of pre-trained object detectors based on the inferred context, without retraining of the detectors. Our methodology is evaluated on a modified KITTI
dataset, in which the camera images and
lidar pointclouds are augmented for varying fog levels. Experimental results show that our approach yields an approximately 14% relative improvement over non-adaptive late fusion baselines, achieving a consistent average precision of 56% across different fog levels.
Read the full abstract ▾
All authors Arunachalam Venkatachalam, Mhafuzul Islam, Marwan Yusuf, David Van Hamme, Ben Stoffelen, James Foresi
Computer Vision
Concealed Handgun Detection using WiFi Signals
Erik Persson, David Gustafsson and Tomas Wilkinson
While conventional camera-based surveillance systems and metal detectors can provide a sufficient level of security when used appropriately, they have fundamental limitations that reduce their applicability. Camera-based systems are unable to detect occluded or concealed threats and metal detectors require each person to enter through a narrow opening one at a time, leading to congestion issues and limited use in large crowds. A WiFi-based security system has the potential to be a complementary modality that is able to handle occlusions or concealment and is not hindered by strict spatial configurations. In this paper, we propose use Channel State Information estimated from WiFi-signals to detect concealed handguns. To this end we collect a
dataset of aligned WiFi signals and RGB images that we call the WiFire
dataset. We also propose a deep learning model called WiFireNet tailored to detect concealed handguns in WiFi signals using
vision-based supervision signals. Our results show that concealed handgun detection is possible in WiFi and has promise as a complementary modality to surveillance cameras and metal detectors. We release the
dataset, model and code to facilitate the further research into WiFi-based handgun detection.
Read the full abstract ▾
Computer Vision
Segment Where You Look: Attention as the Interface Between Language Models and Segmenters
Eslam Abdelrahman, Hamza Omar Alkaf, Yuheng Li et al. (+5 more)
Vision-
language models (VLMs) that segment from
language must pass the location of the referent to a pixel decoder. The dominant interface compresses it into the hidden state of a single [SEG]
token, a d-dimensional vector with no spatial index, and leaves the decoder, which never sees the expression, to recover the referent's extent. We show that the VLM already computes a better signal: the [SEG]
token's
attention over the image
tokens, which is defined on the patch grid by construction. Probing a frozen segmentation VLM, we find that this
attention yields 18 to 22 more gIoU than the [SEG] hidden state when read by the same probe, that it gains nothing from the image where the hidden state does, and that it is concentrated in the late layers. Building on these findings, we introduce AttnSeg, which replaces the [SEG]
embedding with this
attention as the decoder's
prompt. A learned, signed combination of the
attention heads forms a dense spatial
prompt, and an
attention-guidance objective supervises it directly. A two-stage recipe first trains the VLM to localize without the decoder, then trains the decoder to refine, within the budget of standard fine-tuning. Applied to Sa2VA, AttnSeg improves ReasonSeg by 6.4 gIoU and GS-Eval by 10.2 at 4B (4.2 and 9.6 over the same model fine-tuned on the same data), and by 8.8 and 11.3 at 14B. The gains reach 13 to 18 gIoU on object parts, multiple objects, and single objects, where the referent must be singled out, and are negligible on stuff regions, which the decoder already recovers unaided. Code and models are available at https://eslambakr.github.io/attseg_project_page/
Read the full abstract ▾
All authors Eslam Abdelrahman, Hamza Omar Alkaf, Yuheng Li, Siddharth Srinivasan Iyer, Nanxuan Zhao, Eli Shechtman, Krishna Kumar Singh, Mohamed Elhoseiny
Computer Vision
AIRRI: Towards a Human-Readable, Machine-Resistant Reading Interface
Emory Salaun, Syed Ayon, Kaleel Mahmood et al. (+1 more)
The rapid advancement of image-to-text extraction systems and large
language models (LLMs) poses new challenges for educational assessment by making assignment materials easy to transcribe and solve with minimal human effort. To address this problem, we present AIRRI (Artificial Intelligence Resilient Reading Interface), a framework for generating machine-resistant educational content, a
dataset of higher-education assignments for evaluation, and an initial evaluation of the human readability of AIRRI machine-resistant text designs. AIRRI employs two categories of machine-resistance techniques. First, AIRRI applies image transformations including character spacing, word spacing, line spacing, character opacity variation, stripe overlays, and image noise. Second, AIRRI applies decision-based
adversarial attacks: RayS, SurFree, and ADBA. We evaluate AIRRI against three image-to-text systems: Tesseract, EasyOCR, and the
transformer-based GOT-OCR2 model. Our results show that AIRRI substantially degrades automated text extraction performance. AIRRI increased character error rates for EasyOCR (6.9% to 55.9%), Tesseract (2.3% to 100.0%), and GOT-OCR2 (30.7% to 48.1%). Word error rates increased for EasyOCR (24.9% to 96.7%) and Tesseract (10.7% to 100.0%). Among the evaluated decision-based attacks, ADBA consistently produced the largest degradation in image-to-text performance, while RayS and SurFree produced more moderate reductions. EasyOCR and Tesseract were generally more susceptible to both image-based and
adversarial perturbations than the
transformer-based GOT-OCR2 model. Our findings suggest that AIRRI can significantly reduce the reliability of image-to-text-to-LLM workflows, increasing the effort required for AI-assisted assignment completion.
Read the full abstract ▾
All authors Emory Salaun, Syed Ayon, Kaleel Mahmood, Shaun Wallace
Computer Vision · Cross-listed from cs.RO
UniW(A)M: A Unified Mixture-of-Experts Framework for World and World-Action Models
Jingbo Xu, Jianjie Fang, Zhuohang Li et al. (+9 more)
World models (WMs) and world-action models (WAMs) focus on future world prediction and executable action generation, respectively. Although these models learn shared physical dynamics and interaction semantics from embodied trajectories, they are typically trained in isolation, which restricts knowledge reuse between prediction and control. The central challenge is therefore to develop a unified learning framework that shares these dynamics while preserving the distinct modeling capabilities required by each paradigm. We propose UniW(A)M, a mixture-of-experts world-(action) model built upon
diffusion Transformers that unifies action-conditioned world modeling and diverse WAM variants. Shared experts capture common dynamics, while atomic capability experts are reused across interfaces so that each interface keeps its own conditional pathway. At deployment, a plug-in adaptive interface router (PAIR) learns from a small subset of target-task chunk scenarios to select a suitable WAM interface for each execution chunk, balancing task success and execution efficiency while keeping the backbone frozen. We evaluate UniW(A)M on LIBERO, RoboTwin 2.0, and three real-world manipulation tasks, where it achieves 88.3% average real-world success versus 76.7% for the strongest baseline. On WorldArena, it leads action-conditioned world modeling on all three tasks. AC-WM co-training contributes to action generation on every action interface and PAIR raises OOD success from 63.3% to 76.7%.
Read the full abstract ▾
All authors Jingbo Xu, Jianjie Fang, Zhuohang Li, Zihao Song, Zhaolu Wang, Xvyuan Liu, Xin Zhang, Haisheng Su, Yinzhou Tang, Chen Gao, Wei Wu, Yong Li
Computer Vision
CryoTracer: A Synthetic-to-Real System for Filament Instance Tracing in Cryo-EM
Radosław Kuczbański, Wojciech Kozłowski, Mateusz Grzesiuk et al. (+4 more)
Particle picking in cryo-electron microscopy is commonly formulated as an object-detection problem in which individual particles are represented by bounding boxes, a natural representation for globular particles but a poor match for filaments that can extend across an entire micrograph. Existing methods reconstruct filament trajectories by linking local predictions in a separate post-processing stage, decoupling detection from instance formation and creating potential ambiguities at gaps and crossings. Directly preserving filament identity is particularly useful for polymorphic specimens, where segments originating from the same filament should remain associated. However, learning such instance-level representations is challenging because expert centerline annotations are scarce and costly to obtain. We introduce CryoTracer, a synthetic-to-real framework that addresses this limitation through procedural micrograph generation,
transformer-based polyline prediction, and specimen-specific fine-tuning. Procedural pretraining exposes the model to diverse filament geometries, intersections, and imaging conditions before adaptation using a limited number of fully annotated experimental micrographs. CryoTracer represents filaments as ordered centerline polylines and optimizes their geometry using an orientation-invariant discrete Fréchet objective, enabling consistent learning regardless of tracing direction. Evaluation on established and newly expert-annotated
datasets shows that CryoTracer improves instance-level filament recovery over existing particle-detection approaches. These results establish end-to-end polyline prediction as a practical alternative to detection-and-linking pipelines for recovering complete filament instances with limited micrograph annotation budgets. Code and data are available at https://github.com/cryotracer/cryotracer
Read the full abstract ▾
All authors Radosław Kuczbański, Wojciech Kozłowski, Mateusz Grzesiuk, Konrad Karanowski, Lukas Frey, Andrzej Rzepiela, Maciej Zięba
Computer Vision
GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models
Rong Li, Shijie Li, Jiangyong Huang et al. (+7 more)
Vision-
language models (VLMs) struggle with metric and multi-view relationships that remain implicit in RGB observations. Geometry foundation models recover this structure from RGB, but existing methods either fuse the estimated features with visual
tokens and learn to use them from question answering alone, or align VLM latents with geometry features. Capturing geometry, however, does not ensure its use: zeroing the aligned latents of a released feature-
alignment model leaves its accuracy unchanged. We introduce GeoReader, a framework that learns
geometric representations for the VLM that reads them. First, we train lightweight
Geometric Readers to produce continuous input
tokens, supervising them with geometry descriptions decoded by a frozen VLM, so that the Readers express geometry in a form the target model can already read. We then freeze the Readers to preserve this interface while fine-tuning the VLM on spatial QA, with geometry injection conditioned on the question. Geometry descriptions serve only as training targets: at inference, the VLM answers directly from the continuous
geometric tokens without generating a geometry chain of thought. With geometry predicted from RGB images, our approach reaches 94.3 on MindCube-Tiny with Qwen3-VL-4B and, with a multi-source QA mixture, an average of 72.5 across six additional spatial
benchmarks with Qwen3.5-9B. These results suggest that a useful
geometric representation is one the receiving model can use, not merely one from which geometry can be reconstructed.
Read the full abstract ▾
All authors Rong Li, Shijie Li, Jiangyong Huang, Lingdong Kong, Jiaming Zhou, Dongyue Lu, Zeying Gong, Baoxiong Jia, Junwei Liang, Siyuan Huang
Computer Vision
Orientation Estimation of Cryo-EM Images for Molecules with Arbitrary Symmetry by Semidefinite Programming
Yunpeng Shi, Amit Singer, Jinxin Wang et al. (+1 more)
This paper extends the non-unique games framework introduced in Bandeira et al. (2020) to address symmetric structures. Precisely, we consider the problem \[ \begin{aligned} \underset{g_1,\ldots,g_n\in \mathcal{G}}{\operatorname{min}}\,\, \frac{1}{|S|}\sum_{i,j=1}^n \sum_{s\in S} f_{ij}(g_i s g_j^{-1}), \end{aligned} \] where $\mathcal{G}$ is a compact group, $S\subset \mathcal{G}$ is a finite subgroup of
symmetry elements, and $f_{ij}:\mathcal{G}\rightarrow \mathbb{R}$ is a loss function. A notable example is common lines based orientation estimation of cryo-EM images for molecules with point group
symmetry $S$, where $\mathcal{G}$ is the rotation group $\operatorname{SO}(3)$. With a semidefinite programming relaxation, we turn the problem into a convex program that is solved with the alternating direction method of multipliers. Unlike existing approaches which are tailored to specific
symmetry groups, ours provides a unified computational framework for addressing all point group
symmetries. We demonstrate the applicability of the proposed
algorithm on the cryo-EM orientation estimation problem on both simulated and experimental
datasets. In particular, our
algorithm is effective in generating high-quality ab initio models from only a few class averages computed from experimental images.
Read the full abstract ▾
All authors Yunpeng Shi, Amit Singer, Jinxin Wang, Ruiyi Yang
Computer Vision
Just Respect the Semantics: Image-Token Compression via Pre-LLM Semantic Variable-Resolution Tokenization and In-LLM Relevance Pruning
Tianhao Fu, Austin Wang, Charles Chen et al. (+5 more)
Image-
token compression must reduce computation without hurting the semantics. We introduce Varest, guided by semantic fungibility:
tokens within a coherent semantic region can be summarized coarsely, while object boundaries and literal text require finer distinctions. Object and text localizers guide variable-resolution merging, preserving spatial coverage and finer detail at boundaries and text. Learned [REL] registers then prune visual
tokens after early decoder processing. Training across budgets enables one checkpoint per backbone. Across three backbones and 13
benchmarks, Varest outperforms 15 baselines in mean performance retention at every evaluated budget, retaining 86.10-92.33% of full-
token performance with only 5.56% of the original layer-averaged image-
token count.
Read the full abstract ▾
All authors Tianhao Fu, Austin Wang, Charles Chen, Roby Aldave-Garza, Victor Xie, Xiyue Jiao, Yucheng Chen, Martin Tin
Computer Vision
RSIgen: Reciprocal Self-Improvement for Agentic Visual Generation
Yidi Liu, Haozhe Wang, Jie Xiao et al. (+3 more)
Agentic visual generation produces experience that can improve the native understanding (
agent) and generation roles of a unified
multimodal backbone, but task success alone does not determine which role should learn. Generator updates can also make the
agent's action supervision stale. We introduce RSIgen, a task-centric reciprocal self-improvement framework linking learning allocation, supervision refresh, and recursive curriculum construction. Dynamic Responsibility Routing (DRR) uses matched interventions on evidence, inputs, and supported actions to select learning recipients and construct compatible training targets. After generator learning, RSIgen re-evaluates the relative utility of tested actions with branch inputs held fixed to refresh
agent supervision; the retained roles then construct the next curriculum. Across BAGEL, Qwen-Image, and Wan-Image-7B, RSIgen improves AgentGen-Bench Full Overall9 by 3.2–3.3 points over a three-cycle SearchGen adaptation with action costs. Wan controls support allocation gains at matched role-level training exposure, refresh gains at a matched adaptation budget, and benefits from updated curriculum sources while recipients continue learning. In blind evaluation on 200 real-user requests, pooled tie-adjusted preference scores against the same baseline reach 70.0% for task completion and 64.9% for visual quality.
Read the full abstract ▾
All authors Yidi Liu, Haozhe Wang, Jie Xiao, Kai Zhu, Xueyang Fu, Zheng-Jun Zha
Computer Vision
RadarGSOcc: Doppler-Aware 3D Semantic Occupancy Prediction Using 4D Radar Tensor with Geometric-Semantic Feature Extraction
Christofel Rio Goenawan, Dong-Hee Paek and Seung-Hyun Kong
3D semantic occupancy prediction is crucial for autonomous driving, but camera-based methods degrade in low-light conditions and
LiDAR-based methods are sensitive to adverse weather. 4D radar is more robust, yet its sparsity and the high cost of radar tensor processing limit its use. We propose Radar
Geometric-Semantic Occupancy Prediction (RadarGSOcc), an efficient framework that directly processes 4D radar tensors by separately extracting
geometric and semantic features, including Doppler information, with a deformable
attention-based
geometric refinement module as its key component. We also introduce the K-Radar 3DSemOcc
benchmark, the first large-scale
benchmark for 3D semantic occupancy prediction using 4D radar tensors, built on the K-Radar
dataset with refined
LiDAR-based annotations. RadarGSOcc predicts 15 classes within a 51.2m range and achieves 38.2% IoU, 7.6% mIoU, and 25.5% weighted mIoU, outperforming RadarOcc by 17% relative (25.5 vs. 21.9) in weighted mIoU while using fewer parameters and lower computation. It substantially improves truck prediction while trading off car accuracy, outperforms camera-based approaches, and narrows the gap with
LiDAR-based methods. To assess robustness without per-frame ground truth in adverse weather, we further propose a novel proxy metric, Environment-Based KL Divergence (EBKL), under which 4D radar proves a reliable and efficient modality for 3D semantic occupancy prediction. The
dataset and code will be publicly available.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Perception Beyond the Visible: Incompleteness-Aware Scene Understanding for Robot Autonomy
Rohit Mohan
Reliable
robot autonomy requires
reasoning about the world from partial and noisy observations. However, many perception systems model only visible scene regions, rely on fixed semantic categories, and assume reliable sensor measurements. These assumptions break under occlusion, novel object encounters, and sensor degradation. In this paper, I present a research
vision for incompleteness-aware scene understanding. I organize this
vision around three fundamental challenges: incomplete evidence, incomplete knowledge, and incomplete sensor reliability. I discuss my research contributions toward addressing these challenges through amodal perception, semantic occupancy forecasting, open-world scene understanding, label-efficient learning, and uncertainty-guided multi-modal fusion. Building on these contributions, I outline future research directions toward belief-centric 4D scene representations. These representations aim to jointly model geometry, semantics, motion, and uncertainty. I also discuss active sensing and safe adaptation strategies that allow
robots to reduce uncertainty through interaction. The goal is to develop calibrated and actionable scene representations that support reliable autonomy in dynamic, open-world environments.
Read the full abstract ▾
Computer Vision
PlanPath: Exploration-Guided Orchestration for Patient-Level Pathology Reasoning
Jiusong Ge, Mingzhao He, Qidong Liu et al. (+8 more)
A patient's pathology case spans tens of slides from distinct specimens and anatomical sites, each carrying non-redundant, site-specific evidence. Forming a patient-level conclusion therefore requires locating the slides relevant to a given question and integrating complementary findings across them. However, existing pathology
agents analyze one slide at a time and describe morphology rather than measure it, leaving evidence acquisition across a patient's specimens and patient-level quantification underexplored. To address this gap, we propose PlanPath, an agentic framework for patient-level pathology visual question answering (VQA). PlanPath turns the question into an exploration plan that specifies the evidence required, routes to the slides likely to carry it, and drives coarse-to-fine acquisition from tissue architecture down to individual
cells. Tissue- and
cell-level tools supply the counts, proportions, and extents, which PlanPath aggregates across slides into a traceable answer. To enable evaluation, we build two patient-level VQA
datasets for renal and gastric cancer, averaging 15 WSIs per patient, with pathologist-verified question-answer pairs derived from diagnostic reports. Experimental results show that PlanPath outperforms the strongest baseline by 9.4% and 12.1% in overall accuracy on the two
datasets and further improves existing slide-level
agents when plugged into our framework. The
dataset and code are available at: https://github.com/JSGe-AI/PlanPath_1.
Read the full abstract ▾
All authors Jiusong Ge, Mingzhao He, Qidong Liu, Chunbao Wang, Chunze Yang, Di Zhang, Jiashuai Liu, Mireia Crispin-Ortuzar, Huazhu Fu, Chen Li, Zeyu Gao