Computer Vision
SPATIOROUTE: Dynamic Prompt Routing for Zero-Shot Spatial Reasoning
Pawat Chunhachatrachai, Gueter Josmy Faure, Hung-Ting Su et al. (+1 more)
Spatial question answering over
egocentric video is a challenging task that requires
Vision-
Language Models (VLMs) to reason about 3D object positions, scene affordances, and directional relationships, particularly in the zero-shot setting where no task-specific fine-tuning is available. We introduce SpatioRoute, a dynamic
prompt generation approach that routes each incoming question to a semantically tailored
prompt template – without any additional training, fine-tuning, or 3D sensor input. SpatioRoute operates in two complementary modes: SpatioRoute-R, a rule-based router that deterministically maps question typologies (e.g., What, Is, How, Can, Which) to specialized
prompt templates; and SpatioRoute-L, an LLM-driven approach that generates task-specific
prompts from the question and situational context alone, with no
video input at routing time. We evaluate SpatioRoute on the SQA3D
benchmark across VLMs spanning model families. SpatioRoute achieves consistent overall accuracy gains up to 5% over fixed
prompt baselines, establishing a new state-of-the-art for zero-shot
video-only spatial VQA without requiring 3D point-cloud inputs. As an additional finding, we observe that Chain-of-Thought (CoT) prompting, implemented via the Think it Twice architecture, consistently degrades performance in this setting on Qwen series models, confirming that question-aware routing is more effective than uniform
reasoning instructions for spatial
video understanding.
Read the full abstract ▾
All authors Pawat Chunhachatrachai, Gueter Josmy Faure, Hung-Ting Su, Winston H. Hsu
Computer Vision · Cross-listed from cs.RO
RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots
Giorgia Modi, Davide Buoso, Giuseppe Averta et al. (+1 more)
Current approaches to 3D scene graph generation rely on dedicated depth sensors, such as
LiDAR or RGB-D cameras, for metric 3D reconstruction. This limits deployment to specialized robotic platforms and excludes settings where only RGB cameras are available, such as fixed external infrastructure. Existing pipelines also typically operate on passively collected observation trajectories, rather than selecting viewpoints based on the partially built scene representation, and therefore fail to effectively exploit the semantic and spatial information encoded within the graph during exploration. This paper presents a fully visual framework for the active, incremental construction of 3D scene graphs from RGB input only, addressing both limitations. The proposed approach unifies perception and planning around a shared structured representation that captures object semantics, 3D geometry, relational context, and information from multiple viewpoints. Because the framework is hardware-agnostic and relies only on RGB observations, it can incorporate inputs from both onboard
robot cameras and fixed external cameras within the same representation. Experiments on the Replica
dataset show that the RGB-only pipeline achieves F1-score parity with baselines using ground-truth depth. Active exploration experiments on ReplicaCAD further show that semantic-driven viewpoint selection detects more than twice as many objects as a
geometric frontier-based baseline under the same exploration budget. Finally, the external-camera setting demonstrates that complementary RGB views can effectively bootstrap the scene graph and improve contextual understanding at no additional exploration cost.
Read the full abstract ▾
All authors Giorgia Modi, Davide Buoso, Giuseppe Averta, Daniele De Martini
Computer Vision · Cross-listed from cs.AI
Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks
Yajing Zhou and Xiangyu Kong
While Multi-Modal Large
Language Models (MLLMs) demonstrate impressive capabilities in general
reasoning, their
embodied spatial intelligence remains hampered by a "Cartesian Illusion" - a reliance on text-based probability distributions that lack grounded, 3D topological understanding. This limitation is starkly exposed in multi-
agent environments, which demand more than just scene perception; they require second-order Theory of Mind (ToM). Specifically, an
Agent A must be able to infer
Agent B's belief about the environment, governed strictly by
Agent B's physical orientation and sensory limitations. In this paper, we probe the limits of two-stage spatial inference in MLLMs through a novel audio-visual task: requiring
Agent A to predict
Agent B's estimation of A's relative location. To solve this, we propose an Epistemic Sensory Bottleneck module that abandons rigid, rule-based coordinate transformations. Instead, we introduce an Anchor-Based
Embodied Spatial Decomposition Chain-of-Thought (CoT). This guides the MLLM through a "
geometric-to-semantic" projection, forcing it to first establish B's local coordinate system and then dynamically weight visual and auditory modalities based on whether A falls within B's visual frustum. Extensive evaluations reveal that while current MLLMs fundamentally struggle with spatial
symmetry and out-of-view ambiguities (establishing a rigorous zero-shot baseline of 42% accuracy), our sensory-bounded
reasoning chain robustly outperforms pure
egocentric and allocentric baselines. By systematically benchmarking these perceptual bottlenecks, our work exposes the current limits of MLLM spatial
reasoning and establishes a foundational paradigm for epistemic, modality-aware inference in
Embodied AI.
Read the full abstract ▾
Computer Vision
Best Segmentation Buddies for Image-Shape Correspondence
Itai Lang, Dongwei Lyu, Dale Decatur et al. (+1 more)
Finding correspondences is a fundamental and extensively researched problem in computer
vision and graphics. In this work, we examine the underexplored task of estimating
segmentation-to-
segmentation correspondence between images in the wild and untextured 3D shapes. This task is highly challenging due to substantial differences in appearance, geometry, and viewpoint. Our approach bridges the cross-modality gap by linking pixels in the image segment to vertices in the corresponding semantic part of the 3D shape. To achieve this, we first distill deep visual features from a 2D
vision model onto the 3D shape surface, allowing for the computation of feature similarity between image pixels and shape vertices. Then, we identify Best
Segmentation Buddies, vertices whose most similar image pixel lies within the image
segmentation region, enabling the reliable discovery of vertices in semantically corresponding shape parts. Finally, we leverage distilled 3D features from the 2D image
segmentation model to segment the shape directly in 3D, bootstrapping the correspondence process. We demonstrate the generality and robustness of our approach across a wide range of image-shape pairs, showcasing accurate and semantically meaningful correspondences. Our project page is at https://threedle.github.io/bsb/.
Read the full abstract ▾
All authors Itai Lang, Dongwei Lyu, Dale Decatur, Rana Hanocka
Computer Vision
View-Aware Semantic Alignment for Aerial-Ground Person Re-Identification
Quan Zhang, Zeqiang Cai, Peiming Zhao et al. (+4 more)
Aerial-Ground Person Re-Identification (AGPReID) remains highly challenging due to drastic viewpoint variations between drones and fixed cameras. Existing methods typically follow a view-invariant paradigm, aligning shared features across views to achieve robustness. However, view-invariant inherently enforces part-level
alignment, which ignores view-specific cues and discriminative identity information. To this end, this work proposes ViSA (View-aware Semantic
Alignment), a view-aware framework that achieves cross-view semantic consistency containing an Expert-driven
Token Generation Module (ETGM) and a Dual-branch Local Fusion Module (DLFM). Technically, the former constructs a set of view-aware experts to generate adaptive semantic queries that perceive viewpoint-specific patterns, while the latter leverages graph
reasoning to extract and align local regions responsive to different experts. Extensive experiments on three AGPReID
benchmarks including AG-ReID.v2, CARGO and LAGPeR demonstrate that ViSA consistently achieves superior performance, with a notable 10.06% mAP improvement on the challenging CARGO cross-view protocol. The code is available at \href{https://github.com/Cat-Zero/ViSA}{https://github.com/Cat-Zero/ViSA}.
Read the full abstract ▾
All authors Quan Zhang, Zeqiang Cai, Peiming Zhao, Jingze Wu, Cailun Wu, Hongbo Chen, Jianhuang Lai
Computer Vision · Cross-listed from cs.LG
Dual-Rate Diffusion: Accelerating diffusion models with an interleaved heavy-light network
Grigory Bartosh, David Ruhe, Emiel Hoogeboom et al. (+3 more)
Diffusion models achieve state-of-the-art generative performance but suffer from high computational costs during inference due to the repeated evaluation of a heavy neural network. In this work, we propose Dual-Rate
Diffusion, a method to accelerate sampling by interleaving the execution of a heavy high-capacity context encoder and a light efficient denoising model. The context encoder is evaluated sparsely to extract high-dimensional features, which are effectively reused by the light denoising model at every step to refine the sample efficiently. This approach significantly accelerates inference without compromising sample quality. On ImageNet
benchmarks, Dual-Rate
Diffusion matches the performance of standard baselines while reducing computational cost by a factor of $2$-$4$. Furthermore, we demonstrate that our method is compatible with
distillation techniques, such as Moment Matching
Distillation, enabling further efficiency gains in few-step generation.
Read the full abstract ▾
All authors Grigory Bartosh, David Ruhe, Emiel Hoogeboom, Jonathan Heek, Thomas Mensink, Tim Salimans
Computer Vision · Cross-listed from cs.RO
Fixed External Cameras as Common Prior Maps for Active 3D Scene Graph Generation
Giorgia Modi, Davide Buoso, Giuseppe Averta et al. (+1 more)
Commonly available prior information, such as BIM models, floor plans, and remote sensing images, can provide valuable
geometric and semantic context for autonomous robotic systems. In this paper, we treat observations from fixed external RGB cameras as Common Prior Maps (CPMs): wide-field views of the environment that initialize a semantic and
geometric scene prior before any
robot motion begins. We present an RGB-only framework for active, incremental 3D scene graph (3DSG) generation that seamlessly fuses observations from both onboard
robot cameras and fixed external cameras within a single hardware-agnostic pipeline. By relying solely on RGB observations processed by a feed-forward 3D reconstruction model, the system treats all cameras - onboard or external - identically, requiring no hardware modifications. A graph-based active semantic exploration framework then directly leverages the partial scene graph to guide the
robot toward regions of high semantic uncertainty, progressively completing and refining the prior. Experiments demonstrate that bootstrapping the scene graph with even a single external camera increases initial object recall by up to +79%, and that the richer context of the prior significantly improves the efficiency of subsequent active exploration.
Read the full abstract ▾
All authors Giorgia Modi, Davide Buoso, Giuseppe Averta, Daniele De Martini
Computer Vision
Calvin Galagain, Martyna Poreba and François Goulette
Query-based
Vision Transformer segmentation models typically reconstruct dense spatial feature maps to predict masks, inheriting design patterns from convolutional architectures. We show that this explicit image-space reconstruction is not required. We introduce TokenMask, a
token-space mask head that computes mask logits directly from query-
token affinities and performs interpolation in logit space rather than feature space. This reformulation preserves the original linear scoring mechanism while simplifying the computational structure. Across diverse ViT backbones,
datasets and
segmentation tasks, TokenMask consistently improves efficiency over prior approaches by reducing computational and
memory requirements while maintaining competitive accuracy, leading to tangible speedups on NVIDIA Jetson AGX Orin using TensorRT FP16 inference. Overall, TokenMask yields a simpler and more deployment-friendly design for embedded
vision systems.
Read the full abstract ▾
Computer Vision
MARS: Technical Report for the CASTLE Challenge at EgoVis 2026
Haoyu Zhang, Qiaohui Chu, Yisen Feng et al. (+4 more)
This report presents MARS, short for
Multimodal Agentic
Reasoning with Source selection, our system for the CASTLE Challenge at EgoVis 2026. Participants must answer 185 closed-form questions over the CASTLE 2024
dataset. In contrast to prior single-
video egocentric benchmarks, CASTLE requires
reasoning over four days of activity, 15 synchronized perspectives, official transcripts, and multiple auxiliary modalities, including personal photos, auxiliary
videos, gaze, thermal imagery, and heartrate measurements. MARS therefore treats the task as an agentic evidence-selection problem over
multimodal sources rather than a purely text-only pipeline. MARS first follows the official CASTLE directory organization to build evidence
memories from two primary sources,
videos and transcripts, and four auxiliary sources, gaze, heartrate, photos, and thermal imagery. Long
videos are converted into captions and DeepSeek-based summaries only because CASTLE
videos are too long to fit directly into the model context for every question; this step compresses temporal evidence while keeping photos and other auxiliary media available as source-specific evidence. At inference time, a GPT-5.4 decision
agent repeatedly chooses whether to continue
reasoning, request a specific missing modality, produce an answer, or fall back to a random option when the evidence remains insufficient. The resulting system achieved second place on the final CASTLE Challenge leaderboard. Our codes are available at https://github.com/Hyu-Zhang/MARS.
Read the full abstract ▾
All authors Haoyu Zhang, Qiaohui Chu, Yisen Feng, Meng Liu, Weili Guan, Yaowei Wang, Liqiang Nie
Computer Vision
Do You Need Text Rectification? Soft Attention Mask Embedding for Rectification-Free Scene Text Spotting
Antonio Colombo and Giovanni Bianchi
End-to-end scene text spotting, which unifies text detection and recognition within a single framework, has witnessed remarkable progress driven by deep learning advances. However, most existing approaches still suffer from incomplete mask proposals caused by multi-scale variation, arbitrary text shapes, and complex background interference, thereby degrading recognition accuracy. In this paper, we propose a novel Soft
Attention Mask
Embedding module (SAME) that leverages the global receptive field of
Transformer encoders to encode high-level features and compute soft
attention weights, which are then hierarchically embedded with predicted masks to generate refined text-boundary-aware masks that effectively suppress background noise. Building upon this module, we present SAME-Net, a robust end-to-end text spotting framework that requires neither character-level annotations nor auxiliary text rectification modules. Since the soft
attention mechanism is fully differentiable, recognition loss gradients can be back-propagated through the SAME module to the detection branch, enabling joint optimization of detection and recognition objectives. Extensive experiments on challenging
benchmarks demonstrate the effectiveness of our approach: SAME-Net achieves 84.02% end-to-end H-mean on the arbitrarily-shaped Total-Text
dataset, surpassing the previous state-of-the-art GLASS by 1.02% in full-lexicon accuracy without additional training data, and obtains competitive 83.4% strong-lexicon results on the multi-oriented ICDAR 2015
dataset.
Read the full abstract ▾
Computer Vision
Junming Liu, Yuqi Li, Yifei Sun et al. (+6 more)
Vision-
Language Models (VLMs) have made striking progress, yet their spatial
reasoning remains fragile. Models that answer an original input correctly can still fail under valid transformations with predictable answer mappings, revealing a gap between instance-level correctness and robust spatial
reasoning. To address this, we propose Spatial
Alignment via
Geometric Evolution (SAGE), a self-evolving framework that improves robust spatial
reasoning through
geometric and linguistic duality operations. SAGE incorporates duality consistency into GRPO training, encouraging models to produce coherent answers across original and transformed inputs. SAGE co-evolves duality generation and solution, allowing the model to continually expose and address its own
reasoning weaknesses. A dynamic operation pool identifies challenging operations and retires mastered ones, keeping training focused on informative duality signals. SAGE is model-agnostic, data-efficient compared to prior post-training methods, and can be applied as a lightweight adaptation stage to any existing VLM. Experiments on
video and spatial
reasoning benchmarks demonstrate consistent improvements over strong baselines and enhanced generalization to unseen data.
Read the full abstract ▾
All authors Junming Liu, Yuqi Li, Yifei Sun, Maonan Wang, Yiming Cheng, Rui Qian, Piotr Koniusz, Yirong Chen, Ding Wang
Computer Vision
Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
Xinpeng Dong, Min Zhang, Kairong Han et al. (+3 more)
In recent years,
multimodal large
language models (MLLMs) have achieved remarkable progress, primarily attributed to effective paradigms for integrating visual and textual information. The dominant connector-based paradigm projects visual features into textual sequence, enabling unified
multimodal alignment and
reasoning within a generative architecture. However, our experiments reveal two key limitations: (1) Although visual information serves as the core evidential modality in MLLMs, it is treated on par with textual
tokens, diminishing the unique contribution of the visual modality; (2) As generation length increases, particularly within a limited context window, the model's dependence on visual information progressively weakens, resulting in deteriorated
vision-
language alignment and reduced consistency between generated content and visual semantics. To address these challenges, we propose the
Vision Inference Former (VIF), a lightweight architectural module that establishes a direct bridge between pure visual representations and the model's output space. Specifically, VIF continuously injects visual semantics throughout the decoding phase of the inference process, ensuring that the model remains firmly grounded in visual content during generation. We conduct experiments on 14
benchmark tasks covering general
reasoning, OCR, table understanding,
vision-centric evaluation, and
hallucination. Experimental results show that VIF consistently improves model performance across diverse architectures while introducing minimal additional overhead. The code for this work is available at https://github.com/Dong-Xinpeng/VIF.
Read the full abstract ▾
All authors Xinpeng Dong, Min Zhang, Kairong Han, Xu Tan, Fei Wu, Kun Kuang
Computer Vision
Semi-LAR: Semi-supervised Contrastive Learning with Linear Attention for Removal of Nighttime Flares
Xiyu Zhu, Wei Wang, Kui Jiang et al. (+1 more)
Lens
flare removal is challenging due to the large spatial extent of
flare artifacts and their
entanglement with scene structures, while existing methods heavily rely on large-scale paired data. We propose a semi-supervised
flare removal framework that enables stable learning from unlabeled images by jointly addressing pseudo-label reliability and representation discrimination. We propose an adaptive pseudo-label repository that progressively refines pseudo supervision through no-reference quality assessment, momentum-based updates, and invalid label filtering, effectively mitigating error accumulation. Moreover, we propose a
flare-aware contrastive loss that explicitly treats
flare-contaminated inputs as negatives and performs patch-level contrastive learning, encouraging representations that are discriminative against
flare patterns while remaining consistent with reliable pseudo targets. Extensive experiments on multiple
flare benchmarks demonstrate that the proposed framework is model-agnostic and consistently improves performance and robustness.
Read the full abstract ▾
All authors Xiyu Zhu, Wei Wang, Kui Jiang, Zhengguo Li
Computer Vision
Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving
Lijun Zhou, Hongcheng Luo, Zhenxin Zhu et al. (+34 more)
This report presents a unified technical system addressing the two core capabilities of world models for autonomous driving: world representation and world generation. For world representation, we propose WorldRec, a feed-forward reconstruction architecture driven by sparse scene queries. WorldRec initializes structured queries in 3D space, leveraging them to aggregate cross-view, cross-temporal features, thereby naturally enforcing spatial consistency across frames and yielding compact yet high-fidelity 3D Gaussian scene representations. For world generation, we propose WorldGen, a two-stage training framework of bidirectional pretraining followed by causal fine-tuning through three progressive stages (Teacher Forcing, ODE
distillation, and DMD), enabling high-quality online causal
video generation in as few as 4 denoising steps. Building on both modules, we further introduce the JWM, which deeply integrates WorldRec and WorldGen to achieve synergistic gains in generation stability, cross-frame consistency, and visual fidelity, providing a solid foundation for closed-loop simulation, data synthesis, and end-to-end training in autonomous driving.
Read the full abstract ▾
All authors Lijun Zhou, Hongcheng Luo, Zhenxin Zhu, Cheng Chi, Mingfei Tu, Kaixin Xiong, Lei Gong, Zhanqian Wu, Zehan Zhang, Fangzhen Li, Hao Li, Yingying Shen, Jiale He, Haohui Zhu, Shan Zhao, Kai Wang, Zhiwei Zhan, Yuechuan Pu, Kaiyuan Tan, Ruiling Yang, Xianqi Wang, Tianyi Yan, Jiawei Zhou, Lei Zhang, Jingyang Zhao, Xi Zhou, Chitian Sun, Chenming Wu, Jiong Deng, Hongwei Xie, Ming Lu, Kun Ma, Long Chen, Guang Chen, Hangjun Ye, Bing Wang, Haiyang Sun
Computer Vision
Who Generated This 3D Asset? Learning Source Attribution for Generative 3D Models
Sihan Ma, Siyuan Liang and Dacheng Tao
Generative 3D models are deployed in gaming, robotics, and immersive creation, making source attribution critical: given a 3D asset, can we identify whether and which generative model created it? This problem faces two core challenges: dispersed attribution signals, where 3D fingerprints are distributed across multi-view,
geometric, and frequency-domain cues; and realistic deployment constraints, where scarce labels, degraded
prompts, and mixed real/synthetic assets undermine attribution reliability. To systematically study this problem, we construct, to the best of our knowledge, the first passive source attribution
benchmark for modern generated assets, covering 22 representative 3D generators under standard, few-shot, and realistic deployment protocols. Based on this
benchmark, we find that generative 3D models leave two types of stable fingerprints: cross-view inconsistency and structural artifacts reflected in
geometric statistics and frequency-domain cues. To capture these dispersed signals, we propose a hierarchical multi-view multi-modal
Transformer that fuses appearance,
geometric, and frequency-domain features within each view and models global relationships across views. Extensive experiments demonstrate strong performance, achieving 97.22% accuracy under full supervision and 77.17% accuracy with only 1% training data, corresponding to fewer than five samples per generator. These results show that modern 3D generators leave stable and attributable fingerprints, establishing a new
benchmark and methodological foundation for trustworthy 3D content provenance.
Read the full abstract ▾
Computer Vision
Rad-VLSM: A Cross-Modal Framework with Semantics-Assisted Prompting for Medical Segmentation and Diagnosis
Fengyi Zhang, Xujie Zeng, Mohan Liu et al. (+2 more)
Medical image
segmentation is more clinically valuable when it supports diagnosis rather than merely producing lesion masks. However, diagnostically relevant lesion cues are often subtle and localized, while existing models may be distracted by background tissues, acoustic artifacts, and irrelevant visual correlations. To address this problem, we propose Rad-VLSM, a two-stage cross-modal framework for semantics-assisted lesion focusing, robust
segmentation, and visually grounded diagnosis. In the first stage, a BLIP-2-based
vision-
language alignment module identifies lesion-related candidate regions under semantic guidance and converts them into box
prompts. In the second stage, these
prompts are fed into a SAM-based multitask network, where a multi-candidate region aggregation strategy improves
prompt stability and guides lesion
segmentation. The predicted masks are then used as spatial priors for diagnosis, and a visual-radiomics fusion head integrates lesion-aware visual features with selected radiomics descriptors. By using semantic information for localization rather than direct prediction, Rad-VLSM reduces text-to-diagnosis dependence and grounds diagnosis in lesion-level evidence. Experiments on a private clinical breast ultrasound
dataset and public
benchmarks show that Rad-VLSM achieves strong
segmentation and diagnostic performance with favorable generalization.
Read the full abstract ▾
All authors Fengyi Zhang, Xujie Zeng, Mohan Liu, Zengyi Wang, Yalong Jiang
Computer Vision
WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens
Yiwei Guo, Shaobin Zhuang, Zhipeng Huang et al. (+4 more)
Building a unified visual tokenizer is essential for bridging the gap between visual understanding and generation. Yet existing approaches struggle with the inherent conflict between these tasks, as a single
token space is forced to support both high-level semantic abstraction and low-level pixel reconstruction. We propose WinTok, a concise hybrid tokenizer that achieves a win-win performance by explicitly decoupling the two objectives. WinTok supplements pixel
tokens with a set of learnable semantic
tokens, effectively mitigating cross-task interference without incurring the computational overhead of dual tokenizers. To further enhance understanding capability, we introduce an asymmetric
token distillation mechanism: the semantic
tokens are guided by pretrained semantic
embeddings from any visual foundation model, enabling them to inherit strong discriminative power while maintaining flexibility. Across 10 challenging
benchmarks, WinTok delivers consistent improvements in reconstruction, understanding, and generation. Trained on only 50M open-source data, WinTok surpasses the strong baseline UniTok by 11.2% in classification accuracy and achieves a competitive reconstruction rFID of 0.41, despite using substantially less training data. Code is released at https://github.com/markywg/WinTok.
Read the full abstract ▾
All authors Yiwei Guo, Shaobin Zhuang, Zhipeng Huang, Canmiao Fu, Chen Li, Jing Lyu, Yali Wang
Computer Vision · Cross-listed from cs.CL
How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking
Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain et al. (+3 more)
Recent advancements in Large
Language Models (LLMs) and Large
Vision Language Models (LVLMs) have enabled general-purpose systems to demonstrate promising capabilities in complex
reasoning tasks, including those in the medical domain. Medical Visual Question Answering (MedVQA) has particularly benefited from these developments. However, despite Bangla being one of the most widely spoken
languages globally, there exists no established MedVQA
benchmark for it. To address this gap, we introduce BanglaMedVQA, a
dataset comprising clinically validated image-question-answer pairs, along with a comprehensive evaluation of current foundation models on this resource. Consistent with prior findings that report low performance of current models on English MedVQA
benchmarks, our analysis reveals that Bangla performance is substantially lower, reflecting the challenges inherent to low-resource
languages. Even top-performing models such as Gemini and GPT-4.1 mini fail to accurately answer specialized diagnostic questions, indicating severe limitations in fine-grained medical
reasoning. Although certain open-source models, such as Gemma-3, occasionally outperform these models in general categories, they too struggle with clinically complex questions, underscoring the urgent need for top-notch evaluation method.
Read the full abstract ▾
All authors Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain, Tasnimul Hossain Tomal, Md Fahim, Md Farhad Alam Bhuiyan
Computer Vision · Cross-listed from cs.AI
TaskGround: Structured Executable Task Inference for Full-Scene Household Reasoning
ZhiYuan Feng, Yu Deng, Ruichuan An et al. (+11 more)
In real home deployments, household
agents must often operate from a complete household scene and a situated household request, rather than from a clean task specification. Such requests require
agents to identify task-relevant entities, recover intended task conditions, and resolve ordering constraints from the surrounding scene context. We formalize this capability as full-scene household
reasoning: given a complete household scene and a situated household request, an
agent must infer executable task structure before producing a grounded skill-level action sequence. This setting is challenging because complete household scenes contain substantial task-irrelevant information, making direct complete-scene prompting inefficient and error-prone. In practical deployment, this challenge is further amplified by
privacy and local compute constraints, which favor compact open-weight models with limited long-context
reasoning ability. We propose TaskGround, a training-free and model-agnostic Ground-Infer-Execute framework that grounds complete scenes into compact task-relevant scene slices, infers executable task structure, and compiles it into grounded skill-level action sequences. To evaluate this setting, we introduce FullHome, a human-validated evaluation suite of 400 household tasks spanning diverse home-scale environments and both goal-oriented and process-constrained requirements. On FullHome, TaskGround improves task success rates by large margins across both proprietary and open-weight models. Notably, it makes Qwen3.5-9B competitive with GPT-5 under direct complete-scene prompting while reducing total input-
token cost by up to 18x. Our results identify executable task-structure inference as a central bottleneck in full-scene household
reasoning and show that structured grounding can make compact local models substantially more effective for practical household deployment.
Read the full abstract ▾
All authors ZhiYuan Feng, Yu Deng, Ruichuan An, Zhenhua Liu, Qixiu Li, Keming Wu, Zhiying Du, Weijie Wang, Haoxiao Wang, Shuang Chen, Sicheng Xu, Yaobo Liang, Jiaolong Yang, Baining Guo
Computer Vision
DanceHMR: Hand-Aware Whole-Body Human Mesh Recovery from Monocular Videos
Wenhao Shen, Ming Zhou, Hengyuan Zhang et al. (+3 more)
Monocular video human mesh recovery is essential for digital humans, avatar animation, and
embodied simulation, where both temporal stability and expressive whole-body motion are required. Existing
video HMR methods produce coherent body motion but often overlook detailed hand articulation, while image-based whole-body methods recover SMPL-X meshes independently per frame, often leading to jittery and inaccurate hand motion. We present a temporally coherent whole-body HMR framework for challenging in-the-wild
monocular videos. Our model unifies body context and part-specific hand observations through residual body-hand fusion, enabling stable body motion and detailed hand recovery within a single temporal architecture. We further introduce close-up-aware augmentation to improve robustness under upper-body framing. Experiments on whole-body and body-only
benchmarks demonstrate improved hand reconstruction and competitive body accuracy. Our method also produces temporally stable and 2D-consistent SMPL-X motion in challenging real-world
videos.
Read the full abstract ▾
All authors Wenhao Shen, Ming Zhou, Hengyuan Zhang, Siyuan Bian, Youjiang Xu, Yuan Zhang
Computer Vision
SENSE: Satellite-based ENergy Synthesis for Sustainable Environment
Kailai Sun, Mingyi He, Heye Huang et al. (+5 more)
Urban Building Energy Modeling plays a critical role in achieving the United Nations' Sustainable Development Goals 7 and 11. Although existing studies based on satellite imagery and deep learning have achieved remarkable progress, many challenges exist: most existing studies are inherently predictive, failing to reflect the generative nature of urban planning; although generative AI and
diffusion models have seen explosive growth in satellite imagery, they lack the urban functional generation (e.g., energy layer); third, aligned high-quality high-resolution building energy data with satellite imagery is limited and scarce. Here we propose SENSE (Satellite-based ENergy Synthesis for Sustainable Environment), a unified generative UBEM framework that jointly synthesizes realistic urban satellite imagery and aligned high-quality building energy consumption and height maps. By conditioning on road networks and urban density metrics, SENSE, based on a controllable
diffusion model, leverages the knowledge learned by large
vision models to generate urban building energy consumption and height information (annotations) in the latent space. Experiments across four cities (New York City, Boston, Lyon, Busan) demonstrate that SENSE achieves high visual fidelity and strong physical consistency, satisfying the ASHRAE standard metric. Experiments demonstrate that SENSE can generate enough annotated synthetic data using less than 20% labeled energy data, boosting downstream prediction performance by 10% IoU. Compared to SOTA urban energy prediction methods, SENSE significantly reduced prediction error (reduced 3%-11% NMBE and 1%-9% CVRMSE). This study offers an energy-efficiency urban planning and physical generation solution for urban science, energy science and building science. The
dataset and code: https://huggingface.co/
datasets/skl24/MUSE and https://github.com/kailaisun/GenAI4Urban-Energy/.
Read the full abstract ▾
All authors Kailai Sun, Mingyi He, Heye Huang, Can Rong, Alok Prakash, Baoshen Guo, Shenhao Wang, Jinhua Zhao
Computer Vision
The MixCount Dataset: Bridging the Data Gap for Open-Vocabulary Object Counting
Corentin Dumery, Niki Amini-Naieni, Shervin Naini et al. (+1 more)
Object counting is a foundational
vision task with over a decade of dedicated research, yet state-of-the-art models still fail systematically in the mixed-object setting that dominates real-world applications such as industrial inspection and product sorting. We show that this gap is strongly driven by limitations in existing training and evaluation data: real counting
datasets are prohibitively expensive to annotate and suffer from labeling noise, while existing synthetic alternatives lack diversity and realism. We address this with MixCount, a
dataset and
benchmark for mixed-object counting designed to target the failure modes of current counting models. To overcome the high cost of constructing and labeling such data, we develop an automatic generation pipeline that synthesizes images, fine-grained textual descriptions, and pixel-perfect counting annotations at scale, eliminating the labeling ambiguity that plagues prior
datasets. Evaluating state-of-the-art counting models on MixCount exposes severe degradation in the mixed-object setting. More importantly, training these models on our synthesized data yields substantial gains on real-world
benchmarks, reducing MAE by 20.14% on FSC-147 and by 18.3% on PairTally. These results establish MixCount as both a
benchmark and a training
dataset for fine-grained counting, and demonstrate that our pipeline, which produces effectively unlimited labeled data, helps address a long-standing bottleneck in counting models.
Read the full abstract ▾
All authors Corentin Dumery, Niki Amini-Naieni, Shervin Naini, Pascal Fua
Computer Vision
Embedded ConvNet Ensembles: A Lightweight Approach to Recognize Arabic Handwritten Characters
Mohsine El Khayati, Rachid Elouahbi and Abdelillah Semma
Arabic Handwritten Character Recognition (AHCR) has recently advanced significantly with deep Convolutional Neural Networks (ConvNets). However, many models in the literature are deep and computationally expensive in terms of parameters and FLOPs, limiting their deployment on resource-constrained devices, which are increasingly common. This study addresses this gap by proposing a combination of lightweight embedded ConvNet models and ensemble learning techniques. Extensive experiments were conducted to identify best practices in AHCR, considering training hyperparameters, learning strategies, model choices, and ensemble methods. Results show that embedded models can achieve accuracy comparable to, or even surpassing, heavier architectures. Ensemble learning further enhances performance with only modest computational overhead, particularly under challenging training scenarios. Among the ensembling strategies, soft voting yielded the best overall results.
Read the full abstract ▾
Computer Vision
Threats to Arabic Handwriting Recognition: Investigating Black-Box Adversarial Attacks on embedded ConvNet models
Mohsine EL Khayati, Abdelillah Semma, Abdelaziz Courr et al. (+1 more)
Arabic handwriting recognition (AHR) has made significant progress with deep learning models. AHR research has largely focused on performance, with security receiving little
attention. This study provides what appears to be a new line of inquiry by demonstrating the vulnerability of high-performing models to
adversarial black-box attacks. The focus on black-box attacks reflects real-world scenarios where the attacker has no prior knowledge of the model architecture. Extensive experiments were conducted on two
benchmark AHR
datasets containing Arabic handwritten Characters. Results demonstrated the effectiveness of the attacks, with the Pixle attack achieving an attack success rate of 99-100% on most models. Other, less aggressive attacks achieved success rates of 50-96% across most experiments. Despite the higher attack success rate, the attacks maintain the structural integrity of the characters, rendering them almost imperceptible to the human eye. The findings indicate the higher vulnerability of the studied models to
adversarial manipulation. This underscores the need to strengthen efforts to secure these models and ensure their reliability in AHR real-world applications.
Read the full abstract ▾
All authors Mohsine EL Khayati, Abdelillah Semma, Abdelaziz Courr, Rachid Elouahbi
Computer Vision · Cross-listed from eess.IV
CATRF: Codec-Adaptive TriPlane Radiance Fields for Volumetric Content Delivery
Tung-I Chen, Lingdong Wang, Subhransu Maji et al. (+1 more)
Volumetric media promises next-generation content delivery applications, but its bandwidth demand remains a key bottleneck. Implicit and hybrid volumetric representations reduce model sizes, yet still require careful coding to reach 2D
video-like bitrates. We present CATRF, a standard-codec-in-the-loop compression framework for plane-factorized radiance fields. During training, we quantize and pack 2D feature planes into codec-friendly canvases, run a standard codec roundtrip (JPEG/VP9/HEVC/AV1), then unpack and dequantize the decoded features before volume rendering. We use a straight-through
estimator (STE) to insert the non-differentiable, standard codec pipeline into the training loop, allowing radiance-field features to adapt directly to the real, client-side codec distortions without introducing any learned codec parameters. On both static and dynamic
benchmarks, CATRF consistently achieves a better rate-distortion trade-off over codec-agnostic and learned-codec-in-the-loop baselines, and also outperforms recent compressed 3DGS methods in both compression efficiency and decoding speed. These results highlight a practical path toward low-bitrate, compression-resilient volumetric representations for free-viewpoint
video streaming.
Read the full abstract ▾
All authors Tung-I Chen, Lingdong Wang, Subhransu Maji, Ramesh K. Sitaraman
Computer Vision
Efficient 3D Content Reconstruction and Generation
Jiahao Li
Automatic 3D content creation seeks to replace labor-intensive modeling and scanning pipelines with systems that can synthesize or recover 3D assets directly from text or images. Its applications span
video games, virtual reality, robotics, and simulation, enabling rapid asset prototyping, diverse interactive world generation, and efficient 3D data collection for training foundation models. Contemporary solutions largely follow two complementary paradigms: (i) text- or image-to-3D generation, which learns priors over 3D geometry and appearance to create novel assets from natural
language or a single view image; and (ii) 3D reconstruction, which estimates camera poses and geometry from RGB images. This thesis advances both directions. On the generation side, I introduce Instant3D, which combines multi-view
diffusion with feed-forward sparse-view 3D reconstruction to produce high-quality assets in 5-20 seconds. On the reconstruction side, I develop FastMap, a structure-from-motion pipeline that achieves up to 10x speedup over prior state-of-the-art by using first-order optimization with fused GPU kernels extensively, while maintaining comparable pose accuracy and downstream novel view synthesis quality.
Read the full abstract ▾
Computer Vision
OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models
Morunliu Yang, Ruotao Xu, Le Li et al. (+6 more)
Omnimodal large
language models (OmniLLMs) have recently gained increasing
attention for unified audio-
video understanding. However, processing long
multimodal token sequences introduces substantial computational overhead, making efficient
token compression crucial. Existing methods typically rely on fixed, modality-specific guidance, which fails to account for the varying importance of modalities across different queries. To address this limitation, we propose $\textbf{OmniSelect}$, a training-free, modality-adaptive
token pruning framework that dynamically selects appropriate compression strategies for
multimodal inputs. Specifically, we leverage a lightweight AudioCLIP model to estimate cross-modal relevance and categorize each input into three
pruning regimes: Audio-Centric,
Video-Centric, and Uniform
pruning. Based on these relevance scores, OmniSelect further performs fine-grained
token pruning within each temporal group, adaptively allocating
pruning ratios to preserve informative
tokens across modalities. By explicitly modeling modality preference and enabling dynamic strategy selection, OmniSelect effectively avoids the pitfalls of one-size-fits-all compression. Extensive experiments demonstrate that our method achieves efficient
multimodal token reduction while maintaining strong performance, without requiring any additional training.
Read the full abstract ▾
All authors Morunliu Yang, Ruotao Xu, Le Li, Yue Wang, Jianxin Zhang, Juntao Li, Yihang Lou, Siwei Feng, Peifeng Li
Computer Vision
SGSoft: Learning Fused Semantic-Geometric Features for 3D Shape Correspondence via Template-Guided Soft Signals
Soyeon Yoon, Chang Wook Seo and Hyunjung Shim
Learning dense correspondences across deformable 3D shapes remains a long-standing challenge due to structural variability, non-isometric deformation, and inconsistent
topology. Existing methods typically trade off generalization,
geometric fidelity, and efficiency. We address this by proposing SGSoft, a unified intrinsic pipeline that (i) constructs a geodesic correspondence field on a canonical template, (ii) learns
multimodal dense descriptors guided by pretrained semantic priors with this geodesic correspondence field supervision, (iii) retrieves dense correspondences in a single feed-forward pass via nearest-neighbor search in descriptor space. This formulation enables stable and
topology-invariant supervision under large pose variation, structural differences, and remeshing. SGSoft achieves state-of-the-art inter-category generalization while offering the best accuracy-efficiency trade-off among prior methods. It also achieves near real-time inference without pre-
alignment, pairwise optimization, or post-refinement. Learned descriptors can be transferred effectively to downstream tasks such as semantic
segmentation and deformation transfer, establishing a scalable and deployment-ready paradigm for dense 3D correspondence.
Read the full abstract ▾
Computer Vision
Patch Ensembles for Robust Salmon Re-Identification with Weak Trajectory Labels
Espen Uri Høgstedt, Christian Schellewald, Annette Stahl et al. (+1 more)
Salmon re-identification in commercial net-pens is challenging due to large populations, which impose strict accuracy requirements and make large-scale labeled data acquisition infeasible. Trajectory IDs can be used as proxy labels, but this introduces trajectory-ID bias. To address these challenges, we propose a patch-based re-identification framework that fuses patch-level predictions into a salmon identity decision. A key component is the prediction of the salmon's lateral line, enabling extraction of texture-anchored patches and patch slices. To enable realistic evaluation, we introduce an experimental setup using multiple cameras placed 6 m apart, allowing the same fish to be recorded in different trajectories. This enables the construction of a cross-camera test set through manual match confirmation. Our ensemble approach outperforms the full-image baseline in same-trajectory validation (0.932 to 0.965 mAP) and cross-camera testing (0.609 to 0.860 mAP). The substantial improvements in the cross-camera setting demonstrate improved generalizability and robustness. Code and data: https://github.com/espenbh/salmon-reid-patch-ensemble.
Read the full abstract ▾
All authors Espen Uri Høgstedt, Christian Schellewald, Annette Stahl, Rudolf Mester
Computer Vision
What Matters for Grocery Product Retrieval with Open Source Vision Language Models
Emmanuel G. Maminta and Rowel O. Atienza
Multimodal product
retrieval (MPR) underpins checkout-free retail and automated inventory systems, yet it demands fine-grained SKU discrimination that standard
vision-
language benchmarks fail to capture. We present the first systematic zero-shot evaluation of 190 open-source VLMs on the MPR task of the GroceryVision Challenge, isolating pre-training data, architecture, and input resolution. Our analysis yields three actionable findings. (1) Data quality trumps scale. Switching from raw web-scrapes to filtered
datasets delivers up to 16.6% accuracy gains, exceeding the benefit of doubling model parameters. (2) Efficient models can win. MobileCLIP-B (150M parameters) outperforms 351M counterparts trained on noisy data. We introduce semantic power density ($\phi$), an efficiency metric that penalizes sub-threshold accuracy. (3) A precision gap persists. State-of-the-art models achieve 94.5% Recall@5 but suffer a 17.5% drop at Recall@1, revealing that contrastive
embeddings cluster categories effectively but fail to rank visually similar SKUs. Code and evaluation scripts are available at \url{https://github.com/upeee/openmpr}.
Read the full abstract ▾