Computer Vision
Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention
Daniel Shalam, Emanuel Ben Baruch, Avi Ben Cohen et al. (+1 more)
Multimodal large
language models can emit localized predictions, bounding boxes for objects and temporal windows for
video and audio events, but they hallucinate these regions prolifically. The model's own
token log-probabilities are nearly uninformative: they conflate grounding quality with input ambiguity, and coordinate
tokens become near-deterministic once the model commits. We propose Multi-
Token Localized
Attention (MTLA): a training-free, post-hoc score that measures how strongly a prediction's
tokens attend to the region they claim. Prior
attention-based detectors, which sum
attention over the entire input modality and read a single response
token, are weaker special cases; we show that summing only within the claimed region and aggregating across all prediction
tokens recovers a stronger grounding signal. The same recipe applies almost trivially to other modalities and tasks: object detection in images and temporal localization in
video and audio. Across multiple MLLM families and three modalities, MTLA improves
hallucination AUROC by +7 to +38 over the best prior training-free baseline. Used as a confidence score for re-ranking, it nearly doubles the zero-shot COCO detection AP of an open-source 8B generalist (from 20.4 to 37.0), narrowing the gap to supervised detectors without any task-specific training.
Read the full abstract ▾
All authors Daniel Shalam, Emanuel Ben Baruch, Avi Ben Cohen, Tal Remez
Computer Vision
Decoupled Single-Mask Annotation Noise Detection via Cross-Sectional Patch Self-Consistency
Yinheng Zhu and Xiaowei Xu
Vascular computed
tomography datasets are commonly annotated only once per scan, yielding the pervasive yet under addressed problem of single mask annotation noise. Existing solutions either require costly multirater fusion or are coupled with network training, preventing explicit
auditing of where and why labels fail. We introduce a decoupled framework for single-mask annotation noise detection that leverages cross-sectional patch self-consistency to produce interpretable and auditable noise evidence. Tubular anatomy exhibits strong cross-sectional recurrence: patches extracted orthogonally along vessel centrelines recur in appearance across locations and subjects. Thus, anatomically similar patches should have consistent masks, and disagreement signals unreliable annotation. Our method samples cross-sectional patches, retrieves intensity-equivalent neighbours via scalable vector search, and computes a patch-level noise score from statistical mask disagreement, yielding explicit image-mask evidence for every flagged region. Aggregating scores produces scan-level quality maps for
dataset quality assessment or quality-weighted training. Experiments on the coronary CT
dataset validate the detected noise for improving training robustness and reveal systematic annotation biases. Specifically, transverse and oblique vessels exhibit 5.1 times higher error rates than axis-aligned structures, with additional correlations to cross-sectional area and intensity. Code is available here.
Read the full abstract ▾
Computer Vision
NegROI: Click-Centric Uncertainty-Guided Refinement with Scene-Conditioned Negative Prompts for Robust Interactive 3D Segmentation
Shuheng Zhang and Feng Wu
Interactive 3D
segmentation aims to extract object masks in point clouds with minimal user clicks. Despite recent progress, most existing approaches still struggle with (i) coarse voxel resolution that blurs fine boundaries under limited clicks and (ii) hard false positives caused by confusing background structures. These issues are exacerbated by density and scale shifts across
datasets (e.g., dense RGB-D reconstructions vs. sparse
LiDAR scans), where fixed refinement heuristics and purely click-driven decoding generalize poorly. To address them, we propose NegROI – a novel
transformer-based interactive framework that couples click-centric multi-resolution refinement with scene-conditioned negative
prompts. Given a coarse voxel prediction, it refines only a local Region Of Interest (ROI) around the current click on a finer grid and fuses refined logits back to the coarse mask. To improve robustness and efficiency, we introduce uncertainty-driven selective refinement that prioritizes ambiguous regions. Meanwhile, we model hard background patterns via a set of scene-conditioned negative
prompts obtained by cross-
attention over scene
tokens. We further stabilize these
prompts with a diversity regularizer. Finally, we propose boundary-aware hard negative mining to supervise negative-
prompt attention toward boundary-proximal, high-confidence false positives. Our experiments on common
benchmark datasets (i.e., ScanNet, S3DIS, and KITTI) demonstrate improved click efficiency and reduced false positives, with stronger cross-
dataset robustness than the state-of-the-art baselines.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.IR
Ryota Tanaka, Taku Hasegawa and Kyosuke Nishida
Multimodal document
retrieval aims to retrieve relevant pages while preserving both textual and visual content from the original document. However, existing
benchmarks primarily evaluate simple lexical or semantic matching, and most methods encode pages independently. Consequently, they overlook the contextual information in the document required to resolve queries that aggregate information across multiple pages. In this paper, we introduce CMDR and CMDR-Bench, a new
multimodal document
retrieval task and
benchmark that require modeling document context. To address this challenge, we propose CMDR-Embed, a contextual
multimodal embedding framework that explicitly incorporates document context by jointly encoding multiple pages and deriving page-level
embeddings from a shared contextual representation. Furthermore, we introduce CMCL, a contextual
multimodal contrastive learning objective that effectively trains CMDR-Embed by balancing contextual modeling with page-level discriminability. Experiments demonstrate that CMDR-Embed significantly outperforms non-contextual
embeddings, highlighting the importance of context-aware
multimodal embeddings for advancing document
retrieval.
Read the full abstract ▾
Computer Vision
Progressive Reasoning with Primitive Correction for Compositional Zero-Shot Learning
Ziyi Chen, Haoyan Shi, Sunhan Xu et al. (+1 more)
Compositional Zero-Shot Learning (CZSL) aims to combine known attributes and objects as primitives for recognizing previously unseen attribute-object pairs. Prior works either predict attributes and objects independently, missing their strong contextual dependency, or use unidirectional conditional modeling (e.g., object-guided attribute prediction), which is prone to error propagation. We propose PRPC, a Progressive
Reasoning framework with Primitive Correction, which explicitly models the bidirectional dependency between attributes and objects via step-wise inference. PRPC performs mutual correction of primitives to suppress prediction errors in earlier steps. Specifically, we formulate CZSL as structured, Q&A-style Chain-of-Thought
reasoning process and constrain the MLLM to follow predefined semantic steps to generate intermediate decisions. To further enhance the reliability and logical consistency of intermediate
reasoning, we introduce reinforcement learning post-training with a GRPO-based objective, providing step-level rewards aligned with the progressive inference procedure. Extensive experiments on three CZSL
benchmarks demonstrate that PRPC achieves state-of-the-art performance, validating the effectiveness of progressive
reasoning and bidirectional correction for robust compositional generalization.
Read the full abstract ▾
All authors Ziyi Chen, Haoyan Shi, Sunhan Xu, Congyan Lang
Computer Vision
PolicyShiftGuard: Benchmarking and Improving Policy-Adaptive Image Guardrails
Mingyang Song, Luxin Xu, Haoyu Sun et al. (+3 more)
Image guardrails are typically trained and evaluated under a fixed safety policy, implicitly treating safety as an intrinsic property of an image. Real deployments are different: the same image may be allowed in one product, restricted in another, and newly disallowed when a policy boundary changes. We study policy-adaptive image guardrailing, where a model must decide whether an image violates the currently supplied policy and generalize to held-out policy definitions. We introduce PolicyShiftBench, a comprehensive
benchmark with 2,000 policy-discriminative instances over 265 images, where each image is paired with 7.55 policy-conditioned
prompts on average to test whether models adapt to the active policy rather than relying on image-level safety priors. We then propose PolicyShiftGuard, a compact policy-conditioned guardrail trained with a two-stage training recipe that combines Randomized Policy SFT (RP-SFT) with Boundary-Pair Policy Adaptation (BP-Adapt). BP-Adapt trains matched
prompts for the same image and risk category using standard label supervision and a pairwise comparison loss that separates blocking policies from passing policies. Experiments show that existing VLMs and specialized guardrails remain brittle under policy shifts, while PolicyShiftGuard substantially improves policy-sensitive performance. The 7B model achieves SOTA performance of 76.9 Avg. F1 and 72.1 Avg. PSS on PolicyShiftBench, transfers well to UnSafeBench and SafeEditBench, and improves the latency-performance trade-off with a concise output format. Ablations confirm that matched pass/block boundary pairs are essential for stable policy adaptation.
Read the full abstract ▾
All authors Mingyang Song, Luxin Xu, Haoyu Sun, Minzhou Pan, Yu Cheng, Bo Li
Computer Vision
GaussFusion: Towards Multimodal 3D Gaussian Pretraining
Zhixuan You, Jihua Zhu, Yiding Sun et al. (+5 more)
3D Gaussian Splatting provides an explicit representation that jointly models geometry and appearance, serving as a scalable foundation for 3D representation learning. Existing pre-training methods for Gaussian representations, such as masked Gaussian reconstruction, primarily capture local structures but offer limited semantic supervision. In this paper, we propose GaussFusion, a
multimodal pre-training framework for 3D Gaussian representations. GaussFusion integrates image and text supervision into masked Gaussian modeling through cross-modal semantic
alignment, enabling the Gaussian encoder to learn both visual and
language-level semantic information during pre-training. To better adapt masked modeling to the non-uniform distribution of Gaussian primitives, we further propose Gaussian Salience-guided Multi-scale Hole Masking (GSHM). GSHM constructs spatially continuous masked regions based on Gaussian salience. By applying hole masks at multiple scales, GSHM encourages the encoder to capture both fine-grained local patterns and broader structural dependencies. Extensive experiments on downstream tasks demonstrate that GaussFusion improves the transferability of Gaussian representations. Notably, GaussFusion outperforms Gaussian-MAE on ModelNet40 and ScanObjectNN (PB-T50-RS) by 0.61% and 3.85%, respectively.
Read the full abstract ▾
All authors Zhixuan You, Jihua Zhu, Yiding Sun, Zihao Guo, Haozhe Cheng, Dongxu Zhang, Lin Chen, Hainan Luo
Computer Vision
Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation
Cemil-Andrei Dilmac, Florinel-Alin Croitoru and Radu Tudor Ionescu
Coreset selection aims to identify a small and highly representative subset of a massive
dataset for efficient model training. The problem remains challenging even in the few-shot knowledge
distillation (KD) setup, where a full-scale pre-trained teacher informs the student network. Typical sample selection strategies often struggle to surpass the random selection baseline. In this paper, we showcase few-medoids, an embarrassingly simple coreset selection strategy that chooses the samples closest to the centroid (average image) of each class. We present extensive KD experiments on four
datasets, covering a wide range of image classification problems, and three teacher-student model pairs, comprising both convolutional and
transformer networks. Although the proposed method is embarrassingly simple, our empirical results indicate that few-medoids is able to consistently surpass the random selection baseline, as well as the other coreset selection strategies. We therefore consider that few-medoids can be used as a drop-in replacement for commonly-used baselines (e.g. herding or k-center Greedy), in future research on coreset selection. To reproduce the reported results, we publicly release our code at https://github.com/CemilAndreiDilmac/Few-Shot-KD-Coreset.
Read the full abstract ▾
Computer Vision
Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical context
Suneeta Mall, Vladimir Nekrasov, Ashnil Kumar et al. (+5 more)
Imaging demand is growing faster than the radiology workforce can expand, and reporting backlogs cannot be resolved through training and recruitment alone. The most direct opportunity is reducing the time and effort radiologists spend producing reports, a task that requires interpreting images, integrating clinical history and prior studies, and drafting structured findings. We present Harrison.Rad 1.5 (HR1.5), a radiology-specific
multimodal large
language model that accepts interleaved text and visual inputs and generates structured and unstructured text across plain-film radiology, spanning computed radiography, chest, musculoskeletal, abdominal, spine, and pelvic x-rays, and mammography. HR1.5 is trained through a three-stage pipeline: domain adaptation of a base
language model on radiology reports, contrastive
vision-encoder training with curriculum-based hard negatives on ~6 million image-report instances, and visual-question-answering fine-tuning on multi-turn conversations. We evaluate it with a Findings-Diagnosis scoring framework that extends RadGraph-XL entity extraction with ontology-based synonym matching and polarity-contradiction detection, benchmarked on RadBench, a simulated FRCR 2B Short Case examination scored against Angoff-method thresholds, ReXGradient, and internal multi-modality
datasets. HR1.5 is the only system evaluated to meet the simulated FRCR passing standard and achieves the highest accuracy on closed-format clinical questions, across anatomical regions, on internal multi-body-part and mammography reporting, and on the primary clinically-aligned score for public chest reporting. We further examine explainability and model behaviour, including question-sensitive Grad-CAM heatmaps,
attention analysis, and confidence estimation, to support responsible future evaluation toward clinical use, and a framework for clinically grounded assessment of report quality.
Read the full abstract ▾
All authors Suneeta Mall, Vladimir Nekrasov, Ashnil Kumar, Sajith Karunasena, Aiden Nibali, Alix Bird, Mateo Diaz Shine, Jarrel Seah
Computer Vision · Cross-listed from cs.RO
GraspIT: A Dataset Bridging the Sim-to-Real gap and back for Validated Grasping SE(3) Pose Generation
Paul Koch. Adem Karakurt and André Sers
Robust robotic grasping of novel objects requires
datasets that simultaneously provide photorealistic RGB-D observations, physically validated grasp quality annotations, and a principled bridge between simulation and the real world, which existing
datasets lack to provide jointly. GraspIT addresses this gap: tabletop scenes in NVIDIA Isaac Sim are annotated via a four-stage physical slip-test on parallel Franka Panda instances, producing trajectory-reachability checks and continuous quality scores beyond force-closure.Of ${\sim}$2.3M candidates, 83% pass as good ($s{\geq}0.50$); the 17% that passed force-closure but failed the slip-test provide graded hard negatives. A Real$\leftrightarrow$Sim loop back-projects these labels onto 100 real-world scenes. The release provides ${\sim}$316k annotated RGBD frame sets across 1035 sim and 100 real scenes, with instance masks, 6-DoF poses, physical object properties, and scored 6-DoF grasps. All tools are open-source and Docker-containerized. The trajectory planning within Isaac Sim further allows streaming of high resolution demonstrations for tabletop manipulation policy learning and behavior cloning.
Read the full abstract ▾
Computer Vision
AVA-VLM: Adaptive Visual Attention-Vision Language Model for In-the-Wild Construction Site Monitoring
Younggun Kim, Taeheon Kim, Youngseo Kim et al. (+1 more)
Existing construction-site
Vision-
Language Model (VLM) studies have primarily adapted pretrained VLMs through direct QA-style fine-tuning from a single global image, but we argue that this paradigm remains limited in operational range, reliability under reduced-resolution inputs, and inference efficiency. To address these limitations, we propose AVA-VLM, an Adaptive Visual
Attention-
Vision Language Model that follows a human-inspired coarse-to-fine strategy: it first reasons over a low-resolution global image and requests a high-resolution local crop only when detailed inspection is needed. We further introduce a region-aware Chain-of-Thought
dataset that teaches when to inspect, where to crop, and how to use local evidence. Experiments show that, for violation identification, AVA-VLM improves overall F1 from 62.0 to 75.1 while using only 30.6% of the baseline visual-
token budget; for long-distance PPE-violation cases, F1 improves from 16.0 to 63.6. These results demonstrate AVA-VLM's improved robustness to distant and reduced-resolution visual evidence with substantially lower visual-
token usage.
Read the full abstract ▾
All authors Younggun Kim, Taeheon Kim, Youngseo Kim, Seunghee Park
Computer Vision
Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning
Suraj Yadav, Anjaneya Sharma and Siddharth Yadav
Deep neural networks trained with Empirical Risk Minimization (ERM) often fail under distribution shifts because they exploit spurious correlations between object labels and background context. Recent generative approaches address this issue by creating counterfactual images with altered contexts, but typically use these samples as standard data augmentation, leaving the model free to retain background-sensitive representations. We propose a two-stage framework that uses generative intervention to explicitly learn background-invariant visual representations. First, we isolate the foreground object using zero-shot
segmentation and generate context-shifted variants with a structure-preserving
diffusion model, preserving object identity while varying the surrounding environment. We then introduce Cross-Variant Self-Supervised Learning, where variants of the same object under different backgrounds form positive pairs in a contrastive objective. This encourages the encoder to align object-centric representations while suppressing background-specific cues. Then, we fine-tune the pretrained encoder using an ERM warm-up followed by GroupDRO with layer-wise learning rates. Experiments on distribution-shift
benchmarks demonstrate best worst-group performance, achieving 92.5% on Waterbirds, 81.7% on MetaShift, and 87.4% on NICO++. Code: https://github.com/surajyadav-research/GRSSL
Read the full abstract ▾
Computer Vision
Realistic Compound-Lens Defocus Blur Synthesis
Yunkyu Lee, Woohyeok Kim and Sunghyun Cho
Defocus blur degrades fine image structures and limits visual perception, which can adversely affect downstream
vision tasks. Although recent deep learning deblurring methods have achieved strong performance, their effectiveness depends on training data and often degrades across cameras and lenses due to limited optical diversity and realism in existing
datasets. In this paper, we propose a pipeline for synthesizing realistic defocus deblurring
datasets for diverse compound lenses. It integrates efficient wave-optics PSF computation via Debye CZT propagation, depth-aware defocus rendering with occlusion handling, and blur synthesis in the radiometrically linear space with camera ISP simulation. This unified pipeline enables the scalable generation of photorealistic defocus
datasets with diverse lens characteristics. Using our pipeline, we generate CLDefocus, a large-scale synthetic
dataset containing lens-diverse defocus image pairs. We further analyze the limitations of real-captured defocus
datasets and show that such imperfections can bias full-reference evaluation. Extensive experiments demonstrate that models trained on CLDefocus achieve improved cross-device generalization compared to models trained on existing real and synthetic
datasets.
Read the full abstract ▾
Computer Vision
Complementary Roles of Image Classification and Vessel Segmentation in AI-Based Screening for Retinopathy of Prematurity Plus Disease in a Kenyan Preterm Cohort
Fred Mutisya, Oscar Onyango, Sarah Sitati et al. (+8 more)
Background. Retinopathy of prematurity (ROP) is a preventable cause of childhood blindness, with rising burden in low- and middle-income countries where ROP-trained ophthalmologists are scarce. Plus disease, marked by retinal vessel dilation and tortuosity, triggers treatment but is subjective and variable. Automated screening could extend specialist reach, but African evidence remains limited. Methods. We analysed 121 Kenyan preterm infants, covering 237 eyes and 1,635 fundus images graded as No Plus, Pre-Plus or Plus. Vessel annotations from two graders supported
segmentation training. Eleven configurations were evaluated for eye-level Plus detection using patient-grouped nested cross-validation, including image classifiers, multiple-instance learning, multi-task
segmentation-classification, and segment-then-classify pipelines. Results. Vessel
segmentation was feasible, achieving pooled Dice 0.533, IoU 0.368, sensitivity 0.623 and specificity 0.979 on held-out images. RGB classifiers were highly sensitive but over-referred, while
segmentation-coupled models were more specific. Combining approaches improved performance: an OR-based screen achieved the highest sensitivity, an AND-based confirmation achieved the highest specificity, and a probability ensemble gave the best balanced performance, with sensitivity 0.692, specificity 0.914 and balanced accuracy 0.803, outperforming the
vision classifier alone. Conclusions. Classification and vessel
segmentation are complementary for ROP Plus detection in Kenyan data. Classifiers support sensitive case-finding, while
segmentation improves specificity and reduces over-referral. African ROP AI systems should use combined workflows and undergo prospective multi-site validation.
Read the full abstract ▾
All authors Fred Mutisya, Oscar Onyango, Sarah Sitati, Syokau Ilovi, Aeesha NJ Malik, Brenda W'mosi, Brian Makini, Jalemba Aluuvala, Josiah Onyango, Rachael Kanguha Mmene, Steven Wanyee
Computer Vision
TRIG: Trajectory-Rig Decoupled Metric Geometry Learning
Lizhou Liao, Wentao Xu, Handong Wang et al. (+4 more)
Vision-centric autonomous driving requires accurate metric geometry and ego-motion estimation from synchronized multi-camera observations. Recent visual geometry models show strong performance in pose estimation, depth prediction, and 3D reconstruction, but are not tailored to rigid multi-camera driving systems. They often encode camera poses as entangled representations, in which time-varying ego-motion and static camera-rig geometry are jointly modeled, limiting the utilization of vehicle-side
geometric priors. We propose Trajectory-Rig Decoupled Metric Geometry Learning (TRIG), a geometry perception framework for autonomous driving. TRIG factorizes camera poses into ego-trajectory and camera-rig components, enabling separate modeling of ego-motion and static multi-camera
topology. We introduce decoupled pose encoding and supervision, which separately constrain trajectory
evolution and rig geometry for metric-consistent learning. Moreover, sparse Temporal–Spatial
attention separates cross-camera interaction from temporal aggregation, reducing global
attention cost while preserving
geometric reasoning. Experiments on five autonomous driving
benchmarks show that TRIG achieves state-of-the-art performance in pose estimation, metric depth prediction, and 3D reconstruction.
Read the full abstract ▾
All authors Lizhou Liao, Wentao Xu, Handong Wang, Lirong Yang, Shuai Yang, Weiwei Liu, Chang Huang
Computer Vision
Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning
Yake Wei, Yuan Wang, Fengyun Rao et al. (+2 more)
Recent advancements in
Multimodal Large
Language Models (MLLMs) have evolved from static perception to interleaved visual-
language reasoning, often referred to as “thinking with images”. A basic operation in this
reasoning process is to zoom in on regions of interest (often represented with bounding boxes) to acquire finer visual details. In this paper, we propose
Segmentation before Answering (SegAnswer), which shifts the unit of zoom-in from the popular bounding box to pixel-level
segmentation mask. By employing fine-grained masks to isolate the target area from cluttered environments, segmented visual input yields a more precise region of interest, effectively filtering out redundant background and interfering objects. Furthermore, the discrete patches of segmented visual input align more seamlessly with how MLLMs structure visual
tokens via positional
embeddings. In experiments, we evaluate SegAnswer across diverse
benchmarks, including high-resolution perception, general perception, and
hallucination. It achieves consistent improvements and also exhibits considerable performance on
segmentation tasks, validating its capability for reliable pixel grounding.
Read the full abstract ▾
All authors Yake Wei, Yuan Wang, Fengyun Rao, Jing Lyu, Di Hu
Computer Vision
DeSeG: Decoupling Semantic Intent and Geometric Constraints for Physically Plausible Human-Scene Interaction
Jiakun Li, Zhe Li, Wenqiang Wu et al. (+4 more)
Synthesizing physically plausible human-scene interactions (HSI) remains a critical challenge in computer
vision and the development of human avatars. Although recent generative models enable diverse motion synthesis, they suffer from an inductive bias referred to as semantic-
geometric entanglement. Because spatial constraints often strongly correlate with specific actions in training data, monolithic models will learn the shortcut bias, aggressively overriding the semantic intent when faced with strict
geometric cues. Furthermore, this
entanglement exacerbates physical
hallucinations, such as body-scene penetrations. To address these limitations, we propose DeSeG, a hierarchical framework that explicitly decouples semantic intent from
geometric constraints. First, we introduce a Residual Semantic Planner that encodes textual instructions and canonicalized goal voxels into a compact latent space, enabling fine-grained semantic control independent of spatial trajectories. Second, we propose a physics regularized
diffusion executor that incorporates differentiable repulsive potential fields directly into the
diffusion objective, enforcing collision-aware motion generation. Extensive experiments on the Lingo
dataset demonstrate that DeSeG achieves state-of-the-art performance, reducing mean scene penetration by 47% and improving semantic
alignment by 29% over the SOTA baselines.
Read the full abstract ▾
All authors Jiakun Li, Zhe Li, Wenqiang Wu, Zheng Chang, Mingqi Gao, Jinyu Yang, Feng Zheng
Computer Vision
Benchmarking the Robustness of Autonomous Driving to Environmental Illusions: A Lane Perception Perspective
Tianyuan Zhang, Xianglong Liu, Aishan Liu et al. (+6 more)
Environmental illusions (eg., shadows, reflections, and tire marks) are naturally existing yet overlooked phenomena in real-world driving environments. They can disturb visual perception, leading to misinterpretation of the scene and posing serious safety risks to autonomous driving (AD) systems. However, existing researches largely overlook these phenomena, leaving a critical gap. To address this issue, we study AD robustness through the lane perception perspective, a fundamental task supporting core functions like cruise control and lane centering. We focus on two representative models: conventional lane detection (LD) and
vision-
language model-based systems (ADVLMs). In this work, we introduce the first
benchmark, LanEvil++, for evaluating the robustness of lane perception under environmental illusions. LanEvil++ encompasses 14 types of illusions and leverages the CARLA simulator to generate 94 high-fidelity, fully controllable 3D scenes, yielding a
dataset of 90,292 annotated images, 1,596
video clips, and 41,855 visual question answering pairs. Extensive evaluations demonstrate that environmental illusions substantially degrade the performance of state-of-the-art LD methods. On average, LD models experience a 5.27% drop in Accuracy and a 10.49% decline in F1-score, while ADVLMs show a 2.03% reduction in GPT-score and a 0.75% drop in
Language-score. Among all illusions, shadows emerge as the most disruptive factor, reducing accuracy by up to 7.20%. Furthermore, closed-loop simulations reveal that these illusions can lead to incorrect driving decisions. Complementary real-world case studies highlight safety-critical failures in actual traffic scenes. To enhance robustness, we propose the
Multimodal Illusion Defense Approach (MIDA). MIDA achieves substantial gains under challenging conditions, boosting robustness by 4.23% on LD models and 3.82% on ADVLMs.
Read the full abstract ▾
All authors Tianyuan Zhang, Xianglong Liu, Aishan Liu, Lu Wang, Yitong Zhang, Peng Yue, Mingchuan Zhang, Siyuan Liang, Dacheng Tao
Computer Vision · Cross-listed from cs.RO
FuncBridge: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning
Chuhao Zhou, Liquan Wang, Shuxin Cao et al. (+5 more)
While humans readily repurpose a book, a stone, or a shoe to drive a nail,
robots trained on specific tools fail to transfer the same function to novel ones – a gap we formalize as functional generalization. Functionally equivalent tools share visually recognizable functional intent, such as where contact can occur and how a contact region should move to the target. However, this perceptual similarity does not directly carry over to action space, where each tool demands a different motor pattern to realize the function. To bridge this gap, we explore intermediate representations including affordance images, human
video prompts, functional
videos and object masks, and 2D keypoint trajectories, finding that keypoint trajectories best balance functional expressiveness and action groundability. Building on this, we present FuncBridge, a two-stage framework that decouples functional
reasoning from action execution: learning to predict generalizable keypoint trajectories from action-free data, then grounding them into
robot actions with limited demonstrations. Across a
benchmark spanning ten tools and three functions, including hitting, sweeping, and hooking, FuncBridge consistently outperforms state-of-the-art methods on unseen tools in both simulation and the real world.
Read the full abstract ▾
All authors Chuhao Zhou, Liquan Wang, Shuxin Cao, Xiangyu Chen, Yuxuan Hu, Boyu Ma, Animesh Garg, Jianfei Yang
Computer Vision
LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding
Guang Yang, Brian Siyuan Zheng, Victoria Ebert et al. (+1 more)
We propose a novel pipeline, Legato 2, for extracting symbolic notation and semantic knowledge from images of sheet music. Legato 2 features the first large-scale neural model for optical music recognition (OMR) to operate sequentially on a system-by-system basis, following the horizontal lines of notation as they are read on the page, rather than treating the page as an undifferentiated image, enabling better scaling to arbitrarily long inputs. It is also the first OMR model capable of generating symbolic transcriptions that include embedded textual content, such as titles and annotations. The pipeline combines system-level
segmentation with an autoregressive
vision-LM to capture both local notation details and score structure. Across multiple
datasets, Legato 2 consistently outperforms prior state of the art. We also show that symbolic transcriptions complement visual inputs for frontier
language models, improving their interpretation of dense musical documents. Legato 2 establishes new state-of-the-art performance in both OMR and downstream sheet music understanding.
Read the full abstract ▾
All authors Guang Yang, Brian Siyuan Zheng, Victoria Ebert, Noah A. Smith
Computer Vision
Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator
Zihan Wang, Seungjun Lee, Yinghao Xu et al. (+1 more)
Embodied navigation aims to build
agents that interpret
multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned
datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated
embodied data engine that generates high-fidelity observations, executable actions, and diverse
navigation instructions at scale. It converts large collections of
videos and images into nearly 20K interactive scenes and synthesizes more than 10 million
navigation training samples.
Navigation models trained entirely in these neural environments achieve strong improvements on major
benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for
embodied navigation at scale.
Read the full abstract ▾
All authors Zihan Wang, Seungjun Lee, Yinghao Xu, Gim Hee Lee
Computer Vision
Optimized Adaptive Loop Filter in Versatile Video Coding
Xuewei Meng, Jiaqi Zhang, Chuanmin Jia et al. (+3 more)
In the Versatile
Video Coding~(VVC) standard, adaptive loop filter~(ALF), including Geometry transformation-based Adaptive Loop Filter~(GALF) and Cross Component Adaptive Loop Filter~(CCALF), plays an essential role in reducing compression artifacts. However, it also has high coding complexity and requires many picture buffer accesses in the encoder that will increase external
memory access and is unfriendly to the software and hardware design. Therefore, we propose an optimized ALF framework, including the parallel design of GALF and CCALF, the adaptive parameter decision of GALF, and one-pass CCALF scheme by effectively estimating the CCALF filtering distortion without conducting filter operation. Compared to VTM-8.0, the proposed method can reduce the picture buffer access from 152 to 1 and achieve roughly 25% time-savings of the ALF module with negligible coding performance change under RA configuration. Some of the proposed methods have been adopted in the VVC reference software.
Read the full abstract ▾
All authors Xuewei Meng, Jiaqi Zhang, Chuanmin Jia, Xinfeng Zhang, Shanshe Wang, Siwei Ma
Computer Vision
ARMS: Anchor-Relational Motion Streaming for Seamless Solo-Social Motion Transitions
Huakun Liu, Qing Yu, Kent Fujiwara et al. (+2 more)
Generating temporally continuous and socially coherent human motion from text remains a fundamental challenge, particularly in realistic streams where people act alone, enter interactions, and later disengage. Most existing methods generate fixed-length motion clips under static
agent configurations, which makes them brittle to solo-social transitions and unsuitable for incremental generation over long horizons. We propose ARMS, an Anchor-Relational Motion Streaming framework that unifies solo motion and human-human interaction within a single causal generative process. ARMS introduces a dynamics-asymmetric representation that decouples per-person temporal
evolution from inter-person
alignment via a partner-referenced relative-translation term, enabling seamless switching of social coupling without sacrificing long-horizon stability or spatial consistency between
agents. On top of a causal latent space, a causal relational
diffusion model progressively refines motion segment by segment using only past context, capturing both intra-person temporal dependencies and inter-person relations. Mode-aware relational gating activates or masks cross-
agent connections, allowing the same model to support both solo and interaction generation. Experiments show that ARMS improves transition smoothness and social coherence compared to interaction-centric baselines, while also achieving competitive results on human-human interaction
benchmarks.
Read the full abstract ▾
All authors Huakun Liu, Qing Yu, Kent Fujiwara, Hideaki Uchiyama, Kiyoshi Kiyokawa
Computer Vision
SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs
Hossein Rajoli, Fatemeh Lotfi, Niloufar Alipour Talemi et al. (+3 more)
Pre-trained
Vision-
Language Models (VLMs) like CLIP have proven highly effective as foundation models for various downstream applications. However,
prompt learning in VLMs encounters a performance-generalization dilemma: while
prompts can be tuned to achieve high accuracy on seen distributions, this tuning process often undermines their generalizability to unseen data. The limited set of learnable
prompts, which contextualize and condition the input to steer it toward the task within the pretrained VLM, tends to overfit the training data, leading to a trade-off between task-specific performance and preserving generalization. To address this dilemma, we introduce SAMPLe (Sharpness-Aware Minimization
Prompt Learning), a plug-in sharpness-aware optimizer that enhances
prompt generalizability by accounting for loss landscape sharpness. Unlike conventional methods, SAMPLe balances exploration and exploitation by satisfying objective function constraints at each step, dynamically adapting to the current optimization state based on the local
curvature and gradient properties. This approach reduces overfitting on seen distributions and improves adaptability to unseen data, preserving the generalization potential of pre-trained VLM models. We integrate SAMPLe into multiple
prompt learning frameworks, including CoOp, CoCoOp, MaPLe, TCP, and Co-
Prompt, demonstrating its effectiveness across diverse methods. Experiments show that SAMPLe elevates
prompt learning frameworks and consistently outperforms existing optimizers across diverse settings, establishing itself as a robust, model-agnostic solution for
prompt learning.
Read the full abstract ▾
All authors Hossein Rajoli, Fatemeh Lotfi, Niloufar Alipour Talemi, Hossein Kashiani, Xiaolong Ma, Fatemeh Afghah
Computer Vision
Association Restoration Test: Revealing Restorable Shortcuts after Unlearning
Amy Lu and Changxiu Ji
Association unlearning aims to disable learned label-attribute shortcuts while preserving task performance. Existing evaluations mainly measure output-level robustness or probe whether shortcut attributes remain readable in frozen features, but neither test determines whether a retained association remains functionally usable by the original classifier. We propose the Association Restoration Test (ART), a post-hoc diagnostic for functional shortcut restorability. ART estimates class-conditional association directions, amplifies residual components, and evaluates the modified features with the original classifier head. Across Waterbirds, CelebA, SpuCoDogs, and an ISIC timestamp-artifact extension, we show that output metrics, representation probes, and ART characterize distinct aspects of shortcut mitigation. These findings motivate restoration-aware evaluation for unlearning and shortcut-mitigation methods that target learned associations rather than individual classes or concepts.
Read the full abstract ▾
Computer Vision
Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
Zhiwei Yang, Yuanchen Wu, Nan Zhang et al. (+3 more)
Multimodal Large
Language Models (MLLMs) have demonstrated strong perception and
reasoning capabilities. However, most existing models focus on isolated objects and neglect structured relationships for efficient target
navigation, limiting their performance on visually intensive tasks. To address this challenge, we introduce Scene Graph Thinking (SaGe), a novel paradigm that enables fine-grained and structured visual
reasoning through explicit scene-graph representations. Specifically, we first introduce an automated data engine that converts flat image-text corpora into structured scene graphs, where hierarchical entities constitute the nodes and diverse visual relations define the edges. Building upon this, we construct 120K high-quality training data by sampling
reasoning traces from scene graphs. Then, two-stage graph-aligned post-training paradigms are introduced, where supervised fine-tuning internalizes MLLMs with structured
reasoning, and subsequent reinforcement fine-tuning proposes node-as-proxy graph rewards to consolidate efficient graph exploration. With curated data and graph-aligned training, our approach achieves significant improvements across eight
multimodal benchmarks, demonstrating strong effectiveness on fine-grained perception and
reasoning tasks. Code is available at https://github.com/zwyang6/SaGe.
Read the full abstract ▾
All authors Zhiwei Yang, Yuanchen Wu, Nan Zhang, Yucong Meng, Ke Yan, Shouhong Ding
Computer Vision · Cross-listed from cs.LG
FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models
Bowen Xue, Zihan Min, Xingyang Li et al. (+8 more)
Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications. However, post-training of large
diffusion models is still challenging due to the prohibitive
memory footprints and slow training speed, which existing parameter-efficient fine-tuning methods only partially address. To overcome these limitations, we propose FourTune, an efficient post-training framework for
diffusion models based on an end-to-end W4A4G4 paradigm. FourTune introduces a triple-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization-sensitive outliers, enabling stable training under native 4-bit computation. In addition, FourTune employs hardware-efficient block-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce
memory bandwidth overhead. Across customization, reinforcement learning, and
distillation tasks, FourTune matches the quality of full-precision fine-tuning. On FLUX.1-dev (12B), FourTune reduces
memory overhead by 2.25$\times$ and increases end-to-end training throughput by 2.27$\times$ compared to BF16 LoRA.
Read the full abstract ▾
All authors Bowen Xue, Zihan Min, Xingyang Li, Zhekai Zhang, Haocheng Xi, Lvmin Zhang, Maneesh Agrawala, Jun-Yan Zhu, Song Han, Yujun Lin, Muyang Li