Computer Vision
SAM3-LoRA: Parameter-Efficient Adaptation of a Concept-Promptable Foundation Model for Multi-Class Structural Defect Segmentation
P. Malaisree, S. Youwai, S. Janrungautai et al. (+3 more)
Promptable
segmentation foundation models such as SAM3 accept an open-vocabulary text concept and return every instance matching it, but adapting them to a specialized domain by full fine-tuning is computationally prohibitive for the organizations that would benefit most. This study applies Low-Rank Adaptation (LoRA) to SAM3 for multi-class structural defect
segmentation and examines both how such a model can be supervised from conventional annotation and whether the resulting efficiency gain transfers across
datasets. Two contributions are methodological. First, we describe a supervision procedure that trains a concept-promptable model directly from COCO-style class-labeled instance
segmentation by using the category name itself as the
prompt, requiring no
prompt templates, no synonym expansion, and no learned class
embeddings. Second, we identify and mitigate a failure mode specific to this setting: because a conventional annotation file yields positive
prompts exclusively, the model's presence prediction decouples from the text condition and degenerates into responding to any
prompt, a collapse that is invisible to every metric computed on positive
prompts alone. Exhaustive hard-negative prompting, in which every
dataset category absent from an image is issued as a zero-detection query, addresses this at no annotation cost. Two adapter placements were compared under an identical protocol, updating 0.121% and 1.341% of model parameters. On a purpose-built tunnel lining
dataset, pixel intersection-over-union improved from 0.017 to 0.338 and instance-level recall from 0.375 to 0.672; on the independent public Structural Defects
Dataset, from 0.017 to 0.855 and from 0.574 to 1.000. Improvements were directionally consistent across ten metrics on both
datasets, and the largest per-category gains occurred precisely where zero-shot competence was absent.
Read the full abstract ▾
All authors P. Malaisree, S. Youwai, S. Janrungautai, D. Amorndechaphon, P. Rojanavasu, W. Songkitti
Computer Vision
Instance-Guided Report Anchoring for Text-Free 3D Abnormality Segmentation in Chest CT
Zhenyu Bu, Haoyan Ding, Chushu Shen et al. (+7 more)
Accurate 3D abnormality
segmentation in chest CT requires dense spatial supervision, but obtaining expert voxel-level labels is costly. Radiology reports, however, are routinely generated during clinical interpretation and contain instance-specific descriptions that can provide additional guidance without new dense annotation. Existing
vision-
language grounding methods typically require report-derived findings at inference, making localization dependent on paired text and limiting each forward pass to a queried finding. We propose Instance-Guided Report Anchoring (IGRA), a model-agnostic module that preserves the correspondence between each annotated abnormality instance and the report finding that describes it. IGRA pools each instance representation and anchors it to the corresponding finding
embedding during training; all text-related components are discarded at inference. We further reformulate free-text grounding on ReXGroundingCT as multi-label volumetric
segmentation by merging same-category instances, allowing all abnormality categories to be predicted in one image-only forward pass. IGRA improves Dice by 22.5% over the strongest image-only baseline (30.93 vs. 25.25) and is comparable to VoxTell on the single-finding subset (30.29 vs. 30.43). Applied unchanged to four standard 3D
segmentation backbones, IGRA improves Dice and hit rate across all architectures. Zero-shot evaluation on LIDC-IDRI, PleThora, and a private in-house
dataset further shows consistent gains over image-only baselines.
Read the full abstract ▾
All authors Zhenyu Bu, Haoyan Ding, Chushu Shen, Xinyuan Zheng, Peiyu Duan, Xueqi Guo, Sepehr Farhand, Yoshihisa Shinagawa, Gerardo Hermosillo, Chaowei Wu
Computer Vision · Cross-listed from cs.LG
CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning
Baraa Bilbeisi, Mengchen Fan, Baocheng Geng et al. (+1 more)
Conventional
federated learning (FL) relies on parameter averaging, which forces clients to be doubly homogeneous: it demands an identical architecture and degrades under non-IID data. Real-world deployments usually break both assumptions. We sidestep both by building a decentralized knowledge
distillation framework in which each client evaluates its peers' model snapshots on its own local data and distills from the resulting soft predictions. Because knowledge is transferred through the shared class posterior, clients are free to run different architectures; and because every teacher is evaluated on the student's own device, raw data never leaves the client, with no central server or public
dataset required. Within this setting, we identify and address an under-examined problem: how to combine the peer teacher predictions. Existing methods, like uniform averaging, ignore how knowledge reliability varies across teachers and classes. We propose Class-wise Reliability-Aware
Distillation (CRAD), which, per class, first discards teachers that disagree with the peer consensus and then takes a weighted average of the rest, weighting each teacher by its per-class reliability (precision, or inverse variance). Since the variance of an accuracy from $n$ samples scales as $1/n$, support enters automatically: among the teachers that survive filtering, a teacher is trusted for a class to the degree that it is both accurate and well-evidenced for it. On three image-classification
benchmarks (CIFAR-10, CIFAR-100, and PathMNIST colon pathology), across heterogeneous architectures under severe non-IID skew, CRAD consistently outperforms competing methods in global accuracy.
Read the full abstract ▾
All authors Baraa Bilbeisi, Mengchen Fan, Baocheng Geng, Qing Tian
Computer Vision
Unmasking Face Embeddings: Reading, Rendering and Naming with Foundation Models
Fizza Rubab, Yiying Tong and Arun Ross
Modern face recognition (FR) owes much of its success to deep neural networks that learn to extract compact identity
embeddings from face images. These models are typically trained for identity discrimination, producing
embeddings that are highly effective for biometric matching but largely opaque to semantic interpretation. In contrast, foundation models, pretrained on broad visual or
vision–
language tasks, provide rich interfaces for describing, retrieving, generating, and organizing visual content. This contrast raises a natural question: what capabilities become available when face
embeddings from domain-specific FR models are made interoperable with foundation models? Building on recent work on
embedding compatibility across models, we use simple pre-computed linear transformations, estimated from paired
embeddings alone, to connect existing FR models with off-the-shelf foundation models. Once aligned with a foundation model, a face
embedding can be 'unmasked' in multiple ways, without training or modifying either model: it can be read in natural
language, enabling free-form text queries over a gallery of FR
embeddings; rendered into a face image that recovers a person's appearance, using an unmodified
diffusion decoder; and converted to a name, enabling identification even in the absence of an enrolled face gallery. In effect, one linear transformation turns an identity
embedding into a rich
embedding for web-scale foundation models. This interoperability exposes face
embeddings as semantically and visually rich biometric representations, with direct implications for interpretability,
retrieval, reconstruction, and template security.
Read the full abstract ▾
Computer Vision
SlideMix: Enhancing Whole Slide Image Analysis via Multimodal Shuffling
Chad Wong, Sicheng Chen, Tianyi Zhang et al. (+4 more)
Histopathological whole slide images (WSIs) are central to cancer diagnosis, but their gigapixel scale, tissue heterogeneity, weak slide-level supervision, sparse diagnostic regions, and multi-scale evidence make robust automated analysis challenging. Multiple instance learning (MIL) is widely used to aggregate tile-level features into slide-level predictions, yet existing augmentation strategies often perturb tissue regions without preserving diagnostic relevance, slide context, or cross-scale structure. We propose SlideMix, a model-agnostic
multimodal augmentation framework for MIL-based WSI analysis. SlideMix uses a
retrieval-augmented
vision-
language model (VLM)-based Visual-
Language Adaptive Region selector to identify diagnostically relevant regions and reduce weak-label noise. It then performs In-place Tile Shuffling within meaningful tissue regions to mix feature
embeddings while preserving slide-level context. A VLM-based soft-labeling module supervises mixed samples, while a multi-factor, loss-driven online Curriculum-Learning Feedback scheme adaptively controls shuffle granularity, feature similarity, and shuffle ratio to promote cross-scale representation learning. Across 11 WSI
datasets comprising 20,523 slides, 8 diagnostic tasks, and 10 WSI backbones, SlideMix improves accuracy and generalization in most settings and compares favorably with established augmentation baselines, providing a simple plug-and-play approach for more robust and scalable digital pathology models. Source code: https://github.com/Xia-Research-Lab/SlideMix
Read the full abstract ▾
All authors Chad Wong, Sicheng Chen, Tianyi Zhang, Enhui Chai, Yueming Jin, Zeyu Liu, Fei Xia
Computer Vision
FoldingAgent: Inferring Parametric Origami Procedures from Demonstration Videos
Maya Moriya, Sigal Raab, Yael Vinker et al. (+1 more)
We present FoldingAgent, an agentic framework for inferring explicit parametric folding programs directly from origami demonstration
videos. Our framework leverages the
reasoning power of a pre-trained
Vision-
Language Model (VLM) equipped with a suite of specialized tools that enable the
agent to simulate
geometric transitions, verify physical plausibility, retrieve and compare visual content, and evaluate its own predictions. To translate visual content into folding programs, we define a parametric space that consists of the paper's geometry and a set of parametric folding actions. Unlike models that predict static crease patterns, our
agent operates sequentially and possesses the ability to re-plan its actions, effectively mitigating the compounding errors inherent in multi-step folding. Our approach takes a step toward closing the gap between human origami knowledge, which is primarily shared through unstructured visual demonstrations, and computational methods, which typically rely on structured, parametric representations such as a crease pattern or an executable parametric plan. We evaluate our approach on PurelandFold, a newly curated
benchmark of diverse Pureland origami
videos with ground-truth geometry and action labels. Our results demonstrate that by combining VLM
reasoning with a set of specialized tools and physical simulation, we can successfully transform unstructured visual demonstrations into executable, physically plausible folding procedures.
Read the full abstract ▾
All authors Maya Moriya, Sigal Raab, Yael Vinker, Tali Dekel
Computer Vision · Cross-listed from cs.LG
Adapting Without Gradients: Affine Statistics Transport and What Its Certificate Can Tell You
Salim Khazem and Ibrahim Mohamed Serouis
Test-time adaptation (TTA) typically assumes that model parameters can be updated at inference time. This assumption is restrictive for inference-only accelerators, frozen or third-party models, and
memory-constrained deployments, and standard BatchNorm-based TTA configurations may also become inactive on architectures without BatchNorm. We study adaptation when the learned model must remain frozen. We introduce CASTER, a gradient-free method that stores source class statistics in a discriminative subspace, estimates a class-shared affine transformation from target-batch moments, and analytically transports the source class distributions before classification. CASTER requires no backward pass, optimizer state, or stored source feature bank. Across four backbones and seven
datasets, it outperforms k-NN on identical frozen features in 27 of 28 backbone-
dataset settings while retaining a median of 18x less state. Affine transport is not always reliable. On ImageNet-C, where batches contain only 64 samples for 1000 classes, unconditional transport loses 21.2 top-1 points. We therefore introduce an empirical residual-to-margin transportability certificate. Across 307 evaluation
cells, every transport losing more than 10 points has certificate value above 3.9, although benign and destructive regimes are not perfectly separated. Gating converts an average $-3.35$-point effect of unconditional transport into a +1.69-point gain, and performance remains within 0.3 points of the best threshold over a broad threshold range. Finally, we show that this certificate is mechanism-specific: when applied to Tent, it accepts only $4.3\%$ of updates and preserves 0.6% of Tent's available gain. These results position CASTER as a lightweight adaptation mechanism for frozen-model deployment, together with an explicit account of when its safety signal is informative and when it is not.
Read the full abstract ▾
Computer Vision
Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation
Vida Adeli, Soroush Mehraban, Jacob Rommann et al. (+3 more)
Generating co-
speech gestures that are temporally coherent, semantically aligned with
speech, and grounded with surrounding objects remains challenging. Prior
speech-driven gesture models emphasize audio-gesture
alignment but do not explicitly account for posture constraints or surrounding objects, failing to capture the inherent correlation between body gestures and the physical space. We present Puppeteer, a posture-aware, object-grounded co-
speech gesture
diffusion model operating in a causal latent space. We decompose long gestures into structured primitives and learn a causal variational autoencoder that encodes them into temporally ordered latent
tokens, each depending only on the past. We then perform conditional
diffusion directly in the causal latent space, conditioning on
speech signals, motion history, an initial posture reference, and object geometry to synthesize physically consistent gestures. This temporally ordered latent formulation enables explicit temporal control and supports tasks such as gesture in-betweening and gesture completion. To better assess co-
speech gesture synthesis beyond existing measures, we introduce new evaluation metrics tailored to this task. We also created SceneGes, the first curated synthetic 3D
dataset of
embodied co-
speech gestures and corresponding 3D objects, enabling object-grounded gesture generation. Experiments show that Puppeteer generates more diverse and temporally synchronized gestures than prior methods, while enabling object-grounded gesture synthesis.
Read the full abstract ▾
All authors Vida Adeli, Soroush Mehraban, Jacob Rommann, Harrison Sanborn, Cole Clifford, Babak Taati
Computer Vision · Cross-listed from cs.AI
Dr. Claw: An AI Scientist Workspace for Vibe Research
Dingjie Song, Hanrong Zhang, Dawei Liu et al. (+10 more)
Command-line coding
agents (e.g., Claude Code, Gemini CLI) can already read and write files and sustain long sessions, yet end-to-end research still fragments across chat tools, IDEs, terminals, and writing environments, and the decisions that make it auditable are rarely preserved. We present Dr. Claw, an open-source workspace that wraps existing coding-
agent executors in a controllable and auditable human-in-the-loop workflow rather than introducing another autonomous
agent. Persistent state objects, a reusable skill library, and multi-executor coordination link human decisions to AI execution, turning planning, execution, and writing into one traceable, recoverable loop. We demonstrate Dr. Claw through an interactive three-view scenario and a failure-recovery walkthrough, and evaluate it against a bare command-line
agent sharing the same backend executor, so the comparison contrasts the whole orchestration layer (task graph, state objects, and skill library) with the
agent it wraps. Holding the executor fixed, Dr. Claw scores higher on research completeness while persisting an auditable, recoverable process trail. Demo access: repository https://github.com/OpenLAIR/dr-claw, released under AGPL-3.0 with GPL-3.0 upstream components.
Read the full abstract ▾
All authors Dingjie Song, Hanrong Zhang, Dawei Liu, Yixin Liu, Zongxia Li, Zhengqing Yuan, Siqi Zhang, Henry Peng Zou, Zhiling Yan, Yuxuan Zhang, Yanfang Ye, Philip S. Yu, Lichao Sun
Computer Vision
Where Should Experience Live? Hierarchical Hebbian Memory for Continual Vision Transformers
Mohammed Yusuf Mujawar and Noorbakhsh Amiri Golilarz
Vision Transformers provide strong visual representations but typically rely on slowly updated parameters, limiting their ability to organize newly acquired information across different
memory timescales. This work proposes Hierarchical Hebbian
Memory, a three-level
memory architecture composed of rapid Working
Memory, persistent Routed Episodic
Memory, and slower Semantic
Memory. A learned controller regulates
memory contribution, read and write routing, plasticity, retention, and consolidation. A causal read-before-write lifecycle ensures that the current outcome cannot influence the prediction it supervises. The architecture is evaluated on Omniglot 5-way 1-shot recognition and CORe50 continual object recognition. With Swin-Tiny, the hierarchical model reaches 97.39% accuracy on Omniglot and 95.37% final accuracy on CORe50 when combined with experience replay. Learned multi-bank
retrieval reaches 47.50% delayed-association accuracy, compared with 24.17% for a single persistent bank and 25.00% without
memory. After intervening distractors, Episodic
Memory retains approximately 0.96 cosine similarity with stored associations, while Working
Memory falls to approximately 0.05. These results show that Hebbian association and learned
memory routing can jointly organize online visual experience across rapid, persistent, and consolidated
memory timescales within
Vision Transformers.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models
Jungseob Lee, Seongtae Hong, Dongyub Jude Lee et al. (+4 more)
Speculative decoding accelerates generation without changing its output, but on
vision-
language models (VLMs) a self-reinforcing cycle holds it back. Because an autoregressive drafter pays a sequential pass for each drafted
token, it must stay small and can ill afford to attend to the image at each pass. Prior work therefore compresses or hides the image, leaving the drafter weakest on the text the image determines. We present GLANCE, a one-pass block drafter that breaks this cycle on an unmodified VLM target. Its block-
diffusion head drafts a whole block in one forward pass over the target's already fused
vision-
language states, reading the
multimodal context once, however deep the draft. The target verifies a wide candidate tree in one pass and commits exactly its greedy output. In one production engine at a fixed round budget, GLANCE decodes up to 3.05 times faster than autoregressive decoding and outpaces the production EAGLE3-VL head on average and by about 11% on grounded tasks. An
entropy law explains when drafting pays, predicting the longest accepted blocks on grounded tasks, where the target's next-
token entropy is lowest. Our code is available at https://github.com/js-lee-AI/GLANCE.
Read the full abstract ▾
All authors Jungseob Lee, Seongtae Hong, Dongyub Jude Lee, Chanjun Park, Jaehyung Seo, Sugyeong Eo, Heuiseok Lim
Computer Vision
TRUST: Threshold-Recalibrated Uncertainty-Safe Training for Certified Dismissal in Breast Cancer Screening
Parham Hajishafiezahramini, Matthew Hamilton, Edward Kendall et al. (+2 more)
Reducing the review of clearly cancer-negative screening mammograms could lower radiologist workload without compromising cancer detection. We propose a closed-loop threshold-aware training strategy in which the dismissal threshold is recalculated during training and used to penalize cancer-positive images that approach the dismissal region. We evaluated the method on NLBS and RSNA using five controlled training configurations, with case-level assessment based on a one-sided 99% Clopper–Pearson upper bound for cancer prevalence among dismissed cases. The proposed model achieved the highest case-level dismissal rates at both 98% and 95% recall targets. On NLBS, dismissal reached 19.74% and 21.70%, while the cross-
entropy baseline did not meet either recall target. On RSNA, dismissal improved from 7.04% to 14.31% and from 13.49% to 19.69%. In external RSNA$\to$NLBS evaluation, the proposed model achieved dismissal rates of 12.95% and 19.87% at the 98% and 95% recall targets, respectively. These results support closed-loop threshold-aware training for high-recall selective dismissal.
Read the full abstract ▾
All authors Parham Hajishafiezahramini, Matthew Hamilton, Edward Kendall, Gregory Doyle, Oscar Meruvia Pastor
Computer Vision
StreamScout: Learning When to Look Deeper for Streaming Video Understanding
Ce Zhang, Jing Bi, Jinxi He et al. (+9 more)
Streaming
video understanding requires answering questions that arrive at arbitrary moments over an unbounded
video stream. Existing systems primarily focus on what to retain in a bounded
memory, yet access that
memory using the same fixed-cost procedure for every query, despite substantial variation in the evidence required. We argue that deciding how deeply to access
memory for each query is as important as deciding what the
memory should store. To this end, we introduce StreamScout, an adaptive inference framework that maintains only a lightweight textual timeline in context as the stream unfolds. At query time, StreamScout progressively augments the timeline with up to three increasingly informative visual views: a glance at recent frames, a uniform look-back over the past stream, and query-salient
retrieval. At each stage, the model answers immediately if the available evidence is sufficient; otherwise, it escalates to the next view. To improve this stop-or-escalate policy, we probe the cascade on an auxiliary set and distill the model's empirical competence boundary into supervision for a lightweight LoRA adaptation, yielding StreamScout-S. We further refine the policy through reinforcement learning, allowing the model to explore stopping behaviors beyond imitation of the distilled decisions, yielding StreamScout-R. Across three backbones and three streaming
benchmarks, StreamScout and its variants consistently outperform prior streaming methods while substantially reducing inference cost and
token consumption; on OVO-Bench, for instance, StreamScout-S improves Qwen3-VL-8B by 14.65 points while using 59% fewer
tokens than uniform sampling and answering in 1.04 s on average.
Read the full abstract ▾
All authors Ce Zhang, Jing Bi, Jinxi He, Jianshu Zhang, Jingyang Lin, Yunzhong Xiao, Minghao Fu, Yaqi Xie, Zhentao Xie, Weicong Chen, Katia Sycara, Ming Zhou
Computer Vision
CrossFeat: Bridging Imaging Modalities in Feature Descriptor Space
Paul Schneider and Nazim Haouchine
Most advances in keypoint descriptions address monomodal settings, where image variations arise from viewpoint, illumination, or contrast changes.
Multimodal scenarios involve images produced by fundamentally different sensing processes, such as multispectral imaging, RGB-depth, satellite imagery, or medical imaging, causing the same structures to appear differently. A common solution to cross-modal description is to train descriptors for each modality pair, which requires retraining whenever the modalities change, or to train large models, which incur a significant increase in runtime. Instead, we propose CrossFeat, a framework that enables an existing monomodal descriptor to operate across modalities. Our method learns a crossing function in descriptor space that maps features from one modality to a representation compatible with another. To preserve the structural information captured by the original descriptor, CrossFeat introduces a geometry-appearance disentanglement such that only appearance is altered while the
geometric properties are preserved. Experiments across multiple domains and
datasets demonstrate improved performance in
multimodal matching.
Read the full abstract ▾
Computer Vision
CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Prediction
Zhengxu Tang, Guofeng Cui, Ziyu Gong et al. (+8 more)
Long-tail autonomous driving failures are often framed as rare-object recognition errors. We argue that this view is incomplete: the decision-critical question is not only whether a model recognizes an unusual object, but whether it infers how that object changes the ego vehicle's feasible high-level actions. We formalize this problem as decision-level driving affordance prediction, where a model maps a front-view image, ego-motion history, and
navigation command to a structured longitudinal–lateral meta-action. To evaluate this capability, we introduce CoLT-Drive, a 3,536-sample counterfactual long-tail
benchmark that inserts rare objects into otherwise fixed driving scenes and measures whether models predict acceptable action pairs. To improve deployable small VLMs, we propose KPA, a knowledge-preserving adaptation framework that combines structured perception-to-decision prompting, SLERP-based expert merging, and RegMoE, a regime-aware LoRA mixture-of-experts module. KPA preserves the pretrained model's open-world knowledge while allocating lightweight adaptation capacity to different driving decision regimes. Experiments on an in-domain driving split and CoLT-Drive show that KPA achieves 60.8% pair accuracy on CoLT-Drive, outperforming the pretrained Qwen3-VL-2B baseline (50.3%) and LoRA SFT (32.4%) while maintaining competitive in-domain accuracy. Our
benchmark and code are available at https://huggingface.co/
datasets/tangzx2024/CoLT-Drive and https://github.com/tangzhengxu/CoLT-Drive.
Read the full abstract ▾
All authors Zhengxu Tang, Guofeng Cui, Ziyu Gong, Xiaozhou Zhang, Ruifeng Deng, Chengzhi Qi, Ke Chen, Sachin Patil, Tianjun Xiao, Langechuan Liu, Pichao Wang
Computer Vision
Beyond Blind Compliance: Benchmarking Task Verification in OCR Reasoning
Yue Zhou, Yuan Wu and Yi Chang
Multimodal Large
Language Models (MLLMs) have achieved strong performance on OCR-centric document understanding and text-rich visual
reasoning benchmarks. Yet existing evaluations largely assume that every task is valid and answerable. In real-world OCR scenarios, this assumption often fails: questions may rely on illegible text, occluded evidence, nonexistent visual targets, contradictory premises, or missing variables. We study this reliability gap as OCR-grounded Task Verification: before answering, a model should determine whether the Image Premise (IP), Textual Premise (TP), and Question (Q) jointly define an executable task. We introduce VeriOCRBench, a 1,800-sample human-verified
benchmark built from source images drawn from 8 OCR-related
datasets and spanning 8 real-world image domains, with controlled, image-grounded diagnostic tasks. It contains 1,600 trap-injected invalid tasks across 8 trap types and four verification dimensions—Visual, Contextual, Factual, and Logical—plus 200 trap-free controls for measuring over-refusal. Built with a Visual Atomic Fact (VAF)-anchored pipeline and full human
auditing, VeriOCRBench enables decoupled evaluation of task verification, root-cause diagnosis, and over-refusal. Evaluating 15 leading MLLMs reveals persistent blind compliance, diagnosis failures, and
prompt-induced over-refusal, exposing a critical reliability gap in current OCR
reasoning systems. The code is available at: https://github.com/zy001122/Beyond-Blind-Compliance.
Read the full abstract ▾
Computer Vision
Peiyang Xu, Xiaopei Zhu, Jun Zhu et al. (+1 more)
Existing research on object
hallucination in
multimodal large
language models (MLLMs) predominantly attributes the problem to
language priors such as over-reliance on textual co-occurrence statistics. We challenge this view by presenting quantitative evidence for a complementary, under-explored cause: visual-origin
hallucination, where
hallucinations arise from incorrect visual feature extraction and misalignment between image and text
embeddings. Through cosine similarity analysis and Smooth Grad-CAM
entropy measurements, we show that hallucinated samples exhibit systematically lower image-text similarity (average 0.158 vs. -0.122) and inverted
attention patterns, where
attention is dispersed when the target object is present but wrongly concentrated when it is absent. Guided by this diagnosis, we propose
Adversarial Contrastive Fine-Tuning (ACFT). ACFT uses an
Adversarial Hallucination Attribute Flipping (AHAF) procedure, involving minimal, targeted
adversarial perturbations that flip an image's
hallucination attribute, to construct perfectly aligned positive-negative pairs, which are then used for contrastive fine-tuning. AHAF simultaneously serves as a diagnostic probe, revealing that MLLM visual representations lie dangerously close to
hallucination decision boundaries. Requiring only 0.9% of the COCO
dataset and adding zero inference overhead, ACFT achieves state-of-the-art performance on POPE, MME, and four description-level
hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL. Code is available at https://github.com/zxp555/ACFT_MM
Read the full abstract ▾
All authors Peiyang Xu, Xiaopei Zhu, Jun Zhu, Xiaolin Hu
Computer Vision
Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderation
Ruotong Wang, Zihao Zhu, Siwei Lyu et al. (+2 more)
Despite their growing use in
video moderation,
multimodal large
language models (MLLMs) exhibit a compositional safety blind spot:
videos composed of seemingly benign components can convey harmful meaning when interpreted as a whole. We refer to this phenomenon as Distributed Implicit Harm (DIH), where harm arises from relations among components distributed along a decomposition axis of the
video, rather than from any single explicit cue. Among many possible axes, we study two representative cases: temporally distributed harm across visual segments (DIH-T) and cross-modal harm between audio and visual streams (DIH-M). Studying and mitigating DIH at scale requires data that is difficult to collect: such
videos lack compositional harm annotations, evade
retrieval based on local visual cues, keywords, or single-modality signals, and are consequently absent from existing safety
datasets. To bridge this gap, we develop a multi-
agent synthesis framework that composes individually benign components into harmful scenarios and generates diverse DIH
videos with explicit
reasoning annotations, yielding a
dataset of over 9,000
videos spanning visual-only and audio-visual settings. Benchmarking over 30 MLLMs spanning frontier proprietary models and leading open-source systems reveals substantial and consistent deficits in detecting both DIH-T and DIH-M. Notably, this failure persists even among the strongest frontier models: they often correctly assess individual components in isolation but fail to recognize the harmful meaning that emerges from their composition. We further evaluate these models on a manually collected set of real-world DIH
videos from social media and observe the same failure mode, highlighting DIH as a practical and underexplored challenge for
video moderation.
Read the full abstract ▾
All authors Ruotong Wang, Zihao Zhu, Siwei Lyu, Xin Tao, Baoyuan Wu
Computer Vision
A Lagrangian View of Flow Matching
Peyman Milanfar
Modern explicit-time generative models, such as Flow Matching [Lipman et al., 2023] and Rectified Flow [Liu et al., 2023], are typically derived top-down via Optimal Transport and the continuity equation. This standard Eulerian approach focuses on the macroscopic transport of probability mass. In this paper, we present an alternative, bottom-up mechanical derivation grounded in a Lagrangian (particle-centric) perspective. By analyzing the local Taylor expansion of a continuous denoiser, we motivate a strict invariance condition required for optimal, singlestep generation: the conservation of target identity. Enforcing this condition yields a governing quasi-linear advection Partial Differential Equation (PDE). We demonstrate that solving this PDE via the Method of Characteristics analytically yields the straight-line trajectories of Flow Matching. This
geometric perspective isolates the Jacobian of the denoiser as the primary source of trajectory
curvature, providing a direct mathematical explanation for why straight-line flows enable massive step sizes, and why empirical models require
distillation to flatten intersecting characteristics.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
LLM-Driven Autonomous Vehicles Can Inherit Human Driver Biases in Pedestrian Yielding: Results and Implications From a New Benchmark
Irem Yoldas, Jie Zhang, Odinaldo Rodrigues et al. (+1 more)
Public trust in Autonomous Vehicles (AVs) may depend not only on technical success but also on the
fairness of their decision making. While a recent trend in AV research involves using general purpose "common sense" models to guide AV decision making, the degree to which these inherit human biases in driving is still understudied. Given that psychology studies have shown human driver biases exist, such as lower pedestrian-yielding rates to Black pedestrians in the US, we argue that analyses of model bias should also be part of AV evaluation. Concretely, in this paper we propose two new bias testing methodologies for Large
Language Models (LLMs) and Visual-
Language Models (VLMs), "All Else Being Equal" tests and "Self-Consistency" tests, and a new
benchmark (FairYield) to assess bias in pedestrian-yielding decisions. Our findings show that the yielding decisions of both LLMs and VLMs can be influenced by models' predictions of pedestrian gender, ethnicity, religion, disability, age, skin tone, and socio-economic status. While the type and degree of bias is different from model to model, we highlight common patterns, and raise questions about the "common sense" model paradigm, particularly the need to either revise the paradigm or address issues of downstream bias.
Read the full abstract ▾
All authors Irem Yoldas, Jie Zhang, Odinaldo Rodrigues, Martim Brandão
Computer Vision
ZimaBlue: Evolving Generalizable World Action Models through Scalable Video Pre-training
Xionghao Wu, Yijun Yang, Shiyang Zhou et al. (+17 more)
Robotic manipulation faces a fundamental scaling challenge: robust generalization demands broad physical experience, yet action-labeled
robot trajectories are expensive to collect and inherently limited in diversity.
Egocentric videos offer a far more scalable source of
embodied experience, capturing object interactions, contact dynamics, tool use, and long-horizon behaviors across diverse environments. The central challenge is how to convert this abundant but action-free experience into effective
robot control. We introduce ZimaBlue, a scalable framework for learning generalizable World Action Models (WAMs) from large-scale
video. ZimaBlue follows a three-stage training curriculum: it first performs causal
embodied video pre-training on large-scale human and
robot egocentric videos, then grounds the learned visual dynamics in heterogeneous
robot trajectories through
video-action mid-training with a unified action representation, and finally specializes the model to a target
robot for deployment. To make generative WAMs practical for real-time control, ZimaBluefurther adopts an asynchronous Slow-Fast dual-system architecture, where a high-capacity Slow world model provides generalizable spatiotemporal representations and a lightweight Fast branch enables 30 Hz action prediction on NVIDIA RTX 4090. On real-
robot zero-shot evaluations, scaling from target-
robot data alone to over 120,000 hours of
embodied video improves success from 36.1% to 77.8%. ZimaBlue further delivers strong performance across multiple
benchmarks, with particularly pronounced gains on unseen tasks.
Read the full abstract ▾
All authors Xionghao Wu, Yijun Yang, Shiyang Zhou, Haoze Sun, Jianhui Liu, Songsong Yu, Jiyao Zhang, Wenbo Li, Bo Wang, Guoqing Ma, Lin Song, Renjie Liao, Shenghe Zheng, Wei Tang, Xiaojuan Qi, Yanwei Li, Yuan Zhang, Zhuotao Tian, Haoyang Huang, Nan Duan
Computer Vision
Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving
Xin Zhou, Zongchuang Zhao, Zhibo Yang et al. (+13 more)
We present Qwen-Drive-1.0, an initial step towards a
vision-
language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained
vision-
language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird's-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map
segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose
vision-
language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general
vision-
language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.
Read the full abstract ▾
All authors Xin Zhou, Zongchuang Zhao, Zhibo Yang, Mingsheng Li, Humen Zhong, Shuai Bai, Du Chu, Ruizhe Chen, Zhaohai Li, Jun Tang, Qiuyue Wang, Mingkun Yang, Jiazhao Zhang, Dayiheng Liu, Dingkang Liang, Xiang Bai
Computer Vision
BRF-GS: Hyperspectral Bidirectional Reflectance Factor Modeling and Image Generation Based on 3D Gaussian Splatting
Yiling Yao, Wenjuan Zhang, Bowen Wang et al. (+3 more)
The bidirectional reflectance factor (BRF) characterizes the directional radiative properties of terrestrial surfaces. However, existing three-dimensional (3D) radiative transfer models require complex scene construction and computationally intensive radiative transfer solvers, limiting efficient generation of multi-angle hyperspectral reflectance imagery. 3D Gaussian Splatting (3DGS) offers an efficient framework for neural scene representation and novel view synthesis, but its low-order spherical harmonics representation is insufficient for complex directional reflectance, while the high dimensionality and inter-band quality differences of hyperspectral data introduce additional challenges. To address these challenges, we propose BRF-GS, a 3DGS-based framework for BRF modeling and hyperspectral reflectance image generation. BRF-GS introduces a hybrid BRDF-driven kernel to represent complex directional reflectance, selects geometry-reliable spectral bands for robust 3D scene initialization, and adopts a two-stage training strategy that decouples geometry optimization from spectral modeling. We further construct the AIR-BRF
dataset, a multi-angle hyperspectral directional reflectance
dataset comprising three scenes with diverse natural and artificial targets. Experiments demonstrate that BRF-GS achieves superior spatial and spectral fidelity and accurately reproduces characteristic view-dependent BRF responses. The proposed framework provides an efficient data-driven approach for BRF modeling and multi-angle hyperspectral reflectance image generation in remote sensing scenes.
Read the full abstract ▾
All authors Yiling Yao, Wenjuan Zhang, Bowen Wang, Bocheng Li, Wentao Song, Bing Zhang
Computer Vision
BLARM: Animating 3D Objects from Video via Blending Latent Rigid Motion Primitives
Pradyumn Goyal, Yizhak Ben-Shabat, Hsueh-Ti Derek Liu et al. (+6 more)
We introduce BLARM, a feed-forward method for
video-driven 3D mesh animation. Given a
monocular video and a static object mesh, BLARM predicts a temporally coherent animated mesh whose motion follows the
video. Rather than relying on explicit rigs or directly regressing high-dimensional vertex motion, we represent animation using a compact set of learned, time-varying rigid motion components and time-invariant vertex-to-component skinning weights. This yields a low-dimensional deformation space without requiring skeletons, cages, skinning weights, or rig annotations. Our architecture conditions geometry-derived deformation latents on
video features through factorized spatial-temporal
attention, then decodes rigid transformations blended by predicted skinning weights. Trained with trajectory reconstruction,
entropy regularization, and motion-aware contrastive learning, BLARM produces accurate and temporally stable animations while recovering compact, interpretable motion structure from
monocular video.
Read the full abstract ▾
All authors Pradyumn Goyal, Yizhak Ben-Shabat, Hsueh-Ti Derek Liu, Haomiao Jiang, Snehasish Mukherjee, Kyle Spence, Mark Stauber, Evangelos Kalogerakis, Yunze Zeng
Computer Vision
VeriCam: A Verification Baseline for the Classification of Unknown Data
Lucas Wojcik, Gabriel E. Lima, Sergio M. Silva Jr. et al. (+2 more)
The advent of foundation models have enabled a new era in zero-shot classification. Yet, key challenges persist. Despite their impressive generalization power that leverages the immense pre-training knowledge, both foundation models for image and text as well as
vision-text hybrids lack the representational power needed for fine-grained, minutiae-based class separation that some real-world tasks require. To address the current gaps in the literature, we propose VeriCam, a pipeline designed to learn highly specialized features that enable classification of unknown classes in unseen data. VeriCam works by leveraging the representation power of image models trained for the verification task, where the model develops an intricate feature space that incorporates fine-grained details. By training a model to discriminate between pairs of images from the same and different classes, a relational graph is constructed, representing the class relationships between data points. We then present two approaches for graph clustering: a naive
algorithm and a specific setup for the Leiden graph clustering
algorithm. The pipeline is validated on the LPLCv2
dataset, which comprises real-world traffic surveillance images. We show that the
dataset carries an inherent capture device bias that is posed as a generalization challenge for downstream License Plate recognition tasks such as OCR. As such, we dynamically identify capture devices with a label-agnostic approach, enabling the construction of a fair and unbiased
benchmark. In the cross-device scenario, our pipeline reaches an F1-Score of 93.45 in the verification baseline and a V-Measure score of 80.13 in the clustering step. All code is publicly available at https://github.com/lmlwojcik/VeriCam
Read the full abstract ▾
All authors Lucas Wojcik, Gabriel E. Lima, Sergio M. Silva Jr., Eduil Nascimento Jr., David Menotti
Computer Vision
DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution
Jiashu Zhu, Yanhao Zheng, Ruitian Tian et al. (+7 more)
Recent
video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-
video generation system centered on a 7B generator. Conditioned on a first frame and a text
prompt, the generator jointly denoises modality-specialized audio and
video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal
Attention, whose
token- and head-wise output gates modulate each active cross-modal
attention-head output. A unified Audio-
Video Data System constructs and filters temporally coherent clips, produces structured
multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-
video pre-training stages followed by High-Quality Finetuning. Audio-
Video Reinforcement Learning further post-trains the generator with Modality-Aware
Multimodal Feedback that routes
video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-
video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-
video generation and provide an accessible foundation for future research in unified audio-
video generative modeling.
Read the full abstract ▾
All authors Jiashu Zhu, Yanhao Zheng, Ruitian Tian, Rujing Dang, Shen Zhang, Bingze Song, Jiachen Lei, Ruimin Lin, Jiahong Wu, Xiangxiang Chu
Computer Vision
One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning
Yunxiang Fu, Meng Lou and Yizhou Yu
Class-incremental learning (CIL) requires a model to incrementally learn tasks that contain new classes without accessing earlier training data while preserving the ability to recognize all seen classes. Recently, pretrained-model-based approaches have become prevalent by adapting a frozen backbone with additional lightweight trainable modules. Existing methods, however, exhibit limitations: task-specific adapters learn explicit per-task representations but are parameter- and computation-inefficient, while LoRA-based merging methods combine per-task LoRA parameters into a single model whose static aggregated weights cause representation interference during inference. To address these problems, we present FACET: task-conditioned FeAture transformation with ConditionEd feature consisTency, achieving excellent parameter efficiency while producing highly discriminative features during inference. When continually trained on a task sequence, FACET learns a single shared adapter that employs a dynamic task-conditioned feature transformation, shaping the overall feature distribution of the adapter into a mixture of overlap-reduced task-specific components. On the other hand, we propose an efficient replay-free task-conditioned feature consistency loss, aiming to mitigate catastrophic forgetting of the learned mixture distribution in the adapter's feature space. Even when maintaining only a single adapter, FACET demonstrates robust scalability. On both very long task sequences (e.g., 200 tasks) and standard short task sequences (e.g., 20 tasks), our method achieves superior performance while using significantly fewer trainable parameters and GFLOPs. The code will be made open source upon acceptance.
Read the full abstract ▾
Computer Vision
Robust retinal biometrics for patient identity verification and retrieval across age and imaging devices
Jose D. Vargas-Quiros, Dennis Bontempi, Jeroen Vermeulen et al. (+3 more)
Patient identity errors can compromise longitudinal medical records, research databases, and downstream clinical decisions. We present a retinal biometric system for verifying claimed identities and retrieving the correct identity from color fundus images. We trained a 512-dimensional metric-learning encoder combining a ConvNeXtV2 backbone with ArcFace and triplet losses on 227,004 images from 21,851 patient-eye identities in the Rotterdam Study, spanning multiple imaging devices and up to 32.6 years of follow-up. The system was evaluated on held-out Rotterdam Study data and externally on the UK Biobank and Age-Related Eye Disease Study (AREDS). Before evaluation, we used the model to screen for identity inconsistencies and manually adjudicated flagged images, identifying incorrect assignments in 0.588% of Rotterdam Study images, 0.259% of UK Biobank images, and 0.164% of AREDS images. In retrospective-only verification after removing near-duplicate images, the system achieved AUROCs of 0.9998, 0.9997, and 0.9998 in the Rotterdam Study, UK Biobank, and AREDS, respectively. For identity
retrieval using only previously acquired images, Recall@1 was 99.7%, 97.2%, and 97.6%, respectively, from galleries averaging 4436-8510 identities; the correct identity appeared among the top five results in at least 98.6% of cases. Performance remained robust across imaging devices and long follow-up intervals, while lower image quality and inconsistent retinal fields accounted for most failures. These findings establish retinal anatomy as a durable biometric signal, useful for safeguarding the integrity of longitudinal imaging records.
Read the full abstract ▾
All authors Jose D. Vargas-Quiros, Dennis Bontempi, Jeroen Vermeulen, Bart Liefers, Sven Bergmann, Caroline C.W. Klaver
Computer Vision
Real-Time Video Anomaly Detection Using YOLO Pose Estimation and CLIP-Based Semantic Scoring
Vanodhya G. Warnasooriya, Amir Hajian, Watchara Ruangsang et al. (+1 more)
We propose a lightweight two-stage framework for real-time
video anomaly detection. The first stage employs YOLO v11n-pose to detect persons and extract seventeen skeletal keypoints in a single forward pass. The second stage encodes each cropped person region through CLIP ViT-B/32 and computes cosine similarity against predefined textual descriptions of anomalous behaviors. This architecture eliminates the need for optical flow, standalone pose
estimators, and density-based scoring modules. Experiments on CUHK Avenue, ShanghaiTech Campus, and a custom indoor
dataset collected at Chulalongkorn University demonstrate an end-to-end throughput of approximately 51 FPS on an NVIDIA Titan XP GPU, a 3.36x speedup over the multi-feature baseline, while maintaining frame-level AUROC values of 89.26%, 70.26%, and 84.13%, respectively.
Read the full abstract ▾
All authors Vanodhya G. Warnasooriya, Amir Hajian, Watchara Ruangsang, Supavadee Aramvith
Computer Vision
LISynSeg: Data-Centric Label-to-Image Synthesis for Cross-Modality Whole-Heart Segmentation
Jiacheng Wang, Ivana Isgum and Ipek Oguz
Whole-heart
segmentation (WHS) in computed
tomography (CT) and
magnetic resonance imaging (MRI) is affected by acquisition shifts and heterogeneous cardiac annotations. Existing WHS systems combine architectural design, transfer learning, and generic spatial or intensity augmentation. We investigate whether changes to data augmentation and training supervision can improve cross-modality WHS while the
segmentation architecture is held constant. We present LISynSeg, a data-centric approach that augments real-image nnU-Net training with label-to-image synthesis. Synthetic volumes are generated from cardiac label maps using contrast and acquisition perturbations calibrated to the training cohort, then mixed with real images to retain thoracic context absent from the labels (and thus the synthesized images). We model cardiac label variation through controlled changes in myocardial wall thickness and partial supervision of uncertain vessel endpoints. On the CARE Whole-Heart
benchmark, synthetic-only training performs worse than the real-image nnU-Net baseline, whereas calibrated real-synthetic training improves cross-modality
segmentation without changing the architecture; the improvement is larger for MRI than for CT. The results show that modifying the training data strategy can benefit model development for heterogeneous cardiac data. Code and trained weights will be released at https://github.com/MedICL-VU/Care26_LISynSeg.
Read the full abstract ▾