Computer Vision
Deep Residual Injection for Full-Spectrum Forensic Signal Perception in Multimodal Large Language Models
Kaiqing Lin, Zhiyuan Yan, Ruoxin Chen et al. (+8 more)
Multimodal large
language models (MLLMs) have been increasingly adopted in forensics for their robust semantic understanding. As AI-generated images become realistic, semantic-level inconsistencies alone are often insufficient for reliable detection. This motivates a critical question: whether MLLMs can achieve full-spectrum forensic signal perception, i.e., capturing low-level generator artifacts without sacrificing pre-trained semantic knowledge. We further perform a layer-wise analysis of forensic signal perception in MLLMs, showing that semantic information is primarily formed in the early-to-middle layers, whereas direct fine-tuning for artifact learning disrupts these semantic representations. Based on this insight, we propose Deep Visual Residual MLLM (Deep-VRM) to preserve early semantic processing while injecting artifact-specific visual signals as a residual path into an intermediate layer, where they are fused with semantic
token representations and propagated through subsequent trainable layers. This enables later layers to jointly model semantic
reasoning and signal-level forensic cues, and surprisingly, the model learns to adaptively leverage different levels of forensic signals depending on the input, achieving robust and generalizable detection performance. Extensive experiments show that our method achieves state-of-the-art across most
benchmarks. The code and data are available at https://github.com/KQL11/Deep-VRM.
Read the full abstract ▾
All authors Kaiqing Lin, Zhiyuan Yan, Ruoxin Chen, Ke-Yue Zhang, Yue Zhou, Caiyong Piao, Bin Li, Taiping Yao, Bo Wang, Youchang Xiao, Shouhong Ding
Computer Vision
Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigation
Jingyu Li, Zhe Liu, Dongnan Hu et al. (+10 more)
World action models~(WAMs) have shown great promise for autonomous driving and urban
navigation. Built upon
Vision-
Language-Action models or
video generation models, existing approaches suffer key limitations: (1) High inference latency due to future observation prediction at test time, and (2) tightly coupled
video and action modeling leading to representational mismatch and degraded generalization. To address both issues, we propose Metis, an end-to-end WAM framework that decouples
video generation and action prediction. Specifically, Metis employs a Mixture-of-
Transformers architecture with dedicated experts for
video generation and action prediction, preserving the intrinsic distributional properties of each task. To enhance efficiency, we introduce an asymmetric
attention mask that enables joint training of both experts while allowing the action model to bypass explicit
video generation during inference. This design ensures training-inference consistency and significantly reduces computational costs without compromising planning performance. Extensive experiments demonstrate state-of-the-art performance on the NAVSIM navhard and navtest
benchmarks and the CityWalker
navigation benchmark, validating both the generalizability and efficiency across diverse tasks. Real-
robot deployments further confirm the practical feasibility of our approach.
Read the full abstract ▾
All authors Jingyu Li, Zhe Liu, Dongnan Hu, Junjie Wu, Zipei Ma, Wenxiao Wu, Chao Han, Zhihui Hao, Zhikang Liu, Kun Zhan, Jiankang Deng, Xiatian Zhu, Li Zhang
Computer Vision
CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation
Long-Bao Nguyen, Quang-Khai Le, Tam V. Nguyen et al. (+2 more)
Multi-subject reference-based image generation requires jointly preserving multiple human identities, binding per-person objects and fashion items, and respecting a specified background scene, a regime where current
diffusion models remain brittle. Existing
benchmarks evaluate only one axis at a time and none jointly captures multi-identity composition with human-object interaction, background grounding, and spatial plausibility. We introduce CogCanvas, a
benchmark of 1,952 curated reference images spanning 100 celebrity identities, 115 distinctive objects and fashion items, and 29 real-world background scenes including landmarks, from which we construct 1,361 compositional
prompts covering 2-5 person group sizes. The curation pipeline combines DINOv2-based deduplication, two-stage aesthetic filtering, and automated derivation of structured interaction and position graphs that serve as ground-truth supervision. CogCanvas supports three tasks, reference-based multi-human-object generation (primary), text-to-image compositional generation, and reference
retrieval, under a unified six-axis evaluation protocol. We introduce two metrics tailored to the multi-reference setting: BG-Sim, which scores background fidelity on SAM 3-masked regions via DINOv3 feature similarity, and Attr-VQA, which uses a
multimodal LLM to verify per-subject attribute binding and inter-person interactions against the structured graphs. Benchmarking five SOTA methods reveals that every model degrades substantially as group size grows from 2 to 5, with near-complete failure on object/fashion binding beyond three subjects.
Read the full abstract ▾
All authors Long-Bao Nguyen, Quang-Khai Le, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le
Computer Vision
Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery
Yiping Li, Ronald de Jong, Romy van Jaarsveld et al. (+5 more)
Visual Question Answering (VQA) in robotic surgery, referred to as
surgical VQA, requires high-level understanding of complex
surgical scenes and the integration of visual perception with
language reasoning, with the potential to support
surgical training and intraoperative decision-making. Recent
Vision-
Language Models (VLMs) have shown promising performance through parameter-efficient fine-tuning; however, most existing approaches rely on coarse visual grounding, typically limited to bounding boxes, which fails to capture the fine-grained spatial structure of
surgical objects. In this work, we propose a unified framework that jointly performs pixel-level
segmentation and visual question answering within a single framework. Our approach integrates a VLM with a Segment Anything Model (SAM)-based decoder and represents scene elements as object
tokens generated by the VLM. These object
tokens guide answer prediction and are further projected to the SAM-based decoder to produce
segmentation masks. By optimizing the object
token embeddings through both
segmentation and question answering objectives, the model learns spatially grounded representations that enhance visual
reasoning while providing explicit pixel-level grounding. We evaluate the proposed method on the private RAMIE (
Robot-Assisted Minimally Invasive Esophagectomy)
dataset and the public EndoVis18
dataset, where it consistently outperforms baseline methods for
surgical VQA. These results demonstrate that incorporating context-aware object
tokens into
vision-
language models improves fine-grained
surgical scene understanding.
Read the full abstract ▾
All authors Yiping Li, Ronald de Jong, Romy van Jaarsveld, Franco Badaloni, Gino Kuiper, Jelle Ruurda, Josien Pluim, Marcel Breeuwer
Computer Vision
A Dual-Branch Collaborative Framework for Joint Optimization of Underwater Image Enhancement and Object Detection
Liyuan Cao, Zheng Liu, Guanghao Liao et al. (+2 more)
Due to wavelength dependent light absorption and scattering,
underwater images usually suffer from color distortion and blurred details, which limits
underwater object detection performance. Existing
underwater image enhancement methods mainly focus on visual quality improvement, while it is still difficult to balance enhancement quality, processing efficiency, and downstream detection performance. Therefore, this paper proposes an efficient dual-branch
underwater image enhancement framework for object detection. The detail enhancement branch improves brightness and local contrast to recover texture details in dark regions. The color restoration branch uses adaptive compensation to reduce color distortion and improve color gradation. By combining the complementary outputs of the two branches, the proposed framework provides clearer and more informative images for object detection. On the UIEB and EUVP
datasets, the proposed method achieves UIQM scores of 2.249 and 2.576. When applied to the YOLOv8 detection task on the URPC
dataset, the proposed method improves mAP50 by 2.1% compared with the baseline. Extensive experiments show that our method improves object detection in complex
underwater scenes, while balancing enhancement quality and processing efficiency.
Read the full abstract ▾
All authors Liyuan Cao, Zheng Liu, Guanghao Liao, Yonghui Yang, Qi Li
Computer Vision
EmoZone-Talker: Regional Semantic Control of Audio-Driven 3DGS Talking Heads via Facial Action Units
Tingting Chen, Shaojun Wang, Huaye Zhang et al. (+2 more)
3D Gaussian Splatting (3DGS) has shown strong potential for high-fidelity talking head synthesis. However, enabling fine-grained, interpretable, and editable facial expression control remains fundamentally challenging due to intrinsic conflicts between
speech-driven facial dynamics and explicit expression signals. Existing methods rely on implicit
multimodal fusion, leading to spatial
entanglement and temporal instability. We present EmoZone-Talker, a novel framework that reformulates audio-driven facial animation as a structured spatial-temporal coordination problem under cross-modal conflicts. Our approach introduces an explicit spatial disentanglement and temporal dynamics modeling of facial motion. Specifically, we propose Synergy Zones with Prioritized
Attention Bias (SZ-PAB) to explicitly decouple modality contributions via region-wise constraints guided by anatomical priors, and a Channel-Independent Temporal AU Encoder (CIT-AE) to model temporally coherent AU dynamics. By integrating these representations into 3D Gaussian deformation, EmoZone-Talker enables precise and interpretable control over facial expressions. Extensive experiments demonstrate that our method improves expression controllability and realism, with notable gains in upper-face accuracy and temporal coherence, while preserving high rendering quality and accurate lip synchronization. Code will be publicly released to facilitate reproducibility and further research.
Read the full abstract ▾
All authors Tingting Chen, Shaojun Wang, Huaye Zhang, Diqiong Jiang, Chenglizhao Chen
Computer Vision
Learning a Sampling-Free Variational DNN Plugin from Tiny Training Sets to Refine OOD Segmentation With Uncertainty Estimation
Jimut B. Pal and Suyash P. Awate
Deep neural networks (DNNs) frequently fail to generalize to out-of-distribution (OOD) medical images because of variations in scanners and acquisition protocols. Retraining DNN models to address these distribution shifts is often impractical due to the high cost of acquiring and annotating new medical
datasets. To address this, we introduce VarDeepPCA, a novel lightweight variational DNN framework designed to restore/refine degraded
segmentation maps by leveraging intrinsic
geometric priors. Unlike existing approaches that require target-domain data or extensive pre-training, our VarDeepPCA explicitly learns a distribution of valid anatomical geometries using only small in-distribution (ID)
datasets. Theoretically, our novel variational learning framework leverages a reinterpretation of the softmax mapping to implicitly perform exact distribution modeling, thereby enabling computationally efficient, sampling-free learning and inference. This also enables VarDeepPCA to provide uncertainty estimates associated with its restored
segmentation maps. We empirically validate our framework across 4 distinct clinical applications, using 14 publicly available
datasets, involving
segmentation of the myocardium, neuroretinal rim, prostate, and fetal head. Comparisons against 15 existing methods demonstrate that VarDeepPCA consistently restores
segmentation maps produced by the existing methods on OOD data to (i) significantly improve anatomical plausibility of geometries and clinical utility of the
segmentations, and (ii) significantly reduce errors, without needing any more training data than that used by existing methods.
Read the full abstract ▾
Computer Vision
SACE: Concept Erasure at the Semantic Singularity in Visual Autoregressive Models
Siya Yang, Nanxiang Jiang, Zhaoxin Fan et al. (+1 more)
The rapid progress of visual autoregressive (VAR) models has unlocked a transformative frontier for high-fidelity text-to-image synthesis, while heightening concerns over the safety
alignment of generated content. Naive application of existing erasure techniques to VAR models causes catastrophic semantic collapse and visual artifacts, since they are predominantly designed for the homogeneous denoising steps of
diffusion models. To address this foundational challenge, we first propose the Semantic Singularity Axiom, which posits that any target semantic concept embedded within a
prompt is definitively locked at Scale-0. Then rigorously validate this axiom through our proposed Incremental Semantic Saliency Analysis (ISSA),which also enable the community to transparently inspect the coarse-to-fine semantic injection process. Guided by this insight, we introduce the first scale-aware concept erasure framework (SACE) for VAR models. By strictly confining interventions to the first scale, our approach couples an
Entropy-Regularized Erasure Objective to prevent high-
entropy sampling degeneration, alongside a restorative preservation loss to safely anchor the integrity of entangled benign priors. Extensive experiments demonstrate that our method achieves
surgical concept erasure performance across various domains with minimal training overhead, timely and elegently resolute the critical safety vulnerabilities inherent in emerging VAR architectures. Code is available at: https://github.com/limerenceysy/SACE}{https://github.com/limerenceysy/SACE.
Read the full abstract ▾
All authors Siya Yang, Nanxiang Jiang, Zhaoxin Fan, Yunfeng Diao
Computer Vision
CPS4: Class Prompt driven Semi-Supervised Spine Segmentation with Class-specific Consistency Constraint
Qingtao Pan, Hongzan Sun, Bing Ji et al. (+1 more)
Vision Language Model (VLM) has great potential to enhance the quality of pseudo labels in semi-supervised spine
segmentation by leveraging textual class
prompts to generate
segmentation map, but no one has studied it yet. Although promising, it lacks explicit constraints to ensure consistency between spine class
prompts and spine unit region, resulting in unsatisfactory performance in multi-class
segmentation map generation. In this paper, we propose CPS4, the first text-guided semi-supervised spine
segmentation network using class
prompts to enhance the quality of spine pseudo labels. Specifically, CPS4 is implemented through two training stages. (i) Class-specific consistency constrained VLM pretraining stage: we propose
token- and pixel-level
attention loss to optimize the consistency between class
prompts and spine units, forcing the textual class
prompt to be closely coupled with the target spine unit in the semantic space. (ii) Class
Prompt driven semi-supervised spine
segmentation stage: using the pretrained
vision-text encoder, we derive each class-specific binary
segmentation map for the unlabeled spine image and integrate them into an unified multi-class
segmentation map, improving the quality of the spine pseudo label generated by the semi-supervised spine
segmentation network. Experimental results show that our CPS4 achieves superior spine
segmentation performance with Dice of 80.44%, only using 5% labeled data on the public spine
segmentation dataset, surpassing popular semi-supervised learning and VLM methods. Our code will be available.
Read the full abstract ▾
All authors Qingtao Pan, Hongzan Sun, Bing Ji, Shuo Li
Computer Vision
DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing
Artyom Mazur, Nina Konovalova and Aibek Alanov
While transcoder-based
circuit tracing has recently enabled detailed causal analyses of large
language models, it remains unclear how semantic information propagates across denoising steps and how text and image representations interact within
multimodal diffusion transformers (MM-DiT) for image generation.
Attention maps expose only a limited view of
token interactions, and sparse autoencoders find interpretable features but do not reveal how nonlinear MLP layers transform and compose them. In this work, we extend transcoder-based
circuit tracing to
multimodal diffusion transformers. We train timestep-conditioned transcoders that faithfully approximate the input-output behavior of the MLP sublayers in the double-stream blocks of FLUX.1[schnell]. By replacing MLPs with transcoders and linearizing the remaining computation, we obtain exact feature-to-feature attribution and build compact attribution graphs over interpretable features. Empirically, our transcoders match or slightly outperform sparse autoencoders on the sparsity-faithfulness tradeoff. The graphs recover the known shift from
prompt-driven to image-driven computation across denoising steps, point to candidate mechanisms of attribute binding and cross-stream semantic propagation, and yield intervention-testable explanations for systematic generation errors. On
prompt grids labeled blind, graph-guided interventions reach effects that single-feature steering does not; at a matched budget, features chosen from the graph fix a color prior more often than the target feature alone and destroy the object less often than features chosen by co-activation. FLUX.1[dev] reproduces the temporal and color-prior results, and SD3.5-medium reproduces them in direction. Our results demonstrate that transcoder-based
circuit analysis is feasible for state-of-the-art
multimodal diffusion transformers.
Read the full abstract ▾
Computer Vision
Domain-Guided Prompting of the Segment Anything Model for Seismic Interpretation: The Role of Attributes, Visualization, and Hybrid Prompts
Aniq Ahmad, Heather Bedle and Ahmad Mustafa
The advent of large pretrained foundation models for computer
vision has significantly improved the efficiency of visual data interpretation. The Segment Anything Model (SAM), in particular, offers powerful zero shot
segmentation capabilities through
prompt based interaction, thus making it a promising tool for seismic interpretation. However, most existing applications of SAM rely on fine tuning for specific geological targets, which requires extensive labeled data, incurs high computational cost, and often compromises the model's generalization capability. In this study, we introduce a principled framework for zero shot adaptation of foundation models to seismic data. The framework is built on two key components: (1) aligning seismic attributes and visualization choices (e.g., colormaps) with the geological target of interest, and (2) employing a hybrid prompting strategy that combines sparse user defined point
prompts with dense mask
prompts derived from SAM's internal feature activations. We systematically evaluate this framework across multiple geological targets,
datasets,
prompt configurations, and seismic attribute representations. Our results demonstrate that geologic target aware selection of seismic attributes and colormaps, combined with hybrid prompting, enhances the separability of geological features and improves boundary delineation and
segmentation accuracy relative to point based prompting alone. Our findings show that, when these components are jointly applied, SAM can achieve competitive
segmentation performance in a fully zero shot setting, thereby eliminating the need to retrain SAM for each geologic feature. This work establishes a practical and scalable pathway to leverage foundation models in seismic interpretation, reducing reliance on labeled data while preserving model generality.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
Mitigating Visual Hallucinations in Multimodal Systems through Retrieval-Augmented Reliability-Aware Inference
Pratheswaran Hariharan, Haiping Xu and Donghui Yan
Multimodal large
language models (MLLMs) have demonstrated strong capabilities in
vision-
language understanding and natural-
language response generation. However, these systems can still produce overconfident predictions and
hallucination-like outputs, particularly when the visual evidence is weak, ambiguous, or semantically inconsistent. Most existing approaches focus on improving
multimodal representation
alignment or
retrieval-augmented generation, while providing limited mechanisms to quantify instance-level prediction reliability or identify incorrect visual outputs. This work proposes a
retrieval-augmented reliability-aware inference framework for trustworthy
multimodal visual understanding. The proposed framework constructs an external visual evidence database using pretrained visual
embeddings and nearest-neighbor
retrieval over normalized feature representations. Retrieved evidence is used to estimate prediction trustworthiness through multiple reliability indicators, including similarity strength, class-support agreement, evidence margin,
entropy-based uncertainty, and an aggregate reliability score. Based on these signals, a decision gate determines whether the system should accept the prediction, answer with caution, or abstain/fallback when evidence is insufficient. A
multimodal response-generation layer then produces a final user-facing response conditioned on the reliability decision. Experiments on ImageNet-100 demonstrate that the proposed reliability-aware framework improves accepted prediction accuracy from 85.84% to 88.88% at 89.04% coverage. The
hallucination-like accepted wrong-answer rate is reduced from 14.16% to 11.12%. These results show that integrating
retrieval evidence, reliability estimation, and selective decision gating can improve calibration and reduce overconfident visual errors without retraining large
multimodal models.
Read the full abstract ▾
Computer Vision
Faithful Action-unit Causal Reasoning for Counterfactually Faithful Emotion Explanations
Van Thong Huynh, Hong Hai Nguyen, Thuy Pham et al. (+2 more)
Multimodal models can name the action units (AUs) behind a facial
emotion, but their AU->
emotion rationales are typically plausible rather than faithful: nothing forces the AUs a model invokes to be the AUs that actually drive its prediction. We cast AU->
emotion reasoning as a counterfactual-consistency problem between the rationale, the label, and a structural AU->
emotion causal graph G, and propose FACR, which grounds the reasoner in an independently induced, polarity-aware G and trains a counterfactual-faithfulness objective: a do-intervention on an AU that G marks causal for a class must move the prediction, while one it marks irrelevant must leave it unchanged. Faithfulness is thereby both trainable and measurable through a matching interventional metric, which we evaluate against a known causal structure, the PSPI pain-AU composition, as no existing affective-
reasoning benchmark allows. We are explicit that this metric tests fidelity to the supplied structure rather than its rediscovery: it asks whether the trained reasoner invokes the AUs the structure marks causal, on held-out subjects and a second
dataset. Under subject-independent evaluation on UNBC-PAIN, the objective raises the agreement between the invoked AUs and the PSPI composition from a no-objective baseline of 0.08 to 0.57, at a small detection cost; an unfaithfulness control attributes the gain to the objective. On a cross-
dataset emotion transfer, the objective likewise raises fidelity to G on a seven-class task (0.50 to 0.84). Finally, we attach a
language verbalizer and extend the audit to the generated text: biasing each action unit's emission by its latent activation makes the rationale faithful by construction, so that ablating an AU removes it from the explanation, a property that transfers to a second
language-model backbone, whereas a freely generated rationale is unfaithful.
Read the full abstract ▾
All authors Van Thong Huynh, Hong Hai Nguyen, Thuy Pham, Trong Nghia Nguyen, Soo-Hyung Kim
Computer Vision
Ellipse Meets Bit-Planes: A Novel Approach to RNFL based Glaucoma Detection Using Advanced Image Processing and Deep Learning
Snigdha Paul, Sambit Mallick and Anindya Sen
This work proposes an integrated pipeline for automatic glaucoma detection method from easily available colour fundas images based on an adaptive
algorithm for ellipse-based polar transformation, to enhance the analysis of the Retinal Nerve Fiber Layer (RNFL) as the primary biomarker for observing glaucomatous changes, regardless of optic disc and macula position. Utilizing this transformation, we introduce two distinct frameworks tailored to different operational needs. The first framework, a deep learning-inspired feature fusion approach, achieves a 99.3% detection rate, ideal for settings where high precision is essential, despite higher computational demands. The second framework employs a novel image-processing
algorithm based on bit-plane slicing, offering 92.31% accuracy and optimized for environments requiring rapid inference with minimal resource consumption. Both frameworks provide scalable and cost-effective solutions for early glaucoma detection. This study highlights the potential of RNFL-based diagnostic tools in addressing the global challenge of glaucoma, particularly in underserved regions.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Jialei Chen, Kai Wang, Kang Chen et al. (+9 more)
Vision-
Language-Action models (VLAs) leverage large-scale
vision-
language pretraining for semantic
robot control, but often lack explicit foresight into how
robot actions change the scene. World-Action Models (WAMs) address this limitation by conditioning policies on predicted futures, yet existing approaches typically rely on computationally expensive
video generation with substantial pixel-level redundancy. We present LaWAM, a Latent World Action Model that exposes predictive dynamics to
robot policies through compact latent visual subgoals instead of reconstructed future
video. At the core of LaWAM is a latent-action-conditioned Latent World Model (LaWM). We obtain LaWM by training a latent action model in the latent space of a pretrained
vision foundation model and repurposing its forward decoder to predict future observation features for scene
evolution. LaWAM then conditions action generation on these predicted latent visual subgoals to enable dynamics-aware
robot control. LaWAM achieves state-of-the-art or competitive success rates (SRs) across LIBERO (98.6% SR), RoboTwin (91.22% SR), and real-world manipulation tasks while retaining low-latency inference. LaWAM runs in 187 ms per action-chunk prediction and achieves up to 24x lower wall-clock latency than pixel-space WAMs.
Read the full abstract ▾
All authors Jialei Chen, Kai Wang, Kang Chen, Shuaihang Chen, Feng Gao, Wenhao Tang, Zhiyuan Li, Weilin Liu, Zhuyu Yao, Boxun Li, Yuanbo Xu, Chao Yu
Computer Vision
Learning to Orchestrate Vision Foundation Models for Multi-Task Dense Prediction
Donghyun Han, Yuseok Bae, Jung Uk Kim et al. (+1 more)
Vision foundation models (VFMs) exhibit complementary strengths shaped by their pretraining objectives. Yet prevailing methods for multi-task dense prediction still train an entire backbone, either by fine-tuning it under multi-task supervision or by distilling multiple VFMs in an additional stage. We ask whether downstream learning can instead compose the frozen representations already available in foundation models. Dense tasks require composite representations that no individual expert provides alone. Realizing them is difficult: simple fusion yields only marginal gains over the best single expert, while learned routing tends to collapse toward candidates that are strong at initialization, starving newly initialized composers of training signal. We present COVE, which constructs pairwise composite candidates through Synergy Composers and routes among raw and composite candidates with a Task-Conditioned Router. To prevent this collapse, COVE combines Gaussian logit perturbation for exploration with counterfactual supervision that selectively increases under-credited routing allocations. On NYUD-v2 and PASCAL-Context, COVE matches or surpasses ViT-L-based methods on most tasks using a smaller frozen encoder pool and roughly half the computation of recent VFM-based competitors, while exceeding the best single frozen expert on every task.
Read the full abstract ▾
All authors Donghyun Han, Yuseok Bae, Jung Uk Kim, Hyung-Il Kim
Computer Vision
The Circumplex Degeneracy Behind the Rare-Class Limit in Affect Recognition
Van Thong Huynh, Hong Hai Nguyen and Soo-Hyung Kim
In-the-wild expression recognition persistently fails on a few rare
emotions, and the standard explanation is class imbalance. Through a controlled multi-task study on two
benchmarks, we show the failure is instead a property of affect geometry: the rare classes are degenerate on Russell's circumplex, and that degeneracy bounds what any loss or cost can achieve. Our instrument is a circumplex-cost optimal-transport term that prices expression confusions by their valence-arousal distance. The term improves the official score and expression macro-F1, but a control most studies omit shows the gain is not
geometric: a uniform cost, equivalent to a generic confidence penalty, matches it on Aff-Wild2 (p=0.625) and significantly exceeds it on AffectNet (+0.057 over base, larger than the circumplex). What the geometry reshapes is the structure of the errors, making them affectively nearer the truth on Aff-Wild2 (p=0.031 against the uniform control), an effect that does not survive on AffectNet, where a visual confound at the far corner of the circumplex overwhelms it. The rare-class failure, by contrast, is stable across both
datasets we examine: the degenerate pairs (anger-fear on Aff-Wild2, anger-contempt on AffectNet) resist frequency-based interventions, the transport term, and an action-unit-augmented cost built specifically to separate them. We conclude that progress on rare expressions requires representations that distinguish the classes, not supervision that reprices their confusions, and we provide the controls and metrics needed to tell the two apart.
Read the full abstract ▾
Computer Vision
OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning
Maonan Wang, Zhengyan Huang, Kemou Jiang et al. (+13 more)
Traffic scene understanding requires models to reason beyond object recognition, including lane
topology, multi-view geometry, temporal
evolution, and signal-phase semantics. However, existing traffic-oriented
multimodal benchmarks largely emphasize passive visual recognition or isolated
video understanding, offering limited support for evaluating structure-aware traffic
reasoning under controlled conditions. We introduce OmniTraffic, a controllable generation pipeline and
benchmark for spatio-temporal traffic
reasoning. Built around 12 real-world intersections reconstructed into editable 3D traffic environments and complemented by surveillance footage from two countries, OmniTraffic supports both controlled and natural-condition evaluation. It defines a three-level task hierarchy spanning scene perception, multi-view and temporal
reasoning, and decision support. Using structured traffic metadata, OmniTraffic generates synchronized multi-view VQA samples covering vehicle states, lane functions, view–BEV correspondence, temporal dynamics, and signal-phase analysis, resulting in 8M VQA samples and a 3K human-verified test set. Evaluation of eleven frontier MLLMs reveals a large human–model gap, with the most pronounced failures in
topology-grounded and spatio-temporal
reasoning tasks. Fine-tuning a lightweight MLLM on simulated OmniTraffic data further improves performance on real-world traffic scenes, demonstrating the value of simulation-generated supervision for traffic-specific
multimodal reasoning. Beyond a fixed
dataset, OmniTraffic provides an extensible pipeline with configurable intersections, camera views, traffic demands, signal phases, visual conditions, and rare events.
Read the full abstract ▾
All authors Maonan Wang, Zhengyan Huang, Kemou Jiang, Yuhang Fu, Jiayue Zhu, Yuxin Cai, Xingchen Zou, Qiaosheng Zhang, Yi Yu, Ding Wang, Xi Chen, Ben M. Chen, Yuxuan Liang, Zhiyong Cui, Man On Pun, Yirong Chen
Computer Vision · Cross-listed from cs.MM
MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs
Hangling Xie
Multimodal large
language models (MLLMs) have demonstrated remarkable capabilities in understanding complex
multimodal content. However, their performance in sentiment analysis exhibits acute sensitivity to
prompt design, rendering static, uniformly applied
prompts inherently suboptimal for capturing the nuanced
multimodal cues that vary across inputs. To address this limitation, we propose a
Multimodal Adaptive Few-Shot Prompting (MAF) framework, which dynamically retrieves and integrates query-relevant demonstrations to elicit the sentiment
reasoning capabilities of MLLMs in a context-sensitive manner. MAF constructs a demonstration
retrieval module that holistically encodes facial expressions, scene context, and textual semantics, with a lip movement amplitude detection mechanism introduced for accurate speaker identification in multi-person scenarios. Departing from conventional fixed-weight fusion, a lightweight coefficient generation network is trained to output query-conditioned fusion weights in real time, enabling weighted aggregation of
multimodal similarity scores to retrieve the top-K most informative demonstrations. Prediction stability is further enhanced through majority voting over multiple candidate outputs generated by the MLLM. Extensive experiments on public
benchmark datasets demonstrate that MAF achieves substantial and consistent performance improvements over the corresponding backbone variants and remains competitive with strong
multimodal sentiment-analysis baselines.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Learning New Tasks via Reusable Skills: Skill-Compositional Experts for Embodied Continual Learning
Shuaike Zhang, Shaokun Wang, Haoyu Tang et al. (+2 more)
Embodied Continual Learning (ECL) aims to enable
robots to continually acquire new manipulation tasks while retaining previously learned behaviors under closed-loop control. In ECL, feature drift can propagate through sequential decision-making under closed-loop control, turning representation changes into compounding behavioral deviations on previously learned tasks. A key challenge in ECL lies in structured skill reuse across continually evolving tasks, since existing methods primarily focus on skill learning without explicitly organizing them for coherent task execution. To address this issue, we propose SCE, a Skill-Compositional Experts framework for ECL. SCE builds a skill base via Compositional Skill Grounding (CSG), which decomposes task demonstrations into reusable skills. Based on this, Dual Execution-and-Transition Experts (DETE) enable new task learning through skill composition, where one branch ensures skill execution and the other supports transitions between skills for coherent behavior. Experiments on LIBERO
benchmarks and real-world manipulation tasks show that SCE improves retention and overall task performance. Further feature drift analyses and ablation studies verify the effectiveness of our method. Project website: https://eqcy.github.io/sce/.
Read the full abstract ▾
All authors Shuaike Zhang, Shaokun Wang, Haoyu Tang, Jianlong Wu, Liqiang Nie
Computer Vision
3D Consistency Optimization for Self-Supervised Monocular Video Depth Estimation
Yuanye Liu, Ke Zhang, Junzhe Jiang et al. (+3 more)
Reliable
monocular video depth estimation is crucial for downstream 3D
reasoning and
embodied AI in endoscopic
navigation. However, existing self-supervised approaches typically treat
video frames independently or rely on weak temporal regularization. These methods, lacking a holistic perception of the underlying 3D scene, inevitably suffer from geometrically inconsistent predictions and severe cross-frame drift. To address these limitations, we introduce a new paradigm that recasts sequential
video depth estimation as an unconstrained multi-view 3D reconstruction problem, enabling full exploitation of the powerful
geometric priors embedded in recent 3D foundation models. The core of our approach is a 3D consistency optimization framework driven by three constraints: image-level photometric rendering, explicit world-coordinate
geometric alignment, and multi-scale temporal gradient consistency. Such unified optimization elegantly anchors isolated frames to a globally coherent 3D structure. Our method has been validated in both the self-supervised training scenarios and challenging zero-shot clinical environments. Results show that the proposed approach achieves state-of-the-art spatial accuracy, outperforming the frame-based,
video-based depth
estimators and the multi-view 3D reconstruction baselines.
Read the full abstract ▾
All authors Yuanye Liu, Ke Zhang, Junzhe Jiang, Li Zhang, Vishal Patel, Xiahai Zhuang
Computer Vision
CEVAR: Centerline Embedding Extraction for Endovascular Aneurysm Repair
Roman Naeem, Timo Niiniskorpi, Charlotte Sandström et al. (+6 more)
Long-term mortality rates after endovascular aneurysm repair (EVAR) remain elevated due to post-EVAR rupture caused by loss of seal in stent graft sealing zones. Structured CT review using centerline measurements improves detection, but current workflows require manual centerline editing and expert operators. We propose a
transformer framework for automated, protocol-driven sealing zone assessment that combines 3D centerline tracking with
embedding-based
geometric prediction. Two state-of-the-art image-to-graph models are evaluated for aorto-iliac centerline extraction from follow-up CT and for measurement of stent position, vessel diameters, and seal lengths according to EVAR4C protocol. Across the full test set and a challenging no-contrast subset, the proposed fully automatic method outperforms the commercial semi-automatic workflow.
Read the full abstract ▾
All authors Roman Naeem, Timo Niiniskorpi, Charlotte Sandström, Naman Desai, Anders Jeppsson, Ida Häggström, Fredrik Kahl, Håkan Roos, Jennifer Alvén
Computer Vision
OneFocus: Enabling Real-World X-ray Security Screening with a Unified Vision-Language Model
Jiali Wen, Hongxia Gao, Litao Li et al. (+4 more)
X-ray contraband detection is critical for security in large-scale logistics and transportation, yet conventional detectors struggle to adapt to emerging contraband types and lack fundamental visual understanding.
Vision-
language models (VLMs) offer strong generalization but are hindered by the scarcity of high-quality X-ray image-caption data. To bridge this critical gap, we present MMXray, a meticulously curated
benchmark of 52,124 image-caption pairs spanning 28 fine-grained classes of X-ray contraband. To enrich MMXray with realistic occlusion patterns, we further introduce CleanDET, a dedicated synthesis
dataset containing clean foreground contraband images from 28 categories and background images with diverse density levels, together with AnyContraSyn, a controllable synthesis method designed to operate on CleanDET. We also develop OnePipe, an extensible pipeline for systematic data curation. Built on MMXray, we propose OneFocus, a unified VLM that supports four core tasks: visual question answering, contraband localization, classification, and image understanding. OneFocus achieves state-of-the-art performance in X-ray contraband understanding and demonstrates robust cross-domain generalization, establishing a strong
vision-
language baseline for security screening.
Read the full abstract ▾
All authors Jiali Wen, Hongxia Gao, Litao Li, Yixin Chen, Kaijie Zhang, Qianyun Liu, Xiaoqin Wen
Computer Vision
SpatialAvatar-0: High-Quality 4D Head Avatar with Multi-Stage Reconstruction
Yiran Wang, Zeyu Zhang, Yuanming Li et al. (+2 more)
High-quality 4D head avatars from one or a few source portraits are central to telepresence, AR/VR, and digital-human interaction. 3D Gaussian Splatting (3DGS) has emerged as the dominant representation, with two complementary regimes (generalizable feed-forward predictors and per-subject refiners) maturing in parallel. However, existing feed-forward predictors are trained on a single
dataset family with a hard-coded source count, inheriting the corresponding domain bias. Per-subject refiners require 300K–600K iterations and rely on adaptive densification that destroys upstream Gaussian layouts, preventing the two regimes from sharing a representation end-to-end. To bridge both regimes we propose SpatialAvatar-0 on a shared FLAME-mesh-bound Gaussian representation: a feed-forward generator with a parameter-free K-source mean-pool and a
monocular-temporal to multi-view-spatial two-phase schedule that anchors against identity-prior collapse onto the smaller multi-view set. We further introduce a 10K-iter layout-preserving per-subject refinement loop that freezes the FLAME-binding and Gaussian count and replaces densification with a three-component anti-spike regularization. On VFHQ/HDTF cross-domain zero-shot we surpass the in-domain leader GAGAvatar by +1.5 dB PSNR despite never training on either test domain, and on the SplattingAvatar
monocular benchmark we lead every reported metric, surpassing the 300K-iter GeoAvatar by +1.3 dB PSNR at up to 60x shorter per-subject schedule than common SOTA baselines. Website: https://spatialwalk.github.io/SpatialAvatar-0.
Read the full abstract ▾
All authors Yiran Wang, Zeyu Zhang, Yuanming Li, Ziming Wang, Yang Zhao
Computer Vision
Self-Questioning Vision-Language Models: Reinforcement Learning for Compositional Visual Reasoning
Saraswathy Amjith
Vision-
Language Models (VLMs) are AI systems that process both images and text, yet they often struggle with compositional visual
reasoning questions that require chaining multiple steps together, such as identifying objects, counting them, and comparing the results. Existing approaches improve this
reasoning by training models on human-written step-by-step explanations, but creating these annotations is expensive and difficult to scale. We propose a self-questioning framework that trains a VLM to break visual questions into smaller sub-questions and answer each one before producing a final response, using a reinforcement learning
algorithm called Group Relative Policy Optimization (GRPO). The model is never shown examples of how to decompose questions, it discovers this behavior on its own, guided by a reward signal that scores whether the output contains sub-questions and whether the final answer is correct. We apply this framework to a 3-billion-parameter model, training on both synthetic scenes of
geometric shapes (CLEVR) and real-world photographs (A-OKVQA). On A-OKVQA, both self-questioning and standard reinforcement learning substantially improve accuracy over the untrained model (52.2% and 51.6% vs. 46.8%). We introduce the first self-questioning VLM by rewarding not only the final answer like standard RL but additionally for generating intermediate sub-questions, enabling it to discover compositional decomposition strategies. These results suggest that teaching AI systems to ask themselves intermediate questions is a promising strategy for complex visual
reasoning, particularly when the difficulty of a question warrants explicit step-by-step decomposition.
Read the full abstract ▾
Computer Vision
Fusing Transferred Priors and Physics-based Decomposition for Underwater Image Enhancement
Haochen Hu, Yanrui Bin, Zhengyan Zhang et al. (+3 more)
The
underwater images are captured within diverse water-medium conditions, leading to complex degradation, including color bias, low contrast, and blur effect. Recently, learning-based methods have demonstrated their potential for
underwater image enhancement (UIE). However, most of the previous work focus on the training strategy or network design to make the enhanced result aligned well with the labels in
datasets, ignoring that the labels are selected from the enhanced results of previous UIE methods and these pseudo-labels are noisy. Consequently, the performance of their models is not satisfactory to a certain extent. However, collecting the true labels of the
underwater images is challenging. In this work, we propose a transfer learning-based UIE that does not require
underwater images to have paired noisy or true labels for learning. Instead, the UIE task is first divided into global color correction, haze removal, and background noise suppression following the
underwater physics. Then multiple types of prior from other
vision tasks are leveraged as cross-domain supervision in each step. In this way, a novel UIE is available via transfer learning, and the physics-aligned UIE decomposition provides theoretical soundness. Qualitative and quantitative experiments demonstrate that our proposal based on physics and priors fusion achieves SOTA performance in the UIE task and effectively boosts downstream
vision tasks, significantly outperforming
benchmark methods. Project repo: https://github.com/Haru2022/P2-UIE.
Read the full abstract ▾
All authors Haochen Hu, Yanrui Bin, Zhengyan Zhang, Minchen Wei, Chih-yung Wen, Bing Wang
Computer Vision
Toward Open-World Video Segmentation over Long Horizons
Qing Su, Kaiyang Li, Yuan Zhuang et al. (+2 more)
Long
videos challenge open-world
segmentation systems to continually discover objects and preserve their identities through disappearance and reappearance. Savvy, a zero-shot, semi-online, class-agnostic system for persistent object discovery and identity maintenance, and OGA, an evaluation suite that credits coherent part-level predictions when their granularity differs from the reference annotations. Savvy combines modular mask discovery, deferred admission based on accumulated evidence, and track consolidation to maintain an evolving object set. OGA allows multiple predictions to support one reference while preserving prediction IDs, pairing granularity-aware fidelity with diagnostics of identity bleeding and fragmented support. Across 142 ScanNet
videos and 106 HM3D trajectories, evaluated using adaptively sampled prefixes capped at 1,500 source frames, Savvy outperforms DEVA+SAM and EntitySAM in VPQ_inf, STQ, and AQ under both conventional and OGA evaluation. Conventional VPQ_inf reaches 18.44 versus DEVA+SAM's 11.52 on ScanNet and 17.31 versus 6.58 on HM3D. VIPSeg comparisons reveal conventional one-to-one VPQ's sensitivity to annotation granularity. Controlled identity-severing and flicker tests show that OGA detects temporal failures even when frame-level masks remain unchanged. Together, these results demonstrate complementary advances: Savvy improves long-horizon
segmentation and association, and OGA distinguishes coherent part-level support from temporal identity failures. Project website: https://github.com/QingSuML/savvy
Read the full abstract ▾
All authors Qing Su, Kaiyang Li, Yuan Zhuang, Fei Miao, Shihao Ji
Computer Vision
XPASS-Vis: A Dataset for Cross-Domain Personalized Image Aesthetic Assessment
Takato Hayashi, Hiroaki Takahara, Candy Olivia Mawalim et al. (+4 more)
Personalized image aesthetic assessment (PIAA) seeks to model, at the individual level, the subjective nature of aesthetic judgments toward artworks and photographs. Aesthetic preference is known to be both deeply personal and partially consistent across visual domains. Yet existing PIAA
datasets and methods are largely confined to a single domain, or provide too few samples per annotator within each domain to enable personalization across domains. Consequently, the cross-domain generalization of personalized aesthetic preferences remains largely unexplored. To address this gap, we introduce XPASS-Vis, the first
dataset explicitly designed for cross-domain PIAA. XPASS-Vis comprises 6,526 stimuli from three visual domains – art, fashion, and landscape – rated by 129 annotators, yielding 87,836 user-stimulus interactions, each annotated with an overall aesthetic score and nine aesthetic-
emotion ratings. Notably, each annotator rated more than 200 stimuli per domain, providing sufficient per-domain coverage to support personalization both within and across domains. Moreover, we establish baseline models for cross-domain PIAA under unsupervised domain adaptation (UDA), where a model trained on a labeled source domain is transferred to an unlabeled target domain. A systematic evaluation of representative UDA approaches shows that the best-performing method recovers approximately 60% (Spearman's $\rho$ = .28) of the supervised upper bound under a fully unsupervised setting. This provides encouraging evidence that personalized aesthetic preferences are, to a meaningful extent, transferable across visual domains. At the same time, a substantial gap remains, highlighting the need for PIAA-specific adaptation strategies. XPASS-Vis and the accompanying baselines provide a foundation for future research on cross-domain PIAA. All
datasets and code will be made publicly available upon acceptance.
Read the full abstract ▾
All authors Takato Hayashi, Hiroaki Takahara, Candy Olivia Mawalim, Hiromi Narimatsu, Akisato Kimura, Shiro Kumano, Shogo Okada
Computer Vision
NeRD: Neuro-Symbolic Rule Distillation for Efficient Ontology-Grounded Chain-of-Thought in Medical Image Diagnosis
Hongxi Yang, Yiwen Jiang, Siyuan Yan et al. (+8 more)
Interpretability is essential for trustworthy medical image diagnosis. However, existing concept-driven interpretable methods have key limitations: Concept Bottleneck Models (CBMs) require scoring all predefined concepts at inference time and for manual intervention, imposing a substantial burden on clinicians, while rationale-based generative approaches often select concepts by class discriminability, which can drift from diagnostic ontologies. To address these issues, we propose Neuro-Symbolic Rule
Distillation (NeRD), a framework that produces efficient, ontology-grounded
reasoning chains that are sufficient yet non-redundant, without manually crafting diagnostic rules. Experiments on two skin
datasets demonstrate strong diagnostic performance and interpretability, and blinded expert evaluation confirms the clinical plausibility of NeRD rationales. Our method further enables a first expert-in-the-loop study for
Multimodal Chain-of-Thought-based diagnosis, achieving efficient and effective concept-level intervention.
Read the full abstract ▾
All authors Hongxi Yang, Yiwen Jiang, Siyuan Yan, Jamie Chow, Eunis Li, Charlotte Poon, Stephanie Fong, Xiangyu Zhao, Deval Mehta, Yasmeen George, Zongyuan Ge
Computer Vision · Cross-listed from cs.LG
MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers
Maoliang Li, Haojing Chen, Jiayu Chen et al. (+4 more)
Diffusion Transformers with Mixture-of-Experts (DiT-MoE) improve model capacity under sparse activation, but
diffusion inference is still bottlenecked by redundant computation across timesteps. Existing caching methods mainly operate at the
token level, which becomes suboptimal in DiT-MoE because each
token update is internally decomposed into multiple routed expert branches. Our analysis shows that cross-timestep redundancy in DiT-MoE is better characterized at the expert-branch level than at the whole-
token level. Based on this observation, we propose MoECa, a fine-grained caching framework that performs branch-level feature reuse across timesteps. MoECa further introduces expert-aware adaptive control and synchronized cache updates across MoE and
attention paths to maintain stable intermediate states. Experiments on multiple DiT-MoE models show a favorable speed–quality trade-off, with up to 2.93$\times$ speedups while preserving generation quality.
Read the full abstract ▾
All authors Maoliang Li, Haojing Chen, Jiayu Chen, Zihao Zheng, Xinhao Sun, Hailong Zou, Xiang Chen