Computer Vision
Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression
Tianhao Chen, Yuheng Wu, Kelu Yao et al. (+3 more)
Multimodal Large
Language Models (MLLMs) achieve strong
vision-
language reasoning but incur large KV caches and high decoding latency with long visual contexts. Existing compression methods rely on observation window
attention for stable
token importance estimation, yet this aggregation can dilute sparse critical evidence and discard answer-relevant
tokens under aggressive compression. We identify last query
attention as a complementary signal for recovering such evidence, though its irrelevant signals may introduce additional noise. We propose BACON, a plug-and-play method that calibrates observation window
attention with last query evidence while suppressing noise through intra-layer coherence and inter-layer persistence. Across diverse
benchmarks, models, budgets, and compression methods, BACON improves
multimodal KV-cache compression by 7.5% on average under the most aggressive budget, with gains up to 30.9%.
Read the full abstract ▾
All authors Tianhao Chen, Yuheng Wu, Kelu Yao, Xiaogang Xu, Xiaobin Hu, Dongman Lee
Computer Vision
Variational Deep Unfolding with Mamba-Based Nonlocal Modeling for Underwater Image Enhancement
Daniel Torres, Julia Navarro, Catalina Sbert et al. (+1 more)
Underwater imaging plays a crucial role in ocean engineering, although captured data often suffer from poor visibility and color distortion. To address these challenges, we propose a model-based deep unfolding network for
underwater image enhancement that integrates variational modeling into a learnable architecture. The framework is guided by a variational formulation based on a dehazing decomposition, incorporating a multiplicative residual component to absorb remaining artifacts and a nonlocal gradient-type constraint to preserve structural details and enhance edge sharpness. We provide a theoretical analysis establishing the existence of solution for the associated minimization problem. The proposed unfolding method incorporates Mamba layers to efficiently capture self-similarities in the scene. In addition, we introduce a proximal trajectory loss that enforces consistency between the unfolding stages and the iterations of an ideal restoration regularizer. Experimental results demonstrate that the proposed unfolding approach achieves improved visual quality and competitive quantitative performance compared with recent state-of-the-art methods. The source code will be available at https://github.com/MIA-UIB/Variational-Unfolding-Mamba-
Underwater-Enhancement .
Read the full abstract ▾
All authors Daniel Torres, Julia Navarro, Catalina Sbert, Joan Duran
Computer Vision
YTClickbait21K: Human-Annotated Multimodal Dataset for YouTube Clickbait Detection Across Diverse Channels and Content Categories
Md. Minhazul Islam, Md. Tanbeer Jubaer, Amith Khandakar et al. (+5 more)
Clickbait content on
video-sharing platforms poses a significant challenge to information reliability, yet progress in automated detection has been constrained by the lack of large-scale, high-quality
multimodal datasets. We present YTClickbait21K, a human-annotated YouTube clickbait
dataset comprising 21,238
videos collected from 40 channels across 29 countries, covering diverse content categories such as news, entertainment, education, and gaming. Each sample includes structured metadata (title, description, engagement statistics) along with associated thumbnail images, enabling comprehensive
multimodal analysis. To ensure annotation quality, every
video was independently labeled by three annotators using a standardized decision framework that incorporates textual, visual, and cross-modal consistency cues, with final labels determined through majority voting. The
dataset exhibits substantial inter-annotator agreement (k=0.65), confirming reliable labeling despite the inherent subjectivity of clickbait detection. By combining scale, annotation rigor, and
multimodal richness, this
dataset provides a robust
benchmark for developing and evaluating machine learning models, facilitating research in cross-modal semantic understanding, and advancing automated content moderation systems.
Read the full abstract ▾
All authors Md. Minhazul Islam, Md. Tanbeer Jubaer, Amith Khandakar, Shovon Sarker, Sumaiya Rahman, Md. Masum Mia, Mohamed Arselene Ayari, Hamed Noori
Computer Vision
FactCheck: Feasibility-aware Long-term Action Anticipation with Multi-agent Collaboration
Rui Cao, Jiannong Cao, Bo Yuan et al. (+2 more)
Long-term action anticipation (LTA) aims to predict an ordered sequence of future verb-noun actions from a partially observed
video. While this task serves as the foundation for
embodied intelligence, anticipating physically feasible long-term actions remains a critical challenge. Existing methods, which operate in an open-loop manner, often hallucinate non-existent objects, violate object affordances, or disregard object states, as they lack explicit mechanisms to verify action feasibility against the physical environment. To address this, we propose FactCheck, a novel multi-
agent collaboration framework that improves feasibility through a closed-loop "Observe-Plan-Verify" mechanism. FactCheck decomposes the complex LTA task into specialized roles: an Observer that recognizes historical actions from
video observations and constructs a dual-form structured
memory, comprising a History Action Abstract that captures high-level human intentions and environmental status, and a History Action Graph that encodes object states and temporal dependencies; a Planner that generates draft future actions conditioned on both low-level historical actions and high-level History Action Abstract; and a Verifier that rigorously validates the draft against the History Action Graph and refines infeasible actions. Extensive experiments on the EPIC-Kitchens-55 and EGTEA Gaze+
benchmarks demonstrate that FactCheck consistently outperforms state-of-the-art methods. Our work establishes a new paradigm for feasibility-aware long-term action anticipation, effectively closing the loop of action recognition, action prediction and action verification.
Read the full abstract ▾
All authors Rui Cao, Jiannong Cao, Bo Yuan, Zhiyuan Wen, Mingjin Zhang
Computer Vision
JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence
Dingyu Yao, Junhao Zhou, Chenxu Yang et al. (+17 more)
Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a
video call, or a product a viewer wants flashes by in a livestream. Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even
video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale,
vision-first VL-interaction model. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at
vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing
video into the model, making it genuinely present in the world. All other components are pluggable, including ASR/TTS modules,
memory, visualization UI, and a background
brain that can connect to any API or
agent. Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app
video-call assistants of Doubao and Gemini by a wide margin. To our knowledge, this is the first open,
vision-driven interaction model released together with its training recipe, data, and complete deployable system.
Read the full abstract ▾
All authors Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Xiangyu Zeng, Yifei Li, Haowen Hou, Zheming Liang, Congcong Wang, Kaiwen Tuo, Jun Zhang, Yuhan Zhu, Yuhang Cao, Shenglong Ye, Shuai Xie, Shuhuan Gu, Haoyang Huang, Qingyi Si, Nan Duan, Jiaqi Wang
Computer Vision
TSA: Temporal Slot Activation for Persistent Object-Centric Video Representation
Duc Nguyen, Sieu Tran, Hao Vo et al. (+6 more)
Unsupervised
video object-centric learning aims to decompose dynamic scenes into temporally persistent entity representations. Existing recurrent
video slot-
attention methods propagate a fixed set of slots across frames, but typically assume unconditional slot propagation: every slot is updated and decoded at every frame, regardless of whether its corresponding object is visible. We show that this design violates a basic lifecycle requirement for persistent slots: when an object is absent or fully occluded, its slot should preserve its previous state and avoid explaining unrelated visible content. Instead, unconditional propagation creates update-induced state drift, where current-frame evidence overwrites the absent object's representation, and decoder-induced reconstruction interference, where the inactive slot remains coupled to reconstruction through decoder
attention. We propose Temporal Slot Activation (TSA), a lightweight mechanism that learns a per-slot, per-frame activation score $\alpha_{k,t}\in(0,1)$ without visibility supervision. TSA uses this activation as a shared latent control variable for slot lifecycle modeling. When a slot is inactive, TSA anchors its state to the previous slot via activation-gated updating and suppresses its decoder participation through an activation-dependent additive bias on
attention logits before softmax normalization. This jointly reduces state drift and reconstruction-driven interference. To improve decisions under partial occlusion and gradual reappearance, TSA further conditions activation prediction on a per-slot temporal
memory produced by a Temporal Context Encoder. We evaluate TSA on MOVi-C, MOVi-E, YTVIS-HQ, and the occlusion-heavy OVIS
benchmarks, using standard (FG-ARI, mBO) and tracking-based (IDF1, HOTA) metrics. TSA improves object decomposition and temporal identity preservation, with large gains on long, heavily occluded
videos.
Read the full abstract ▾
All authors Duc Nguyen, Sieu Tran, Hao Vo, Khoa Vo, Duy Minh Ho Nguyen, Nghi D. Q. Bui, Anh Nguyen, Long Mai, Ngan Le
Computer Vision · Cross-listed from cs.AI
Orchestra-o1: Omnimodal Agent Orchestration
Fan Zhang, Vireo Zhang, Shengju Qian et al. (+8 more)
The recent success of
agent swarms has shifted the paradigm of large
language model (LLM)-based
agents from single-
agent workflows to multi-
agent systems, highlighting the importance of
agent orchestration for task decomposition and collaboration. However, existing orchestration frameworks are limited to a narrow set of modalities and struggle to generalize to more complex settings where heterogeneous modalities coexist and interact. This limitation becomes particularly pronounced in omnimodal scenarios, where tasks require the unified understanding and coordination of diverse inputs such as text, image, audio, and
video. In this work, we propose Orchestra-o1, an omnimodal
agent orchestration framework designed to support efficient
agent collaboration across multiple modalities. Orchestra-o1 introduces a unified orchestration mechanism that enables modality-aware task decomposition, online sub-
agent specialization, and parallel sub-task execution. This scalable design allows
agent systems to effectively tackle complex real-world tasks involving heterogeneous information sources, surpassing the second-best approach by 10.3% accuracy on the OmniGAIA
benchmark. Furthermore, we introduce decision-aligned group relative policy optimization (DA-GRPO), an efficient agentic reinforcement learning approach for training Orchestra-o1-8B, which also achieves state-of-the-art performance against all existing open-source omnimodal
agents.
Read the full abstract ▾
All authors Fan Zhang, Vireo Zhang, Shengju Qian, Haoxuan Li, Hao Wu, Jinyang Wu, Donghao Zhou, Zhihong Zhu, Zheng Lian, Xin Wang, Pheng-Ann Heng
Computer Vision
Garvita Allabadi, Matteo Sodano, Roberto Estevão et al. (+4 more)
In-Context Learning (ICL) has become a powerful mechanism for adapting Large
Language Models (LLMs) to new tasks without fine-tuning. Extending this concept to Large
Multimodal Models (LMMs),
Multimodal In-Context Learning (M-ICL) relies on retrieving relevant examples, such as images, captions, or question-answer pairs, to guide predictions across tasks like classification, captioning, and visual question answering (VQA). Most existing approaches select in-context examples based on feature-space similarity, assuming that semantically similar samples provide the most useful context. However, our systematic analysis reveals that this assumption does not always hold: visually similar examples are not necessarily those that most effectively enhance in-context learning performance. To address this, we propose the Guided
Retrieval of In-context
Prompts (GRIP), a learnable
vision-only
retrieval framework that leverages feedback from LMMs to identify examples that truly improve model predictions. GRIP learns to distinguish beneficial from detrimental in-context examples through contrastive training, refining
retrieval beyond pure similarity. Across three
multimodal tasks, namely classification, captioning, and VQA, GRIP improves consistently over similarity-based
retrieval on Qwen2.5-VL-7B, with its strongest gains in classification on Idefics2-8B. Moreover, we demonstrate that retrievers trained with feedback from one open LMM can be transferred to other models without retraining, including closed-source GPT-4o and Gemini, enabling scalable and cost-efficient deployment of M-ICL. Code will be published upon acceptance.
Read the full abstract ▾
All authors Garvita Allabadi, Matteo Sodano, Roberto Estevão, Yuxiong Wang, Vikram Adve, Emre Kiciman, Ranveer Chandra
Computer Vision · Cross-listed from cs.RO
EquiDexFlow: Contact-Grounded SE(3)-Equivariant Dexterous Grasp Generative Flows
Clinton Enwerem, John S. Baras and Calin Belta
Most learned dexterous grasp generators relegate contact forces to a downstream verification step, so a kinematically-plausible pose can still violate the conditions for a stable physical grasp. We address this with EquiDexFlow, an SE(3)-equivariant flow-matching model that jointly predicts wrist pose, joint angles, fingertip contacts, surface normals, and contact forces from an object point cloud. Our architecture projects contacts onto the object surface and forces into the Coulomb friction cone by construction, so placement and friction compliance hold without loss penalties. We prove end-to-end SE(3) equivariance and verify it empirically over 200 rotations, with wrist residuals below $0.04^\circ$ and exactly zero joint deviation. Trained on 8,100 force-closure grasps across 81 objects for the 16-DoF Allegro Hand, our model achieves zero friction violations, the best composite score, and the lowest wrench residual among all ablation variants. We retarget decoded fingertip contacts to a 16-DoF LEAP Hand via per-finger inverse kinematics, and our hardware-feasible refinement places every joint at least 5% inside its actuator envelope while preserving wrench balance. On the physical
robot, retargeted EquiDexFlow-decoded grasps complete open-loop pick-and-hold trials on all six test objects, with every asymmetric object succeeding at both the canonical pose and a $120^\circ$ co-rotation.
Videos, code, and checkpoints are available at https://equidexflow.github.io.
Read the full abstract ▾
Computer Vision
VLADriveBench: Evaluating CoT-Action Relationship in VLA for Autonomous Driving
Thach Nguyen, Danhua Guo, Tom Lampo et al. (+2 more)
Vision-
language-action (VLA) models generate chain-of-thought (CoT)
reasoning alongside driving trajectories, but existing
benchmarks evaluate only trajectory quality and do not assess whether the CoT is relevant, consistent, or causally connected to the driving action. We introduce VLADriveBench, a framework that combines observational metrics (mentioning,
hallucination, contradiction, action
alignment) with a CoT intervention protocol to provide complementary views of the CoT-action relationship. Applying VLADriveBench to three models across two architectures, we find that the two analyses can diverge sharply: ORION scores highest on observational
alignment yet its CoT is epiphenomenal, while Alpamayo v1.5 scores lower yet its CoT is strongly causal, with visual salience gating the extent of CoT influence.
Read the full abstract ▾
All authors Thach Nguyen, Danhua Guo, Tom Lampo, Fei Wu, Burhan Yaman
Computer Vision
SalArt-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images
Ruotian Zhang, Xiaoxiao Sun, Junzhe Huang et al. (+2 more)
Vision-
language models (VLMs) are increasingly used to detect whether AI-generated images contain visible artifacts, yet their ability to analyze such artifacts remains poorly understood. A correct image-level decision can still hide important failures: a model may correctly flag an artifact while relying on the wrong visual cue, selecting the wrong region, or describing a defect that the image does not support. To evaluate these behaviors directly, we introduce SalArt-VQA, a diagnostic
benchmark for fine-grained SALient ARTifact understanding in AI-generated images. SalArt-VQA contains 950 images and 3,681 human-authored multiple-choice questions spanning artifact images, matched real reference images, and paired generated reference images. Four aligned question types evaluate presence detection, semantic localization, spatial grounding, and evidence-grounded defect identification, while the reference splits test calibration and abstention when the annotated defect is absent. Across 20 VLMs, SalArt-VQA reveals failures that image-level detection accuracy hides: the strongest model reaches 99.37% detection recall on artifact images but answers all four artifact-side questions correctly on only 53.26% of images. Comparing artifact images with artifact-free references reveals a sensitivity-calibration tradeoff: sensitive models often make unsupported artifact claims, while conservative models avoid false alarms largely by missing real artifacts. These results show that high artifact detection accuracy alone does not imply grounded artifact understanding. SalArt-VQA exposes these hidden failure modes and provides a fine-grained evaluation of whether VLM artifact claims are supported by local visual evidence.
Read the full abstract ▾
All authors Ruotian Zhang, Xiaoxiao Sun, Junzhe Huang, James Burgess, Serena Yeung-Levy
Computer Vision · Cross-listed from cs.CR
Amnesia: A Stealthy Replay Attack on Continual Learning Dreams
Ahmed Sharshar, Naveen Kumar Kummari and Mohsen Guizani
Continual learning (CL) models often use experience replay to reduce catastrophic forgetting, but their robustness to replay sampling interference remains underexplored. Existing CL attacks alter inputs or training pipelines (poisoning/
backdoors) and rarely include explicit auditable constraints, limiting realism. Here, auditability means a monitor can verify compliance from sampler-visible telemetry - e.g., logged replay index/label statistics - by checking that the realized replay class histogram stays close to a nominal baseline and that replay rate is unchanged per batch and/or over a rolling window. We study a limited-privilege insider who controls only replay index selection, not pixels, labels, or model parameters, while staying within auditable limits such as queue priorities. We introduce Amnesia, a replay composition attack that maximizes degradation under two budgets: a visibility budget delta bounding the TV/KL divergence from a nominal class histogram p0, and a mass budget f fixing the replay rate. Amnesia has two steps: (i) compute lightweight class utilities, such as EMA loss or confidence, to tilt p0 toward harmful classes; and (ii) project the tilt back into the delta-ball using efficient KL (exponential tilt) or TV (balanced mass redistribution) optimizers. A windowed scheduler enforces rolling audits. Across challenging CL
benchmarks and strong replay baselines, Amnesia consistently lowers final accuracy (ACC) and worsens backward transfer (-BWT). The KL variant delivers high impact while remaining largely undetected under multiple audit schemes, including per-batch and rolling-window checks. The TV variant is more damaging but easier to detect, especially under tight per-class constraints. These results expose index-only replay control as a practical, auditable threat surface in CL systems and establish a principled impact-visibility trade-off.
Read the full abstract ▾
Computer Vision
CD-RCM: Generalizable Continuous-Depth Novel View Synthesis for Reflectance Confocal Microscopy
Tooba Imtiaz, Milind Rajadhyaksha, Kivanc Kose et al. (+1 more)
Reflectance confocal microscopy (RCM) provides noninvasive, cellular-resolution "optical biopsies" of human skin in vivo by acquiring en-face images at successive depths, forming a sparse z-stack. Due to optical limitations, these stacks are anisotropic 3D volumes with lateral resolution (0.5 $\mu$m) $\sim$6 times higher compared to axial resolution, which is defined by the optical sectioning (3 $\mu$m), limiting the interpretation of tissue. Our goal is to provide continuous-depth visualization by interpolating intermediate sections and making the 3D volume isotropic. Such a representation permits arbitrary-direction sectioning, including histopathology-like cross-sectional examination, without requiring per-patient optimization. To that end, we introduce the first RCM-specific novel-view synthesis (NVS) approach, CD-RCM, a feedforward model that predicts realistic, unseen depths from sparsely sampled RCM stacks. Classical neural rendering methods focus on reconstruction from surface-level multi-view observations. In contrast to surface-level camera views, RCM can acquire optically sectioned en-face images of tissue beyond the surface up to 200 $\mu$m. However, during visualization of the RCM stacks, observations of the shallower sections (towards the surface) obscure the deeper ones. This unique axial imaging geometry and layer-dependent anatomical organization motivated our development of a tailored architectural and training framework that explicitly accounts for RCM's depth-resolved, occlusive imaging physics. Experiments demonstrate that CD-RCM achieves high-fidelity novel-view synthesis with sub-second inference time.
Read the full abstract ▾
All authors Tooba Imtiaz, Milind Rajadhyaksha, Kivanc Kose, Jennifer Dy
Computer Vision
ECA: Efficient Continual Alignment for Open-Ended Image-to-Text Generation
Jiangtao Kong, Peijun Zhao, Chun-Fu Chen et al. (+4 more)
Incremental Learning (IL) for Open-ended Image-to-Text Generation (OpenITG) enables models to continuously generate accurate, contextually relevant text for new images while preserving previously acquired knowledge. Unlike prior studies, this paper addresses a more practical scenario in which the predominant category of visual data shifts over time as environments evolve. In this context, we introduce a new notion of continual
alignment, which incrementally adapts the
alignment module within pre-trained VLMs to preserve high-quality cross-modal representations. Based on this idea, we propose Efficient Continual
Alignment (ECA), a novel exemplar-free IL approach for OpenITG. The key challenge is enabling the model to acquire new, task-specific features while minimizing interference with the established
alignment without accessing raw data from previous tasks. To address this, ECA employs three core mechanisms: a Mixture of Query (MoQ) module that adapts task-specific query
tokens, a Fisher Dynamic Expansion (FeDEx) that dynamically expands model structure based on a Fisher Information
Matrix (FIM)-based metric, and an
embedding dictionary with Dictionary Replay (DR) to retain past knowledge. To evaluate ECA's performance, we construct four new IL OpenITG
benchmarks that better reflect real-world scenarios. Experimental results demonstrate that ECA significantly mitigates catastrophic forgetting and improves IL performance compared to baseline methods. Code and
benchmarks are available at https://github.com/Snowball0823/ECA.
Read the full abstract ▾
All authors Jiangtao Kong, Peijun Zhao, Chun-Fu Chen, Youngwook Do, Shaohan Hu, Tianyi Zhou, Huajie Shao
Computer Vision
Context-Aware Feature-Fusion for Co-occurring Object Detection in Autonomous Driving
Binay Kumar Singh and Niels Da Vitoria Lobo
Object detection in autonomous driving requires precise localization and an inherent understanding of the relational context between co-occurring objects. In extremely complex heterogeneous environments rare classes, small-scale objects, and frequently appearing objects are difficult for standard object detection frameworks to handle. In this paper, we propose a novel framework called Context-Centric Feature Fusion (CCFF), which utilizes two
attention-based modules, Local Context Fusion Module (LCFM) uses the RoI-to-RoI self-
attention mechanism to resolve spatial interactions, mainly considering small and partially obscured objects, while Global Context
Attention Module (GCAM) converts the co-occurrence of objects priors by pooling top-K RoI features into a global context
attention token, avoiding the computational overhead of pixel-level global pooling. This fusion of local and object-centric global features yields contextualized
embeddings that enhance classification results and co-occurring objects detection. Our method is evaluated on two
datasets, Cityscapes and BDD100K which demonstrate significant improvement on relational consistency, achieving a Category-level Consistency Strategy (CCS) of 0.973 and 0.969, respectively. Furthermore, our approach produces substantial gains in small object detection (AP_S: 14.1%) and successfully recovers rare classes such as "Train" that are typically lost in large distributions. Our efficiency report shows that the framework processes images in real time with a 0.2 FPS overhead. The code is available at https://github.com/BinayKSingh/CCFF.
Read the full abstract ▾
Computer Vision
Dual-State Slot Attention: Decoupling Appearance and Identity for Video Object-Centric Learning
Sieu Tran, Duc Nguyen, Hao Vo et al. (+2 more)
Unsupervised
video object-centric learning aims to decompose dynamic scenes into persistent, object-level representations without supervision. However, existing slot-based methods struggle to maintain stable object identity in challenging settings such as rapid motion and partial occlusion. First, they typically encode both the per-frame appearance of an object and its identity across frames in a single slot vector, creating an objective conflict that leads to slot swapping: reconstruction requires sensitivity to transient visual changes, whereas temporal consistency requires invariance to them. Second, the
token renormalization used in Slot
Attention can amplify weakly attending slots, allowing them to absorb
tokens from other objects and destabilize slot-to-object correspondence. We propose Dual-State Slot
Attention (DSSA), a fully self-supervised framework that addresses these limitations by separating appearance from identity and by reducing spurious updates from weakly matching slots. DSSA decomposes each slot into a local state for per-frame appearance and an identity state for temporally stable object information, thereby aligning reconstruction and temporal consistency with separate representations. The identity state is updated through a learned recurrent transition that acts as a temporal filter on the local state, while competition-modulated aggregation (CMA) down-weights updates from weakly matching slots and prevents them from absorbing
tokens from other objects. Experiments on MOVi-C, MOVi-D, and YouTube-VIS demonstrate that DSSA consistently improves
segmentation quality and temporal consistency over prior methods, while also yielding stronger downstream object recognition and
video dynamics prediction. Code is available at \href{https://uark-aicv.github.io/dssa/}{uark-aicv.github.io/dssa}.
Read the full abstract ▾
All authors Sieu Tran, Duc Nguyen, Hao Vo, Khoa Vo, Ngan Le
Computer Vision · Cross-listed from cs.LG
Emerging Flexible Designs for Geospatial Multimodal Foundation Models
Philipe Dias, Waqwoya Abebe, Abhishek Potnis et al. (+4 more)
Foundation models are rapidly transforming Earth observation by enabling scalable pretraining across diverse unlabeled geospatial modalities. However, their architectural diversity ranging from encoder-only to encoder-decoder and masked autoencoding paradigms makes it challenging to assess performance trade offs in a consistent manner. In this work, we present an apples-to-apples comparison of leading FM architectures designed for geospatial
multimodal reasoning, with a particular focus on flexibility across varied spectral band configurations. We standardize pretraining using identical self supervised learning objectives and training
datasets, and evaluate all models under consistent parameterization on the GEOBench
benchmark across classification and
segmentation tasks. Our results offer new insights into the design trade-offs between model flexibility, modality
alignment, and downstream task performance. By highlighting architectural strengths and limitations under controlled conditions, this study provides practical guidance for building next generation geospatial foundation models capable of robust
multimodal reasoning.
Read the full abstract ▾
All authors Philipe Dias, Waqwoya Abebe, Abhishek Potnis, Aristeidis Tsaris, Dan Lu, Xiao Wang, Dalton Lunga
Computer Vision
Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs
Shayan Mohammadizadehsamakosh, Pritam Sarkar, Leonid Sigal et al. (+2 more)
Preference optimization is increasingly used to post-train medical large
vision-
language models (LVLMs), yet it operates at a much coarser granularity than the one that defines clinical correctness. Whether one response is clinically better than another usually comes down to a few decisive phrases, such as an anatomical laterality or a lesion attribute, and to whether each is supported by the image region the question concerns. Direct Preference Optimization (DPO) and its variants, by contrast, reduce the comparison to a single response-level scalar, an objective structurally unable to represent which
tokens carry clinical meaning. We show that this mismatch is costly: of the objectives we compare, response-level DPO places the least reward on the phrases that decide clinical correctness, and substituting supervised references for preferred responses opens a stylistic gap that the model exploits as a reward-hacking shortcut, raising preference accuracy but not clinical accuracy. To capture the full feedback in a clinical comparison, we propose Fine-grained Regularized Medical Preference Optimization (FiRe-MPO). Preference pairs are built by minimally editing the model's own generations, so that preferred and rejected responses differ only on clinically decisive spans, and each is paired with a lesion-corrupted image withholding the supporting visual evidence. Because span-localized rewards are sparse, we stabilize optimization with a bidirectional
token-wise KL regularizer. Across medical visual question answering and report generation, FiRe-MPO outperforms DPO, RRPO, and competing fine-grained objectives on two popular LVLMs, one medical and one general-purpose, while strengthening visual grounding and placing more reward on the medical phrases than the alternatives.
Read the full abstract ▾
All authors Shayan Mohammadizadehsamakosh, Pritam Sarkar, Leonid Sigal, Ali Etemad, Elham Dolatabadi
Computer Vision
High-Fidelity Two-Step Image Generation via Teacher-Aligned End-to-End Distillation
Dongyang Liu, Ruoyi Du, David Liu et al. (+7 more)
Few-step
diffusion distillation has become increasingly mature for 4-8-step generation, yet pushing further to 2 steps remains challenging. In this work, we introduce Z-Image Turbo++, a high-quality 2-step image generation model distilled from the 8-step Z-Image Turbo teacher. Our method addresses the central bottlenecks of increased task difficulty and limited model capacity in 2-step generation through three simple but effective design choices tailored to this regime. First, we propose Distribution-Aligned
Adversarial Learning, which uses teacher-generated images rather than external real images as real samples for GAN training, providing a more attainable and informative
adversarial target. Second, we adopt Step-Decoupled Parameterization, assigning independent model parameters to the two denoising steps to better match their distinct capacity demands. Third, we perform End-to-End Training with Iterative Regularization, allowing the first step to receive gradients from final image quality while preserving a meaningful intermediate generation through an explicit step-1 loss. Together, these designs substantially narrow the quality gap between 2-step and 8-step generation in both qualitative and quantitative evaluations, highlighting the potential of carefully tailored
distillation strategies for improving the quality-efficiency trade-off in few-step generation.
Read the full abstract ▾
All authors Dongyang Liu, Ruoyi Du, David Liu, Dengyang Jiang, Liangchen Li, Qilong Wu, Zhen Li, Steven C.H. Hoi, Hongsheng Li, Peng Gao
Computer Vision
HairPort: In-context 3D-aware Hair Import and Transfer for Images
Alireza Heidari, Amirhossein Alimohammadi and Ali Mahdavi-Amiri
Transferring hairstyles between images is an important but challenging task in computer graphics, computer
vision, and visual effects. It enables users to explore new looks without physically altering their hair, with applications in virtual try-on systems, augmented reality, and entertainment. Most prior works operate best under small pose gaps, and they fall short under large viewpoint and scale differences, where missing hair content must be synthesized rather than transferred. We propose HairPort, a 3D-aware hairstyle transfer framework that attempts to solve these issues by explicitly separating hair removal from transfer and enforcing
geometric consistency before synthesis. We introduce a Bald Converter, which produces realistic bald versions of faces through LoRA-based in-context adaptation of FLUX.1 Kontext. To train our Bald Converter, we introduce a new
dataset, Baldy, containing 6,000 paired bald and original images across diverse identities and conditions. We also use a 3D-Aware Transfer Pipeline that reconstructs and re-renders the reference hairstyle from the target viewpoint before compositing it onto the source image. Being 3D aware, our method supports large pose and scale discrepancies between the source and target. Finally, a conditional flow-matching generator synthesizes the transferred result from the bald source and geometry-aligned reference guidance. Together, our method enables accurate, pose-consistent, and identity-preserving hairstyle transfer, outperforming existing methods both qualitatively and quantitatively.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.SD
AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generation
Zeyue Tian, Lei Ke, Zhaoyang Liu et al. (+8 more)
Audio and music generation based on flexible
multimodal control signals is a widely applicable topic, with the following key challenges: 1) a unified
multimodal modeling framework, 2) large-scale, high-quality training data, and 3) the prohibitive inference cost of multi-step
diffusion sampling. As such, we propose AudioX-Turbo, a unified and efficient framework for anything-to-audio generation that integrates varied
multimodal conditions (i.e., text,
video, and audio signals) in this work. AudioX-Turbo follows a teacher-student paradigm. The teacher AudioX-Base is built on a
Multimodal Diffusion Transformer with a
Multimodal Adaptive Fusion module that aligns diverse
multimodal inputs for high-fidelity synthesis, and is then distilled into the few-step student AudioX-Turbo via Distribution Matching
Distillation adapted to flow matching, complemented by a
diffusion-based discriminator for high-quality few-step generation. To support the training of AudioX-Turbo, we construct a large-scale, high-quality
dataset, IF-caps-Pro, comprising approximately 9.2M samples curated through a two-stage data collection and annotation pipeline. We
benchmark AudioX-Turbo across a wide range of tasks, finding that our model achieves superior performance, especially on text-to-audio and text-to-music generation, while operating at only 4 sampling steps and requiring approximately 25x fewer function evaluations (NFE) than multi-step baselines. These results demonstrate that our method is capable of audio generation under flexible
multimodal control, showing efficient and powerful instruction-following capabilities. The code and
datasets will be available at https://zeyuet.github.io/AudioX-Turbo/.
Read the full abstract ▾
All authors Zeyue Tian, Lei Ke, Zhaoyang Liu, Ruibin Yuan, Liumeng Xue, Yujiu Yang, Weijia Chen, Xu Tan, Qifeng Chen, Wei Xue, Yike Guo
Computer Vision
Stereo Vision-Based Fall Prediction and Detection using Human Pose Estimation on the AMD Kria K26 SOM
Shreyas Narasimhiah Ramesh, P. D. Rathika, Mahasweta Sarkar et al. (+3 more)
Background and Objective: Falls among elderly people can cause serious injury and reduce quality of life. Timely prediction and detection are essential to prevent harm and support well-being. We propose a portable, low-power, battery-operated,
vision-based fall prediction and detection system using HPE on an AMD Kria K26 System-on-Module (SOM). The objective is a non-intrusive,
privacy-preserving system for real-time fall detection. Methods: The system uses an Intel RealSense D455 range-sensing camera connected to the K26 SOM by USB. It captures synchronized RGB and depth frames, 640 x 480 x 3 and 640 x 480 pixels, at 60 FPS. The SOM runs a three-stage pipeline with quantized YOLOX, Anchor-to-Joint (A2J), and fall-detection models. YOLOX identifies human bounding boxes from RGB frames, then discards the RGB frames to preserve
privacy. A2J uses depth frames to estimate 15 joint keypoints per person. A CNN uses selected joint coordinates (x, y, z) to classify fall activity. YOLOX was trained on CrowdHuman; A2J on ITOP, MP-3DHP, UR Fall Detection, and a custom SDSU PSG
dataset; and the CNN on UR Fall Detection and SDSU PSG. The design used a single-core DPU with a serial pipeline and a dual-core DPU running YOLOX and A2J with multiple threads. Results: Quantized accuracy was evaluated using IoU >= 50% for YOLOX, mAP with a 10-cm rule for A2J, and classification accuracy, (TP + TN)/(TP + TN + FP + FN), for the CNN. Accuracies were 74%, 84.13%, and 75.85%. Throughput improved from 2.5 FPS for the single-threaded pipeline to 4.5 FPS for the multi-threaded version. Conclusion: Results demonstrate the feasibility of
privacy-preserving fall detection on an AMD Kria K26 edge device. On-device HPE and fall classification runs without cloud dependency, supporting elderly monitoring and assistive healthcare. Future work will improve model accuracy and speed.
Read the full abstract ▾
All authors Shreyas Narasimhiah Ramesh, P. D. Rathika, Mahasweta Sarkar, Kristen Wells, Michel Audette, Christopher Paolini
Computer Vision
Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models
Cheng-Yu Yang, Shao-Yuan Lo and Yu-Lun Liu
Vision-
language models (VLMs) project images into hundreds to thousands of visual
tokens, making decoder inference expensive in both
attention computation and KV-cache
memory. Existing visual-
token reduction methods largely follow a rank-and-remove paradigm: they score visual
tokens, keep a compact subset, and permanently discard the rest. We show that this irreversible action is fragile because visual-
token importance changes across decoder depth;
tokens ranked low at one stage may become relevant in later layers, especially for grounding-sensitive queries. We propose Reroute, a training-free plug-in that replaces removal with recoverable routing. At each routing stage, selected
vision tokens pass through decoder blocks, while deferred
tokens bypass the stage and re-enter the candidate pool at the next routing decision. Reroute reuses existing
attention-score ranking rules and stage-wise schedules, preserving the theoretical TFLOPs and KV-cache budget class of the
pruning method it augments. Across FastV, PDrop, and Nüwa variants on LLaVA-1.5 and Qwen backbones, reroute improves grounding under aggressive
token reduction while maintaining general VQA performance. These results suggest that VLM
token reduction should not be viewed only as irreversible
pruning, but also as recoverable routing. The code can be found here: https://github.com/elmma/mllm-reroute/
Read the full abstract ▾
Computer Vision
How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology
Kian R. Weihrauch, Thomas A. Buckley, William Lotter et al. (+1 more)
General-purpose large
language models (LLMs) are routinely used as baselines when evaluating specialized pathology models on whole-slide images (WSIs). Because WSIs exceed contemporary model context limits, LLM baselines routinely use small, high-magnification patches processed independently via majority voting, without systematic evaluation of seemingly inconsequential design choices such as patch size, patch count, and magnification. Generalist LLMs have consistently underperformed specialized systems, reinforcing the perception that domain-specific training or architectural adaptation is necessary for pathology tasks involving WSIs. Here, we conduct a systematic factorial analysis of four input design factors: inference mode, patch size, magnification, and patch count. We demonstrate that prior studies have overstated the gap between specialized models and general-purpose LLMs by choosing non-optimized input configurations. On the MultiPathQA
benchmark, switching to a single balanced configuration (large patches at lower magnification, processed jointly) raises GPT-5 from 15.1% to 39.5% on cancer-type classification (TCGA) and from 38.1% to 62.9% on organ classification (GTEx). Per-task optimization yields further gains up to 43.9% (TCGA) and 71.6% (GTEx). The same configuration generalizes to two other models and to a fully held-out CPTAC cohort, where it improves Gemini 3 Flash by 23.4 percentage points without any task-specific tuning.
Read the full abstract ▾
All authors Kian R. Weihrauch, Thomas A. Buckley, William Lotter, Arjun K. Manrai
Computer Vision · Cross-listed from cs.RO
DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?
Jadelynn Dao, Milan Ganai, Yasmina Abukhadra et al. (+7 more)
Vision-
Language Models (VLMs) are increasingly deployed as high-level planners for
embodied agents, with an emerging strategy of scaling test-time compute to improve capability. However, we observe that doing so increases latency,
token usage, and FLOPs while yielding uneven, often diminishing gains in downstream success, limiting where
embodied agents can be deployed. We argue that choosing when and where to spend test-time compute is central to bringing frontier performance to the real world, and we can budget inference at the planning level. We introduce DIRECT, a routing framework that uses
multimodal scene context to allocate compute at each step of a closed-loop
agent, selecting a planner for every subtask, improving the success–cost Pareto frontier over fixed model selection. Across three dominant scaling axes, namely chain-of-thought depth, model size, and
memory history, our experiments on VLABench and RoboMME show that test-time compute is not a uniform lever: different axes yield qualitatively distinct capability gains. We validate these insights on a physical Franka arm in a DROID setup spanning zero-shot manipulation and long-horizon chaining, where our router matches or exceeds a stronger model's success rate at up to 65% lower average latency. Ultimately, our results show that naively scaling test-time compute is wasteful, and DIRECT can provide frontier-level
embodied planning at a fraction of the cost. Project page can be found at jadee-dao.github.io/direct/.
Read the full abstract ▾
All authors Jadelynn Dao, Milan Ganai, Yasmina Abukhadra, Ajay Sridhar, Mozhgan Nasr Azadani, Katie Luo, Clark Barrett, Jiajun Wu, Chelsea Finn, Marco Pavone
Computer Vision
VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving
Jin Yao, Dhruva Dixith Kurra, Tom Lampo et al. (+3 more)
Vision-
language-action (VLA) models can describe scenes and reason about them in
language, yet still struggle to ground their actions in the dense 3D world around them. Existing approaches either inject features from a frozen 3D foundation model without an objective that ensures the policy uses them, or constrain geometry with sparse box and map losses that provide no dense spatial signal. We introduce VLGA, the first
vision-
language-action model supervised to reconstruct the dense 3D world it drives through. VLGA introduces geometry as a fourth modality alongside
vision,
language, and action through a dedicated expert supervised by a per-pixel pointmap regression loss against
LiDAR. Extensive experiments conducted on challenging nuScenes and Bench2Drive
datasets for open-loop and closed-loop evaluations, respectively, show the superiority of VLGA over counterpart VLA methods. In particular, on open-loop nuScenes, VLGA sets a new state of the art among VLA methods without ego status, with the lowest L2 (0.50\,m average) and 3-second collision rate (0.18%). On closed-loop Bench2Drive, VLGA attains the state-of-the-art driving score of 79.08, +0.71 over the strongest prior VLA, at comparable efficiency and comfort.
Read the full abstract ▾
All authors Jin Yao, Dhruva Dixith Kurra, Tom Lampo, Zezhou Cheng, Danhua Guo, Burhan Yaman
Computer Vision
Illumination-Robust Camera-Based Heart-Rate Estimation for Physiological Sensing in Robots
Zhi Wei Xu and Torbjörn E. M. Nordling
Physiological awareness is important for service, social, and assistive
robots that interact with humans in everyday environments. Remote photoplethysmography (rPPG) enables non-contact heart-rate (HR) estimation from an RGB camera, making it a promising sensing modality for
robot-mounted
vision systems. However, illumination variation remains a major barrier to robust deployment. This paper presents an end-to-end spatial-temporal
transformer framework for remote HR estimation on a new
dataset with varied illumination. Our
estimator integrates PRNet-based 3D face
alignment, clip-level illumination augmentation, the Residual Temporal Standardization Module, and controlled hybrid temporal-frequency supervision. The training objective combines a Soft-Shifted Pearson waveform loss with a spectral Kullback-Leibler divergence loss, where a tuned weight ($\mathbf{\beta}$) controls the contribution of frequency-domain heart-rate guidance. Experiments on a static all-level mix protocol covering three illumination levels show that $\mathbf{\beta}=5$ provides the strongest result among the tested beta settings, achieving a best-run HR mean absolute error (MAE) of 0.79 bpm and an HR correlation of 0.982. Compared with the PhysFormer baseline evaluated on our
dataset, our
estimator reduces HR MAE by 93.6 %, while increasing HR correlation from 0.088 to 0.982, making it usable when illumination varies.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Semantically-Aware Diver Activity Recognition Framework for Effective Underwater Multi-Human-Robot Collaboration
Sadman Sakib Enan and Junaed Sattar
Effective multi-human-
robot collaboration is essential for expanding human-led operations in the challenging and high-risk
underwater environment. For autonomous
underwater vehicles (AUVs) to become true teammates, they must be able to comprehend their surroundings and recognize a diver's activities to offer assistance and ensure safety. Towards this goal, we introduce DAR-Net, a novel
transformer-based framework that analyzes complex
underwater scenes to classify diver activities. Our contribution lies in a semantically guided learning formulation that couples
transformer-based temporal
reasoning with pixel-level scene supervision. This multi-loss training strategy explicitly aligns global activity recognition with local human-
robot interaction semantics, which is particularly critical in low-visibility
underwater conditions. To address the significant challenge of data scarcity in this domain, we present the first-ever
Underwater Diver Activity (UDA)
dataset, a foundational resource containing over 2,600 annotated images with pixel-level masks. Through rigorous experimental evaluations in a controlled environment, we demonstrate that DAR-Net achieves promising accuracy in recognizing six distinct diver activities, outperforming state-of-the-art models. While this
dataset provides a crucial baseline, our work serves as a pioneering step, laying the groundwork for future research and facilitating the development of more intelligent, collaborative
underwater robotic systems.
Read the full abstract ▾
Computer Vision
A Turbo-Inference Strategy for Object Detection and Instance Segmentation
Zhen Zhao, Gang Zhang, Xiaolin Hu et al. (+1 more)
Object detection and instance
segmentation tasks are closely related. Existing top-down instance
segmentation methods usually follow a detect-then-segment paradigm, where an initial detector is used to recognize and localize objects with bounding boxes, followed by the
segmentation of an instance mask within each bounding box. In such methods, the detection accuracy directly influences the subsequent
segmentation performance. However, previous research has seldom explored the impact of the instance
segmentation task on object detection. In this paper, we present a turbo-inference strategy for the top-down methods that leverages the complementary information between detection and
segmentation tasks iteratively. Specifically we design two modules: turbo-detection head and turbo-
segmentation head, which facilitate communication between the tasks. The two modules form a closed loop that interlaces the detection and
segmentation results without retraining the model. Comprehensive experiments on the COCO, iFLYTEK, and Cityscapes
datasets demonstrate that our method substantially enhances both detection and
segmentation accuracies with a certain increase in computational cost. The proposed method represents a tradeoff between prediction accuracy and inference speed. Codes are available at https://github.com/zhaozhen2333/Turbo-Learning.git.
Read the full abstract ▾
All authors Zhen Zhao, Gang Zhang, Xiaolin Hu, Liang Tang
Computer Vision
DepthMaster: Unified Monocular Depth Estimation for Perspective and Panoramic Images
Pengfei Wang, Shihao Wang, Liyi Chen et al. (+3 more)
While
monocular depth estimation has achieved significant progress, achieving generalized metric depth estimation for both narrow field-of-view (FoV) perspectives and $360^\circ$ panoramas remains an unsolved challenge. Existing methods are often tailored to specific camera types and struggle to produce accurate metric depth that generalizes across diverse settings. This limitation stems from two key challenges: the inherent
geometric discrepancy between perspective and panoramic cameras, and the scarcity of panoramic training data with metric annotations. In this work, we introduce DepthMaster, a unified metric depth estimation framework. Rather than employing specialized networks to learn spherical distortions, we reformulate the problem by decomposing panoramic images into overlapping perspective patches. Crucially, distinct from prior projection-based methods that rely on ad-hoc architectural modifications to handle boundaries, we introduce a novel Correspondence Consistency Loss (CCL) and inject virtual projection cameras as
geometric priors, allowing us to seamlessly stitch the patches while avoiding specialized operators and keeping the backbone largely compatible with standard
Transformer designs. This strategy also resolves the
geometric differences by unifying all inputs into a canonical perspective representation, and effectively circumvents data scarcity by directly unlocking powerful metric priors from vast perspective
datasets. Trained on a mixed
dataset that contains only one panorama
dataset, DepthMaster achieves state-of-the-art zero-shot performance on 13 diverse
datasets, outperforming not only universal methods but also leading specialist models in both perspective and panoramic domains.
Read the full abstract ▾
All authors Pengfei Wang, Shihao Wang, Liyi Chen, Zhiyuan Ma, Guowen Zhang, Lei Zhang