Computer Vision
Hongsen Cao, Mona Jaber, Shanxin Yuan et al. (+1 more)
Token-
pruning policies are usually designed for a single recognition pipeline, but pretrained
Vision Transformers are reused across tasks with different spatial demands. We ask which parts of a
pruning policy transfer across image classification, semantic
segmentation, and object detection. For each pipeline, controlled probes freeze the no-
pruning checkpoint and apply a series of parameter-free reduction criteria at one eligible layer at a time without retraining. The probes reveal three differences:
segmentation and detection rank the criteria differently, classification is especially sensitive to
attention-based
pruning in the earliest layers, and the dense tasks prefer opposite recovery endpoints. These findings motivate Task-Adaptive
Pruning (TAP). Existing register
tokens serve as task-agnostic storage for feature artifacts. TAP instead introduces one task register per task and activates only the current one. Its evolving state ranks
tokens, distributes an exact removal budget over depth, and sets the recovery scale for dense features. At a final keep rate of $\rho=0.5$, our jointly adapted model, TAP-J, reaches $47.0$ mIoU at $1.30\times$ encoder throughput on ADE20K and $53.7$ box AP at $1.32\times$ encoder throughput on COCO while remaining competitive on ImageNet-1K.
Read the full abstract ▾
All authors Hongsen Cao, Mona Jaber, Shanxin Yuan, Ahmed Sayed
Computer Vision
PEAK: Precise and Persistent Concept Erasure via k-Sparse Autoencoders
Man Jiang, Ouxiang Li, Weibao Xue et al. (+4 more)
Erasing concepts from large-scale text-to-image (T2I)
diffusion models has become increasingly crucial due to the growing concerns over copyright infringement,
privacy violations, and offensive content. Existing approaches struggle to achieve both precise and persistent concept erasure: inaccurate localization of concept-related representations may cause unintended semantic interference, while incomplete removal of the underlying concept knowledge allows
adversarial recovery. To address this dilemma, we propose PEAK, a \textbf{precise} and \textbf{persistent} concept erasure framework via k-Sparse Autoencoders (kSAEs). PEAK first trains a kSAE on internal activations of the
diffusion denoising network to decompose dense representations into interpretable sparse features. By contrasting sparse activations induced by target and non-target
prompts, PEAK identifies a compact set of target-specific features according to both activation strength and frequency. These localized features are then used for parameter optimization, where PEAK selectively suppresses target-related activations while preserving complementary non-target ones towards the original model. This feature-guided optimization embeds concept erasure directly into
diffusion parameters, eliminating the need for additional inference-time intervention and facilitating effective persistence against
adversarial attacks. Extensive experiments demonstrate that PEAK achieves effective and robust concept erasure. On the I2P
benchmark, PEAK reduces NudeNet detections from 582 to 6, lowers the average attack success rate (ASR) from 96.52% to 5.63%, and preserves general generation quality on MS-COCO with a near-zero KID. Our code and models are available at: https://github.com/manmanTAT/PEAK
Read the full abstract ▾
All authors Man Jiang, Ouxiang Li, Weibao Xue, Zhenhua Tang, Yuan Wang, Shuo Wang, Yanbin Hao
Computer Vision
ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes
Xinrui Lin, Sha Zhang, Shumin Wang et al. (+3 more)
Task-driven 3D affordance grounding aims to localize the functional region in a cluttered 3D scene that enables an action specified by a natural-
language instruction. Existing methods either predict 3D masks directly or construct them by selecting and fusing intermediate 2D/3D regions. However, they remain vulnerable to two intertwined failure modes: the predicted or selected regions may miss the target interaction area or have unsuitable granularity, while
language grounding may confuse visually similar alternatives under relational instructions. To this end, we introduce ThinkAfford, which decouples high-recall affordance proposal generation from instruction-grounded
reasoning. Specifically, the Affordance Proposal Generation module first uses learnable affordance
prompts and multi-level visual features to predict interaction-conditioned heatmaps, extracting a variable number of fine-grained proposals without parsed object or part names as
segmentation prompts. Visual-Prompted Affordance
Reasoning then reasons over labeled proposal overlays using the full instruction, returning identifiers in a structured "think-then-answer" response. Moreover, Group Relative Policy Optimization uses proposal-level rewards from lifted 3D overlap to align VPAR selection with final 3D grounding. On the SceneFun3D validation split, ThinkAfford achieves 10.69% AP50 and 25.46% AP25 under the official evaluator, outperforming comparable 3D open-vocabulary and
vision-
language-model-based 2D-to-3D baselines. Module-level diagnostics further show that APG attains 77.5% recall at 25% intersection-over-union, while GRPO-trained VPAR achieves 72.1% selection accuracy on APG-covered queries, compared with 63.4% under supervised fine-tuning.
Read the full abstract ▾
All authors Xinrui Lin, Sha Zhang, Shumin Wang, Zenghuan Zhu, Jiajun Deng, Yanyong Zhang
Computer Vision
A Dataset and Benchmark for Optical Music Recognition of String Quartet Scores
Dongmin Kim, Brian Liu, Jose J. Valero-Mas et al. (+1 more)
Optical music recognition (OMR) transcribes music scores into digital formats. While the field has advanced significantly on monophonic and piano-form scores, multi-part score transcription remains underexplored, largely due to the absence of a suitable
dataset. We introduce OpenScore String Quartet for Optical Music Recognition (OSSQ-OMR), the first
dataset dedicated to multi-part OMR. Built on the OpenScore String Quartet corpus, OSSQ-OMR pairs digitally encoded scores with their original scanned editions from IMSLP, with all images visually aligned to their transcriptions. The
dataset is released with score images at system and staff levels, and paired transcriptions in three encoding formats: Extended Linearized MusicXML (LMXE), **kern, and ABC. In total, OSSQ-OMR contains 24,544 system images and 98,172 staff images drawn from 116 string quartet scores. We accompany the
dataset with a
benchmark protocol and baseline results from two representative OMR models, evaluated across four random score-level splits with mutually exclusive test sets. Baselines reach OMR-NED as low as 3.6% on synthetic and 5.9% on scanned inputs; results reveal substantial effects of encoding and
segmentation choices, with the LSTM-based baseline degrading on scanned inputs roughly 2.6 times less than the
Transformer-based baseline.
Read the full abstract ▾
All authors Dongmin Kim, Brian Liu, Jose J. Valero-Mas, Dasaem Jeong
Computer Vision
CARE: Confidence-Aware Reasoning for Reliable Medical VQA
Yuetian Du, Yucheng Wang, Zhenyuan Chen et al. (+9 more)
Reinforcement Fine-Tuning (RFT) has enabled medical
Multimodal Large
Language Models (MLLMs) to produce Chain-of-Thought (CoT)
reasoning for visual question answering, yet these models suffer from $\textit{confidence miscalibration}$—a systematic gap between expressed certainty and actual diagnostic accuracy that undermines clinical trust. We propose $\textbf{CARE}$, a $\textbf{C}$onfidence-$\textbf{A}$ware medical $\textbf{RE}$asoning framework that jointly optimizes accuracy and calibration through a dual-stage pipeline. First, a scalable Medical-CoT synthesis provides structured cold-start data for Supervised Fine-Tuning. Second, Group Relative Policy Optimization (GRPO) with a novel $\textbf{Confidence-Aware Reward (CAR)}$ mechanism ties the model's confidence to diagnostic correctness within the reward signal. Across three Medical VQA
benchmarks, $\textbf{CARE}$ achieves the highest diagnostic accuracy while obtaining the lowest Expected Calibration Error and
Hallucination Rate, establishing a foundation for trustworthy clinical decision support. Our code is available at https://github.com/anotherbricki/CARE.
Read the full abstract ▾
All authors Yuetian Du, Yucheng Wang, Zhenyuan Chen, Luyuan Chen, Rongyu Zhang, Jinjian Zhang, Wei Zhou, Zhijie Xu, Ming Kong, Zhan Zhou, Jie Liu, Qiang Zhu
Computer Vision
Once Poisoned, Arbitrarily Controlled: A Programmable Backdoor in VLMs
Tao Lin, Gaojie Jin, Zongxin Liu et al. (+2 more)
Existing
vision-
language model (VLM)
backdoors are usually treated as static vulnerabilities: one-to-one and N-to-N attacks bind one or more triggers to a finite set of targets before victim training. This assumption substantially underestimates the threat. We show that a single poisoning phase can implant a programmable
backdoor into a VLM, allowing an attacker to choose previously unseen target-caption semantics at inference time and synthesize corresponding stealthy triggers on demand. Unlike fixed-mapping attacks, the proposed any-to-any caption-control paradigm decouples post-training target selection from poisoning, enabling dynamic control of target captions without retraining the VLM. Our method has two components. First, a heuristic poisoning strategy exposes the model to diverse trigger-caption pairs, encouraging it to learn a general trigger-as-instruction rule rather than memorize a specific
backdoor pattern. Second, a feature-space trigger steganography method maps any attacker-specified target caption to a stealthy visual trigger, implemented as either a norm-controlled perturbation or a non-semantic patch. Once inserted into arbitrary images, these triggers cause the poisoned VLM to generate outputs semantically aligned with the chosen target caption, even when the target was unseen during poisoning. Extensive experiments show that our attack achieves high any-to-any caption-control success rates, preserves clean model utility, and remains effective under several classical
backdoor defenses.
Read the full abstract ▾
All authors Tao Lin, Gaojie Jin, Zongxin Liu, Peng Wu, Lijia Yu
Computer Vision
Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenes
Zhaoyang Wei, Bowen Jiang, Xumeng Han et al. (+6 more)
While
Multimodal Large
Language Models (MLLMs) demonstrate impressive performance in benign scenarios, their cognitive reliability deteriorates significantly in complex scenes under adverse conditions. In these settings, models often rely on implicit inference without sufficient visual evidence, leading to a disconnect between perception and
reasoning. Meanwhile, existing outcome-oriented
benchmarks evaluate only final predictions and fail to diagnose failures in the underlying
reasoning process. To address this gap, the authors propose AD2-Bench, which introduces a Hierarchical Visual Diagnosis framework that decomposes
reasoning into a structured Chain of Evidence (CoE). This fine-grained diagnosis reveals that robust
multimodal reasoning fundamentally depends on accurate evidence acquisition. Building on this perspective, the authors formulate
reasoning from a probabilistic viewpoint and identify two primary causes of
reasoning failure: Spatial Ambiguity, where models fail to distinguish target objects from background clutter, resulting in localization errors; and Semantic Uncertainty, where degraded visual features lead to incorrect semantic interpretation, resulting in understanding errors. To overcome these evidence deficiencies, they further propose Evidence-grounded Visual
Reasoning (EGVOR), which replaces implicit
reasoning with the explicit generation of Evidence Atoms - structured spatial-semantic triplets that enforce tight
alignment between localization and semantic understanding. The model is trained through a hierarchical curriculum that progresses from reflective supervision construction to reinforcement learning, where reducing
reasoning variance is explicitly rewarded. Extensive experiments demonstrate that EGVOR substantially improves
reasoning stability under adverse conditions, providing a more robust framework for trustworthy
multimodal cognition.
Read the full abstract ▾
All authors Zhaoyang Wei, Bowen Jiang, Xumeng Han, Jiashu Li, Xuehui Yu, Yuling Liu, Guorong Li, Zhenjun Han, Jianbin Jiao
Computer Vision
Multiple Scale Latents for Learned Image Compression
Jonas Brenig and Radu Timofte
Most learned image compression systems rely on a single latent representation combined with a hyperprior, which limits their ability to efficiently capture image structure across spatial scales. In this work, we propose a hierarchical latent representation to improve the efficiency of the
entropy model. By using multiple latents at different scales, each with its own
entropy model, we better capture the spatial structure of the latent representation. Our experiments show that this approach achieves a 17.9% BD-rate reduction over VVC on Kodak, demonstrating the effectiveness of multi-scale latent representations. Furthermore, the approach is orthogonal to other advances in learned image compression, making it a versatile addition to existing methods.
Read the full abstract ▾
Computer Vision
StreamFlow: Dynamic Memory Flows for Streaming Video Understanding
Muxin Fu, Yifan Zhang, Wentao Zhang et al. (+5 more)
Streaming
video understanding requires
multimodal large
language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded
memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while
memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual
memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term
memory that filters temporal redundancy before visual encoding with a latent long-term
memory that consolidates historical
video content into visual latents accessible to subsequent
reasoning. During generation, an
attention-guided
retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming
video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-
video benchmarks. Relative to the vanilla setting, it improves the visual
attention score (VAS) by 59.1% while reducing end-to-end latency and peak
memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient
reasoning.
Read the full abstract ▾
All authors Muxin Fu, Yifan Zhang, Wentao Zhang, Fangming Guo, Qian Chen, Guibin Zhang, Shuicheng Yan, Bo An
Computer Vision
Mixture-of-Experts-based Entropy Model for Learned Image Compression
Jonas Brenig and Radu Timofte
Learned image compression has seen significant progress in recent years with the development of end-to-end learned models that achieve better compression efficiency than state-of-the-art conventional methods. Recently, Mixture of Experts (MoE) approaches have seen promising results in NLP and computer
vision tasks. In this paper, we introduce the MoE approach to learned image compression. We propose a MoE-based
Entropy model (MoEE) for learned image compression, allowing the model to selectively activate only the subset of parameters required for the input image. Our model achieves a BD-Rate improvement over VVC of -16.85% on the Kodak
dataset.
Read the full abstract ▾
Computer Vision
GS-CPE: Unified 6-Degree-of-Freedom Camera Pose Estimation via 3D Gaussian Splatting
Huaiyuan Weng, Chul Min Yeum and Su-Min Kang
Despite substantial progress in visual localization, from scene coordinate regression to direct camera pose regression, achieving both robust generalization and high accuracy remain challenging. This study introduces GS-CPE (Gaussian Splatting based Camera Pose Estimation), a coarse-to-fine framework for 6-DoF camera pose estimation that unifies geometry-based coarse pose estimation with robust 3D Gaussian Splatting (3DGS) warping based pose refinement. GS-CPE first estimates a coarse pose via
retrieval-guided
geometric pose estimation on a 3DGS scene representation, then refines it by minimizing a visibility aware masked RGB warping objective in a multi-scale optimization framework, with adaptive re-rendering. Extensive experiments on indoor and outdoor
benchmarks including 7Scenes, Cambridge Landmarks, FAST-LIVO2
datasets, and a custom
dataset demonstrate state-of-the-art performance, consistently outperforming in both accuracy and generalization.
Read the full abstract ▾
Computer Vision
SafeCA: Safe Cross-Attention Localization and Regulation for Text-to-Video Jailbreak Defense
Siyuan Liang, Yupeng Qiu, Junfeng Fang et al. (+3 more)
Text-to-
Video (T2V) generative models are vulnerable to jailbreak attacks in real-world deployment, leading them to produce harmful or inappropriate content. Existing defense approaches mainly rely on input filtering or reconstruction, which not only incur high computational latency but also tend to distort semantics. To address these issues, we experimentally and systematically analyze the differences between clean and jailbreak samples in the cross-
attention feature space, revealing for the first time a cumulative separation effect and a progressively increasing trend of linear separability between the two during the
diffusion process. Based on this insight, we propose SafeCA, a feature-level defense mechanism for safe cross-
attention localization and regularization. Firstly, we identify key defensive regions and values through
attention stability analysis using cross-
attention features collected from clean
prompts within a single inference. Secondly, SafeCA mitigates anomalous activations via
attention masking with energy normalization and introduces a lightweight semantic-space adapter to redirect abnormal semantic flows. Furthermore, we detect and suppress potentially malicious
tokens by back-propagating feature anomaly signals to the input cue words, thereby enhancing the deployability of the defense in commercial models. Experimental results show that SafeCA reduces the jailbreak success rate by about 20% on mainstream T2V models, adds almost no inference overhead (+0.1s), and maintains good text-
video semantic consistency. Overall, SafeCA provides an architecture-level, deployable protection paradigm for T2V generation models.
Read the full abstract ▾
All authors Siyuan Liang, Yupeng Qiu, Junfeng Fang, Rong-Cheng Tu, Jiaxing Huang, Dacheng Tao
Computer Vision
Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation
Dazhao Du, Shiyan Du, Jian Liu et al. (+8 more)
Understanding camera motion is fundamental to
video perception, with applications in spatial intelligence and controllable
video generation.
Multimodal large
language models (MLLMs) provide a natural interface for this task, but existing work typically assigns one or more labels to an entire clip. Such clip-level recognition overlooks two defining properties of real camera motion: it can change within a shot, and multiple movements can occur simultaneously. We therefore formulate camera-motion understanding as temporally grounded, compositional recognition, which requires a model to localize motion-consistent intervals and identify every movement active within each interval. We introduce CamChoreo, a
benchmark of 4,229 real single-shot clips with expert-annotated temporal segments. Its annotations use a compact vocabulary of 20 direction-aware labels, and nearly half of the segments contain compound camera motion, with multiple movement primitives active simultaneously. Recognizing such fine-grained, compositional motion is hard for current MLLMs, whose visual encoders emphasize semantic content rather than the
geometric evidence on which camera motion depends. Directly injecting features from a frozen 3D foundation model addresses this gap, but requires running the expensive geometry model on every input; we refer to this baseline as CamInject. We instead propose CamDistill, which distills the same
geometric knowledge into lightweight camera
tokens during training and removes the 3D model at inference. CamDistill matches the accuracy of direct feature injection without running the 3D teacher at inference. Together, CamChoreo and CamDistill advance camera-motion understanding from clip-level labeling to temporally grounded, compositional recognition. Project page: https://ddz16.github.io/cammotion.github.io/.
Read the full abstract ▾
All authors Dazhao Du, Shiyan Du, Jian Liu, Yongjian Yu, Bohai Gu, Tao Han, Hualuo Liu, Eric Liu, Yujia Zhang, Xi Chen, Song Guo
Computer Vision
Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences
Martina Ianaro, Guilherme Fernandes, Maurizio Gabbrielli et al. (+1 more)
As generative multimedia evolves from static image synthesis to complex, interleaved visual narratives, a foundational bottleneck has emerged: the judgment crisis. While human perception naturally synthesizes the temporal and logical flow of a story, automated evaluation systems remain largely "blind" to sequential continuity, often failing to distinguish between a coherent narrative and a semantically shuffled or contradictory sequence. This work identifies a critical structural gap in current
multimodal evaluation paradigms, arguing that the reliance on Large
Vision-
Language Models (LVLMs) as judges is fundamentally limited by architectural biases. Our analysis reveals a profound performance dichotomy: while models may appear competent in isolated pointwise scoring, they suffer a catastrophic collapse when required to perform pairwise discrimination of temporal order. We demonstrate that this is not merely a data-scarcity issue but a structural one. Through a series of diagnostic probes, we uncover systematic positional asymmetries, specifically primacy and recency effects, where a model's judgment of a story is significantly influenced by the placement of a frame, often more than by its semantic consistency. These biases, potentially rooted in causal masking and rotary
embeddings, suggest that current
transformer-based judges are inherently ill-equipped for long-form visual
reasoning. By exposing these blind spots, we challenge the multimedia community to move beyond snapshot-centric metrics and instead pioneer Temporally-Aware Evaluation paradigms that treat visual sequences as unified logical structures rather than unordered collections of frames.
Read the full abstract ▾
All authors Martina Ianaro, Guilherme Fernandes, Maurizio Gabbrielli, Joao Magalhaes
Computer Vision
VIDS-Seg: Towards Reliable Uncertainty Quantification in Pediatric Cardiac Ultrasound Segmentation
Paul Fischer and Ece Ozkan
Reliable clinical deployment of machine learning requires models that know when they are likely to fail, particularly for subgroups underrepresented in training data. A common case is pediatric care, where models trained on adult cohorts can silently under-perform on children with no indication that something has gone wrong. As retraining with labeled pediatric data is often infeasible, detecting such failures at inference time is a critical clinical need. Building on the VIDS (Variational Inference under Distribution Shifts) framework, we introduce VIDS-Seg, which applies amortized variational inference over a lightweight prediction head to make this adaptive, OOD-aware prior tractable for dense image
segmentation. We evaluate VIDS-Seg on left ventricular
segmentation in echocardiography, a setting where pediatric anatomy differs systematically from the adult population most
segmentation models are trained on, training on an adult cohort (EchoNet-Dynamic) and evaluating zero-shot on a pediatric cohort (EchoNet-Pediatric). Across all age strata, VIDS-Seg matches competitive baselines in
segmentation accuracy while producing substantially higher spatial correspondence between predicted uncertainty and
segmentation error, an advantage that persists even after applying temperature scaling to all baselines. Downstream, it yields more accurate and stable ejection fraction estimates and more reliable detection of cardiac malfunction in the infant subgroup. Our results indicate that OOD-aware uncertainty quantification can serve as a practical safety layer for deployed
segmentation models, enabling detection of silent failures in underrepresented subgroups without retraining or additional labeled data.
Read the full abstract ▾
Computer Vision
Sensor-Informed Per-Point Covariance for Structured-Light 3D Imaging
Sehoon Tak and Jae-Sang Hyun
Per-point uncertainty models are important in structured-light 3D reconstruction for probabilistic registration, fusion, and quality assessment. In practice, however, point-cloud covariances are often modeled as isotropic constants or inferred from local surface geometry and therefore do not explicitly reflect the measurement process. This is a limitation in fringe projection profilometry (FPP), where phase noise propagates through calibrated reconstruction and produces strongly anisotropic 3D uncertainty. This paper presents a sensor-informed first-order method for constructing a per-point 3 x 3 covariance field from experimentally measured phase precision and calibrated phase-to-depth and phase-to-3D mappings. The formulation separates a rank-1 phase-induced covariance from an effective full-rank completion obtained by incorporating fitted lateral image-space perturbation scales. Repeated-plane experiments under fixed imaging conditions show close
alignment of the dominant covariance direction with the viewing ray, and consistency between the dominant phase-induced uncertainty scale and scalar depth uncertainty. In G-ICP registration, the proposed covariance substantially improves over a constant isotropic model while providing a sensor-derived uncertainty representation complementary to conventional geometry-based covariances.
Read the full abstract ▾
Computer Vision
GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenes
Ermanno Bartoli, Buwei He, Dennis Rotondi et al. (+6 more)
Robots operating in human environments need
memories that capture not only what objects exist and where, but also how people use them over time and how individual interactions compose into goal-directed activities. Existing 4D scene graphs preserve object and place histories but omit activity structure, whereas activity representations are either not grounded in persistent 3D scenes or rely on externally provided event boundaries and object associations. We present GESTO (Grounded Event and Spatio-Temporal
memOry), a spatio-temporal
memory that couples a persistent 4D scene graph with a two-level hierarchy of atomic human–object interactions and goal-driven events. From an RGB-D observation stream, GESTO automatically extracts timestamped interactions, grounds them to persistent scene entities, groups them into events, and uses event context to refine uncertain object associations. A relation-aware tool-calling
agent queries the resulting
memory for activity-centric spatio-temporal
reasoning. We evaluate GESTO on the reproducible text, binary, and time categories of an existing
benchmark, together with 40 new Space2Event and Event2Space queries. GESTO achieves scores of 0.71, 0.75, and 0.70 on the standard categories, approaching a method supplied with ground-truth event and object grounding, while substantially outperforming the same
reasoning framework when these inputs are removed. It further achieves 0.73 and 0.75 on Space2Event and Event2Space queries. Ablations show that hierarchical event structure and context-aware grounding refinement provide complementary benefits, supporting activity-grounded hierarchical
memory for retrospective
reasoning in dynamic human environments.
Read the full abstract ▾
All authors Ermanno Bartoli, Buwei He, Dennis Rotondi, Sebastian Koch, Federico Tombari, Kai O. Arras, Patric Jensfelt, Yixi Cai, Iolanda Leite
Computer Vision
ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral
Md Rabiul Islam, Samir Abdaljalil, Erchin Serpedin et al. (+1 more)
Pulmonary nodule malignancy prediction typically depends on image-trained specialist deep learning (DL) models that require substantial annotated imaging data and task-specific training. We investigate whether a generalist large
language model (LLM), reading only a faithful natural-
language rendering of standard nodule attributes, can serve as a calibrated triage layer. We propose ConfTriage, a confidence-calibrated method built on three pillars:
language as the modality, calibration as the safety mechanism, and a selective specialist DL backstop for low-confidence cases. We prove two guarantees: a finite-sample combined-error bound yielding an explicit per-threshold operational certificate, and an oracle inequality showing that excess risk over the Bayes-optimal deferral classifier is controlled by the L1 calibration error of the LLM probability. A controlled seven-way input ablation across five frontier LLMs on LIDC-IDRI shows that natural-
language descriptions dominate the diagnostic signal, while low-level image statistics are essentially diagnostically vacuous. ConfTriage achieved an F1 score of 88.22% and an AUC of 0.92, resolving 76.5% of cases using zero-shot LLM inference alone and referring only uncertain cases to the specialist DL backstop. These results demonstrate that clinically meaningful diagnostic information can be captured through structured radiological descriptions and leveraged by calibrated LLMs for selective referral. The framework suggests a practical pathway for combining generalist LLM prediction with specialist AI models in medical decision-support systems. Source code is publicly available at https://github.com/rabiul-ai/ConfTriage.
Read the full abstract ▾
All authors Md Rabiul Islam, Samir Abdaljalil, Erchin Serpedin, Hasan Kurban
Computer Vision
NullEdit: Stealthy Image Protection via VLM Condition Redirection
Weiyao Huang, Liqin Wang, Ziqi Sheng et al. (+1 more)
Modern image editors combine
vision-
language models (VLMs) with
diffusion transformer backbones to modify a single reference image according to instructions without fine-tuning. This capability also enables unauthorized manipulation of publicly released images. Existing inference-time defenses either invalidate edits through conspicuous corruption, thereby exposing the protection, or allow them to proceed with identity or reference content drift, thereby failing to prevent the editing behavior itself. We instead target a stealthy and harmless no-op in which the requested edit is suppressed, the output remains natural and source-preserving without conspicuous artifacts or identity replacement, and harmful semantics requested by malicious instructions are absent. We propose NullEdit, which targets the VLM representation jointly formed from the reference image and instruction before it conditions the downstream DiT backbone. Using normal-edit and no-edit anchors, NullEdit redirects this representation, while cross-
prompt gradient averaging transfers protection to held out instructions. Across Step1X-Edit and Qwen-Image-Edit on CelebA-HQ and VGGFace2, NullEdit reduces the EditReward IF score by 0.813 on average relative to the SOTA baseline while preserving subject identity and source content.
Read the full abstract ▾
All authors Weiyao Huang, Liqin Wang, Ziqi Sheng, Wei Lu
Computer Vision
Kiet T. Nguyen, Hanbo Shim, Jinwoo Kim et al. (+1 more)
Vision-
language models (VLMs) have achieved strong image and
video understanding, yet their visual-spatial representations remain geometrically fragile, leading to failures in spatial
reasoning needed for
embodied AI, robotics, and autonomous driving. Prior approaches to geometry grounding either fine-tune VLMs on spatial question answering, which can perpetuate spurious visual representations, or fuse features from large geometry-grounded
vision models, which substantially increases model size at inference. Knowledge
distillation from geometry-grounded
vision models offers an alternative, but directly matching multi-view teacher features can disrupt the pretrained
alignment between visual and textual representations, degrading object- and
language-semantic capabilities. We propose multi-view relational
distillation (MVRD), which distills patch-wise cosine similarities across views instead of the teacher features themselves. These relations encode
geometric correspondences adequate for spatial understanding, while leaving the student representation underdetermined, allowing it to remain close to its pretrained
vision-
language space. Across representative VLMs, MVRD improves visual-spatial
reasoning, outperforming supervised fine-tuning and feature
distillation while approaching feature fusion methods with considerably fewer added parameters and lower latency. We show that MVRD makes visual representations more
geometric while retaining
language alignment, and generalizes to 3D scene understanding tasks such as object grounding, dense captioning, and question answering.
Read the full abstract ▾
All authors Kiet T. Nguyen, Hanbo Shim, Jinwoo Kim, Seunghoon Hong
Computer Vision · Cross-listed from cs.RO
Flex-$\pi$: A Multi-Stream World-Action Model with Compute Flexibility
Ge Yan, Jinghao Liu, Yuzhi Fan et al. (+4 more)
World-action models (WAMs) predict the future to act better, but nearly all of them predict only RGB latents, trained purely for pixel reconstruction, with no explicit signal for the 3D geometry or object semantics manipulation needs. We find a surprising free lunch: the same frozen
video-generation VAE that encodes RGB also encodes 3D pointmaps almost losslessly, with no pointmap-specific training at all. This lets us supervise Flex-$\pi$, a 6B-parameter WAM, on 3D geometry and object-centric DINO semantics alongside RGB, at no cost in new sensors, new pre-training, or inference latency. Every visual signal is projected into this shared latent space and denoised jointly with actions inside a Mixture-of-
Transformers backbone; per-stream dropout with cross-modality forcing then lets a single trained checkpoint run on any subset of these streams, from a fast action-only mode to full joint generation. The result is a policy that is exceptionally demonstration-efficient and generalizes well, beating the strongest baselines by up to 2-7$\times$ on dexterous, precise, real-world bimanual manipulation tasks both in and out of distribution, all while running faster than $\pi_{0.5}$. Our project website: https://flex-pi.github.io/
Read the full abstract ▾
All authors Ge Yan, Jinghao Liu, Yuzhi Fan, Lei Cai, Minwen Liao, Jesse Zhang, Dieter Fox
Computer Vision
The GENEA Challenge 2026: A Large-Scale Disentangled Evaluation of Speech-Driven Gesture Generation on the Seamless Interaction Dataset
Rajmund Nagy, Silvia Arellano García, Hendric Voss et al. (+4 more)
This preprint presents the results of the fourth GENEA Challenge, a large-scale human evaluation of five
speech-driven gesture-generation systems trained by participating teams on the Seamless Interaction
dataset of dyadic conversations. As in the 2023 GENEA Challenge, we used a disentangled evaluation methodology to assess motion quality and
speech alignment without confounding between the two, and performed a dyadic mismatching study to isolate the effect of listening and reacting to the interlocutor. We additionally introduce a new semantic gesture-generation task and a text-mismatching evaluation methodology using the Grounded Gestures subset of the data. In total, we ran four large-scale user studies, collecting over 23,000 votes from 869 test-takers. In the motion-realism study, the
dataset's filtered segments had substantially higher motion quality than all challenge submissions (68-95% pairwise winrate). In the
speech-
alignment study, the motion-capture segments provided a conceptual ceiling at 62%
alignment score, with the top submission significantly behind at 32% and the rest only slightly above the 0% expected of an input-independent system. In the dyadic study, motion capture again set the ceiling at 65% appropriateness score, but no submission scored substantially above chance, indicating that the systems could not yet respond to the interlocutor. Finally, the semantic mismatching evaluation found highly expressive gestures in the
dataset (test-takers identified the matching transcript 79% of the time), yet almost all submissions failed to generate semantically expressive motion, with the best achieving only an 8% appropriateness score. The collected votes and outputs will be made publicly available at https://genea-workshop.github.io/2026/challenge/ to facilitate reproducibility and further research.
Read the full abstract ▾
All authors Rajmund Nagy, Silvia Arellano García, Hendric Voss, Mihail Tsakov, Taras Kucherenko, Youngwoo Yoon, Gustav Eje Henter
Computer Vision
PolyLayout: Hierarchical VLM-Guided Layout Generation Beyond Rectangular Rooms
Yutong Jiang, Zahra Atashgahi, Carlos Soto Garcia Delgado et al. (+4 more)
Generating physically plausible 3D room layouts is essential for home furnishing retail, enabling customers to visualize products in their own homes and confidently make purchasing decisions. However, a gap exists between academic research and real-world application: existing solutions primarily focus on algorithmic strategies for furniture placement, largely neglecting the non-rectangular geometries and strict door/window constraints prevalent in real homes. To bridge the gap, we introduce a hybrid, hierarchical framework tailored for retail, specifically designed to support scalable spatial planning applications. Our system decouples generation into three stages: (1) functional furniture clustering and fine-grained intra-zone placement; (2) macro-routing guided by a
vision-
language model (VLM) to anchor both these clustered zones and any remaining standalone furniture within diverse polygonal boundaries; and (3) rule-based optimization for collision-free micro-arrangements that respect architectural constraints. We evaluate our system on production-scale catalogs and a representative set of irregular real-world
topologies. Our results show that our approach attains the highest perceptual plausibility while maintaining good
geometric compliance at relatively low latency, and extends to irregular boundaries that existing methods do not natively support.
Read the full abstract ▾
All authors Yutong Jiang, Zahra Atashgahi, Carlos Soto Garcia Delgado, Ruben Brokkelkamp, Davide Zanutto, Efşan Sökmen, Shahin Shahkarami
Computer Vision
UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations
Dvir Samuel, Guy Bar-Shalom, Fabrizio Frasca et al. (+4 more)
Large
Vision-
Language Models (LVLMs) achieve impressive visual
reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input. Effective mitigation requires
token-level localization, enabling targeted intervention without discarding the entire response. Existing detectors require expensive full-model fine-tuning, rely on external verifiers that ignore the model's generation process, or reduce internal signals to isolated features and hand-crafted statistics, discarding spatial, sequential, and relational structure. We introduce UniProbe, a lightweight, unified, learnable detector that models a frozen LVLM's heterogeneous computational trace from a single forward pass. UniProbe constructs a directed graph over image patches, query
tokens, and generated
tokens, with
attention weights encoding their relations. It processes this trace with alternating structure-aware modules: a GNN for relational evidence, a ViT for 2-D visual geometry, and a GRU for response order. Interleaving them allows spatial, relational, and sequential evidence to interact throughout the detector. We further develop a streaming variant for
hallucination-aware decoding, which detects and resamples hallucinated
tokens during generation, and a self-adaptation strategy aligning the detector with the LVLM's own generations. Across diverse LVLM backbones, UniProbe achieves state-of-the-art
token-level and object-
hallucination detection. During decoding, it reduces object
hallucinations by up to 55% at $1.06\times$ the latency of standard generation.
Read the full abstract ▾
All authors Dvir Samuel, Guy Bar-Shalom, Fabrizio Frasca, Ethan Fetaya, Yftah Ziser, Gal Chechik, Haggai Maron
Computer Vision
MIRA: Medical Image Reflection for Agentic Diagnosis
Shengzhi Wang, Jun Yang, Kai Wu et al. (+11 more)
Medical visual
agents can use tools to inspect images and retrieve external knowledge, but indiscriminate tool use may introduce noisy or misleading evidence. Reliable diagnosis therefore requires not only acquiring additional observations, but also verifying whether tool actions are necessary and whether the resulting evidence supports the current hypothesis. We introduce MIRA (Medical Image Reflection for Agentic Diagnosis), a medical visual diagnostic framework for autonomous evidence search and reflective verification. MIRA dynamically invokes image-processing operations, including zooming, grounding, pointing, rotation, and measurement, as well as web search, while evaluating the relevance and consistency of the acquired evidence. We develop MIRA through a two-stage training strategy. First, a tool-augmented Monte Carlo Tree Search data engine explores diverse diagnostic hypotheses and jointly verifies visual grounding accuracy and semantic consistency to construct supervised fine-tuning trajectories. Second, reinforcement learning further improves decision-making through online reflective principle
evolution: failure cases are distilled into candidate principles, and only principles that improve held-out rollout rewards are retained. Across nine medical visual
reasoning benchmarks, MIRA achieves an average score of 64.73, improving its Qwen3-VL-8B backbone by 7.44 points. It also increases useful tool-use judgments from 56.2% to 73.8% and reduces harmful judgments from 8.9% to 1.6%. Qualitative analyses show that MIRA can re-examine evidence, correct premature conclusions, and adapt its tool-use strategy. Project page: https://MIRA-VL.github.io/
Read the full abstract ▾
All authors Shengzhi Wang, Jun Yang, Kai Wu, Xiaozhong Ji, Yiwen Ye, Ziyang Chen, Mingliang Xiong, Wen Fang, Mingqing Liu, Mengyuan Xu, Miaoxuan Shan, Caiyan Liu, Bin He, Qingwen Liu
Computer Vision · Cross-listed from cs.RO
Neural Introspection Gating for Adaptive KV-Cache Reuse in Vision-Language-Action Models
Zhijie Wu, Kento Kawaharazuka and Kei Okada
Vision-
Language-Action(VLA) models map camera images and
language instructions directly to motor commands through a single autoregressive
transformer. In real-time control, they still spend substantial compute recomputing key-value(KV) representations for visual
tokens that barely change across neighboring frames. Recent work such as VLA-Cache reduces that cost by reusing KV states for visually static patches, but its policy relies only on observation-space heuristics and does not account for the model's own uncertainty. We propose Gated VLA-Cache, a lightweight, training-free extension that augments visual-similarity caching with neural introspection. The method monitors the logit margin between the top two predicted action
tokens, a zero-cost confidence signal available during decoding. When the margin drops below a threshold, the cache is invalidated and a full recompute is triggered. Evaluated on four LIBERO
benchmark suites with both OpenVLA and OpenVLA-OFT, Gated VLA-Cache improves reliability when blind caching hurts. On LIBERO-Goal and LIBERO-Long, it recovers over 100% of the lost accuracy while retaining 80% of the compute savings.
Read the full abstract ▾
Computer Vision
Modelling Geographic Atrophy Progression using Implicit Neural Representations
Simone Sarrocco, Paul Friedrich, Florentin Bieder et al. (+4 more)
Age-related Macular Degeneration (AMD) is the major cause of blindness in the Western world. Its late dry phase is characterised by irreversible atrophic areas, namely Geographic Atrophy (GA). Longitudinal Fundus Autofluorescence (FAF) image acquisitions are currently the main tool for assessing lesion growth over time at the image level. However, due to its highly individualised progression, the
evolution of late AMD remains poorly understood. In this work, we propose using Implicit Neural Representations (INRs) to model GA progression at the individual level in a low-data setting. Our approach generates both FAF and GA
segmentation at both past and future time points. Among the comparison models, our method achieves competitive
segmentation quality across different scenarios, yielding the lowest Mean Absolute Error (MAE) for the GA lesion area and the highest DICE score, without sacrificing FAF image quality. The code is available at https://github.com/SimoneSarrocco/ga-progression-with-inrs.
Read the full abstract ▾
All authors Simone Sarrocco, Paul Friedrich, Florentin Bieder, Christina Bornberg, Philippe Valmaggia, Peter Maloca, Philippe Cattin
Computer Vision
Fast and Memory-Efficient Wavelet Convolutions via I/O-Aware Reformulation
Amit Aflalo, Shahaf E. Finder, Roy Amoyal et al. (+2 more)
Wavelet convolution (WTConv) has emerged as an increasingly popular drop-in replacement for standard convolutions, expanding a network's receptive field exponentially with the number of decomposition levels while keeping the parameter count linear. However, its reference implementation is severely
memory-bound due to excessive data movement through high-bandwidth
memory (HBM). We develop an I/O model of WTConv to characterize this bottleneck and use it to guide three algebraic reformulations: (1) recomputing the inexpensive Haar analysis butterfly on chip, (2) collapsing the multi-level synthesis cascade into a single closed-form pass indexed by output-coordinate bits, and (3) folding learned per-channel scales into the convolution weights. Together, these reformulations enable an I/O-aware fused implementation that substantially reduces HBM traffic. We evaluate the WTConvNeXt configuration across decomposition levels and a broad range of tensor shapes. Despite performing comparable arithmetic, the reference WTConv is substantially slower than the depthwise convolution it replaces. Our reformulation reduces modeled HBM traffic by approximately $2.55\times$, yielding up to a $4.35\times$ training speedup over the reference while roughly halving peak
memory usage. Thus, our reformulation preserves the benefits of WTConv while substantially reducing its execution time and
memory footprint, removing the systems overhead that previously limited its practical efficiency.
Read the full abstract ▾
All authors Amit Aflalo, Shahaf E. Finder, Roy Amoyal, Eran Treister, Oren Freifeld
Computer Vision
BPG: Balancing Plasticity and Generalization for Domain Incremental Learning
Qiang Wang, Songlin Dong, Shaokun Wang et al. (+5 more)
Deep neural networks excel in various tasks but struggle to generalize across evolving data distributions, leading to significant performance degradation under domain shifts. Domain incremental learning (DIL) addresses this challenge by enabling models to continuously adapt while retaining prior knowledge. Among existing DIL approaches, the parameter-isolation paradigm achieves state-of-the-art performance. However, these methods often adopt a one-size-fits-all approach to adapt to new domains, resulting in either insufficient learning capacity or redundant parameters. In this work, we propose BPG, a unified framework that addresses both challenges through two complementary components: BPG-Adapter, which dynamically determines each domain's adapter hidden dimension based on domain-specific feature separability, and BPG-Inference, a soft domain mixture strategy that integrates multiple domain-specific models at test time, mitigating domain ID misselection. Experimental results on DomainNet, CDDB, and CORe50 demonstrate that BPG consistently outperforms uniform adapter-based approaches and hard domain selection strategies, achieving state-of-the-art average accuracy while reducing forgetting to as low as 0.22% on DomainNet.
Read the full abstract ▾
All authors Qiang Wang, Songlin Dong, Shaokun Wang, Jizhou Han, Xiang Song, Chenhao Ding, Yuhang He, Yihong Gong
Computer Vision
Evaluating Semantic and Spatial Guidance for Foundation Model Segmentation of Small-Scale PV in Remote Sensing Imagery
Roni Blushtein-Livnon, Tal Svoray, Osher Rafaeli et al. (+4 more)
Spatio-temporal PV data are essential for understanding adoption processes in off-grid regions, yet such data remain largely unavailable. Automated
segmentation of remote sensing (RS) imagery offers a promising solution; yet, residential PV systems remain challenging targets because of their small size and sparse distribution, resulting in severe target-background imbalance.
Vision-
language foundation models (FMs) provide a data-efficient paradigm through
prompt-based semantic and spatial guidance, but the relative contribution of different
prompt types remains unclear. We systematically evaluate SAM3 for small-scale PV
segmentation in RS imagery by comparing textual,
geometric, and hybrid prompting, under varying supervision levels, training strategies, spatial resolutions, and imaging conditions. Multi-temporal aerial imagery from a large off-grid rural region serves as a study site, with findings validated across three additional
datasets. Prompting strategy emerged as the dominant factor governing model behavior. Textual prompting consistently produced the lowest performance and showed the greatest sensitivity to supervision and imaging conditions. In contrast, spatial guidance substantially improved both
segmentation accuracy and robustness. Hybrid prompting achieved the highest accuracy and stability, indicating that semantic and spatial guidance provide complementary information. Most performance gains were achieved with only a few hundred annotated samples, demonstrating strong data efficiency. Transfer learning had limited overall impact, with only modest improvements observed for textual prompting under limited supervision. Overall, our findings establish prompting strategy as a key determinant of SAM3 adaptation, robustness, and generalization, highlighting the potential of promptable FMs for scalable PV mapping in data-constrained off-grid regions.
Read the full abstract ▾
All authors Roni Blushtein-Livnon, Tal Svoray, Osher Rafaeli, Michael Dorman, Itay Fischhendler, Havazelet Yahel, Emir Galilee