Computer Vision
Storing Less, Finding More: How Novelty Filtering Improves Cross-Modal Retrieval on Edge Cameras
Sherif Abdelwahab
Always-on edge cameras generate continuous
video streams where redundant frames degrade cross-modal
retrieval by crowding correct results out of top-k search. This paper presents a streaming
retrieval architecture: an on-device epsilon-net filter retains only semantically novel frames, building a denoised
embedding index; a cross-modal adapter and cloud re-ranker compensate for the compact encoder's weak
alignment. A single-pass streaming filter outperforms offline alternatives (k-means, farthest-point, uniform, random) across eight
vision-
language models (8M-632M) on two
egocentric datasets (AEA, EPIC-KITCHENS). Combined, the architecture reaches 45.6% Hit@5 on held-out data using an 8M on-device encoder at an estimated 2.7 mW.
Read the full abstract ▾
Computer Vision
BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation
Johann-Ludwig Herzog, Mathis Jürgen Adler, Leonard Hackel et al. (+5 more)
Vision-langugage models (VLMs) have shown strong performance in computer
vision (CV), yet their performance on remote sensing (RS) data remains limited due to the lack of large-scale, multi-sensor RS image-text
datasets with diverse textual annotations. Existing
datasets predominantly include aerial Red-Green-Blue imagery, with short or weakly grounded captions, and provide limited diversity in annotation types. To address this limitation, we introduce BigEarthNet$.$txt, a large-scale, multi-sensor image-text
dataset designed to advance instruction-driven image-text learning in Earth observation across multiple tasks. It contains 464044 co-registered Sentinel-1 synthetic aperture radar and Sentinel-2 multispectral images with 9.6M text annotations, including: i) geographically anchored captions describing land-use/land-cover (LULC) classes, their spatial relations, and environmental context; ii) visual question answering pairs relevant for different tasks; and iii) referring expression detection instructions for bounding box prediction. A comparative statistical analysis shows that BigEarthNet$.$txt surpasses existing RS image-text
datasets in textual richness and annotation type variety. We further establish a manually-verified
benchmark split and extensively evaluate existing VLMs in RS and CV. The results show the limitations of these models on tasks that involve complex LULC classes, whereas fine-tuning using BigEarthNet$.$txt results in consistent performance gains across all considered tasks. Our code and data are publicly available at https://txt.bigearth.net.
Read the full abstract ▾
All authors Johann-Ludwig Herzog, Mathis Jürgen Adler, Leonard Hackel, Yan Shu, Angelos Zavras, Ioannis Papoutsis, Paolo Rota, Begüm Demir
Computer Vision
Unify-Agent: A Unified Multimodal Agent for World-Grounded Image Synthesis
Shuang Chen, Quanxin Shou, Hangting Chen et al. (+16 more)
Unified
multimodal models provide a natural and promising architecture for understanding diverse and complex real-world knowledge while generating high-quality images. However, they still rely primarily on frozen parametric knowledge, which makes them struggle with real-world image generation involving long-tail and knowledge-intensive concepts. Inspired by the broad success of
agents on real-world tasks, we explore agentic modeling to address this limitation. Specifically, we present Unify-
Agent, a unified
multimodal agent for world-grounded image synthesis, which reframes image generation as an agentic pipeline consisting of
prompt understanding,
multimodal evidence searching, grounded recaptioning, and final synthesis. To train our model, we construct a tailored
multimodal data pipeline and curate 143K high-quality
agent trajectories for world-grounded image synthesis, enabling effective supervision over the full agentic generation process. We further introduce FactIP, a
benchmark covering 12 categories of culturally significant and long-tail factual concepts that explicitly requires external knowledge grounding. Extensive experiments show that our proposed Unify-
Agent substantially improves over its base unified model across diverse
benchmarks and real world generation tasks, while approaching the world knowledge capabilities of the strongest closed-source models. As an early exploration of
agent-based modeling for world-grounded image synthesis, our work highlights the value of tightly coupling
reasoning, searching, and generation for reliable open-world agentic image synthesis.
Read the full abstract ▾
All authors Shuang Chen, Quanxin Shou, Hangting Chen, Yucheng Zhou, Kaituo Feng, Wenbo Hu, Yi-Fan Zhang, Yunlong Lin, Wenxuan Huang, Mingyang Song, Dasen Dai, Bolin Jiang, Manyuan Zhang, Shi-Xue Zhang, Zhengkai Jiang, Lucas Wang, Zhao Zhong, Yu Cheng, Nanyun Peng
Computer Vision
Video-Oasis: Rethinking Evaluation of Video Understanding
Geuntaek Lim, Sungjune Park, Jaeyun Lee et al. (+5 more)
The inherent complexity of
video understanding makes it difficult to determine whether
Video-LLM
benchmark performance stems from visual perception, linguistic
reasoning, or knowledge priors. While many
benchmarks have emerged to assess high-level
reasoning, shared criteria for evaluating
video understanding remain largely overlooked. Instead of introducing yet another
benchmark, we take a step back to re-examine the criteria for evaluating
video understanding. In this work, we introduce
Video-Oasis, a sustainable diagnostic suite for systematically
auditing existing
video understanding
benchmarks. This audit reveals that 55% of existing
benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining
video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust
video understanding. We hope our work provides a practical foundation for constructing rigorous
video benchmarks and evaluating future
Video-LLMs. Code is available at https://github.com/sejong-rcv/
Video-Oasis.
Read the full abstract ▾
All authors Geuntaek Lim, Sungjune Park, Jaeyun Lee, Inwoong Lee, Taeoh Kim, Dongyoon Wee, Minho Shim, Yukyung Choi
Computer Vision
FlowID : Enhancing Forensic Identification with Latent Flow-Matching Models
Jules Ripoll, David Bertoin, Alasdair Newson et al. (+2 more)
Every day, many people die under violent circumstances, whether from crimes, war, migration, or
climate disasters. Medico-legal and law enforcement institutions document many portraits of the deceased for evidence, but cannot immediately carry out identification on them. While traditional image editing tools can process these photos for public release, the workflow is lengthy and produces suboptimal results. In this work, we leverage advances in image generation models, which can now produce photorealistic human portraits, to introduce FlowID, an identity-preserving facial reconstruction method. Our approach combines single-image fine-tuning, which adapts the generative model to out-of-distribution injured faces, with
attention-based masking that localizes edits to damaged regions while preserving identity-critical features. Together, these components enable the removal of artifacts from violent death while retaining sufficient identity information to support identification. To evaluate our method, we introduce InjuredFaces, a novel
benchmark for identity-preserving facial reconstruction under severe facial damage. Beyond serving as an evaluation tool for this work, InjuredFaces provides a standardized resource for the community to study and compare methods addressing facial reconstruction in extreme conditions. Experimental results show that FlowID outperforms state-of-the-art open-source methods while maintaining low
memory requirements, making it suitable for local deployment without compromising data
privacy.
Read the full abstract ▾
All authors Jules Ripoll, David Bertoin, Alasdair Newson, Charles Dossal, Jose Pablo Baraybar
Computer Vision
Emotion Diffusion Classifier with Adaptive Margin Discrepancy Training for Facial Expression Recognition
Rongkang Dong, Cuixin Yang, Cong Zhang et al. (+2 more)
Facial Expression Recognition (FER) is essential for human-machine interaction, as it enables machines to interpret human
emotions and internal states from facial affective behaviors. Although deep learning has significantly advanced FER performance, most existing deep-learning-based FER methods rely heavily on discriminative classifiers for fast predictions. These models tend to learn shortcuts and are vulnerable to even minor distribution shifts. To address this issue, we adopt a conditional generative
diffusion model and introduce the
Emotion Diffusion Classifier (EmoDC) for FER, which demonstrates enhanced
adversarial robustness. However, retraining EmoDC using standard strategies fails to penalize incorrect categorical descriptions, leading to suboptimal recognition performance. To improve EmoDC, we propose margin-based discrepancy training, which encourages accurate predictions when conditioned on correct categorical descriptions and penalizes predictions conditioned on mismatched ones. This method enforces a minimum margin between noise-prediction errors for correct and incorrect categories, thereby enhancing the model's discriminative capability. Nevertheless, using a fixed margin fails to account for the varying difficulty of noise prediction across different images, limiting its effectiveness. To overcome this limitation, we propose Adaptive Margin Discrepancy Training (AMDiT), which dynamically adjusts the margin for each sample. Extensive experiments show that AMDiT significantly improves the accuracy of EmoDC over the baseline model with standard denoising
diffusion training under 100-step evaluations. Additionally, AMDiT-enhanced EmoDC has better generalization and robustness than state-of-the-art discriminative classifiers.
Read the full abstract ▾
All authors Rongkang Dong, Cuixin Yang, Cong Zhang, Yushen Zuo, Kin-Man Lam
Computer Vision
Generating Key Postures of Bharatanatyam Adavus with Pose Estimation
Jagadish Kashinath Kamble, Jayanta Mukhopadhyay, Debaditya Roy et al. (+1 more)
Preserving intangible cultural dances rooted in centuries of tradition and governed by strict structural and symbolic rules presents unique challenges in the digital era. Among these, Bharatanatyam, a classical Indian dance form, stands out for its emphasis on codified adavus and precise key postures. Accurately generating these postures is crucial not only for maintaining anatomical and stylistic integrity, but also for enabling effective documentation, analysis, and transmission to broader global audiences through digital means. We propose a pose-aware generative framework integrated with a pose estimation module, guided by keypoint-based loss and pose consistency constraints. These supervisory signals ensure anatomical accuracy and stylistic integrity in the synthesized outputs. We evaluate four configurations: standard conditional generative
adversarial network (cGAN), cGAN with pose supervision, conditional
diffusion, and conditional
diffusion with pose supervision. Each model is conditioned on key posture class labels and optimized to maintain
geometric structure. In both cGAN and conditional
diffusion settings, the integrated pose guidance aligns generated poses with ground-truth keypoint structures, promoting cultural fidelity. Our results demonstrate that incorporating pose supervision significantly enhances the quality, realism, and authenticity of generated Bharatanatyam postures. This framework provides a scalable approach for the digital preservation, education, and dissemination of traditional dance forms, enabling high-fidelity generation without compromising cultural precision. Code is available at https://github.com/jagidsh/Generating-Key-Postures-of-Bharatanatyam-Adavus-with-Pose-Estimation.
Read the full abstract ▾
All authors Jagadish Kashinath Kamble, Jayanta Mukhopadhyay, Debaditya Roy, Partha Pratim Das
Computer Vision
Quantization with Unified Adaptive Distillation to enable multi-LoRA based one-for-all Generative Vision Models on edge
Sowmya Vajrala, Aakash Parmar, Prasanna R et al. (+4 more)
Generative Artificial Intelligence (GenAI) features such as image editing, object removal, and
prompt-guided image transformation are increasingly integrated into mobile applications. However, deploying Large
Vision Models (LVMs) for such tasks on resource-constrained devices remains challenging due to their high
memory and compute requirements. While Low-Rank Adapters (LoRAs) enable parameter-efficient task adaptation, existing Mobile deployment pipelines typically compile separate model binaries for each LoRA + a copy of the foundation model, resulting in redundant storage and increased runtime overhead. In this work, we present a unified framework for enabling multi-task GenAI inference on edge devices using a single shared model. Our key idea is to treat LoRA weights as runtime inputs rather than
embedding them into the compiled model graph, allowing dynamic task switching at runtime without recompilation. Then, to support efficient on-device execution, we introduce QUAD (Quantization with Unified Adaptive
Distillation), a quantizationaware training strategy that aligns multiple LoRA adapters under a shared quantization profile. We implement the proposed system with a lightweight runtime stack compatible with mobile NPUs and evaluate it across multiple chipsets. Experimental results demonstrate up to 6x and 4x reduction in
memory footprint and latency improvements, respectively, while maintaining high visual quality across multiple GenAI tasks.
Read the full abstract ▾
All authors Sowmya Vajrala, Aakash Parmar, Prasanna R, Sravanth Kodavanti, Manjunath Arveti, Srinivas Soumitri Miriyala, Ashok Senapati
Computer Vision
Transmittance-Guided Structure-Texture Decomposition for Nighttime Image Dehazing
Francesco Moretti, Giulia Bianchi and Andrea Gallo
Nighttime images captured under hazy conditions suffer from severe quality degradation, including low visibility, color distortion, and reduced contrast, caused by the combined effects of atmospheric scattering, absorption by suspended particles, and non-uniform illumination from artificial light sources. While existing nighttime dehazing methods have achieved partial success, they typically address only a subset of these issues, such as glow suppression or brightness enhancement, without jointly tackling the full spectrum of degradation factors. In this paper, we propose a two-stage nighttime image dehazing framework that integrates transmittance correction with structure-texture layered optimization. In the first stage, we introduce a novel transmittance correction method that establishes boundary-constrained initial transmittance maps and subsequently applies region-adaptive compensation and normalization based on whether image regions correspond to light source areas. A quadratic Gaussian filtering scheme operating in the YUV color space is employed to estimate the spatially varying atmospheric light map. The corrected transmittance map and atmospheric light map are then used in conjunction with an improved nighttime imaging model to produce the initial dehazed image. In the second stage, we propose a
STAR-YUV decomposition model that separates the dehazed image into structure and texture layers within the YUV color space. Gamma correction and MSRCR-based color restoration are applied to the structure layer for illumination compensation and color bias correction, while
Laplacian-of-Gaussian filtering is applied to the texture layer for detail enhancement. A novel two-phase fusion strategy, comprising nonlinear Retinex-based fusion of the enhanced layers followed by linear blending with the initial dehazing result, yields the final output.
Read the full abstract ▾
Computer Vision
All-in-One Augmented Reality Guided Head and Neck Tumor Resection
Yue Yang, Matthieu Chabanas, Carrie Reale et al. (+5 more)
Positive margins are common in head and neck squamous
cell carcinoma, yet intraoperative re-resection is often imprecise because margin locations are typically communicated verbally from pathology. We present an all-in-one augmented reality (AR) system that relocalizes positive margins from a resected specimen to the resection bed and visualizes them in situ using HoloLens 2 depth sensing and fully automated markerless surface registration. In a silicone phantom study with six medical trainees, markerless registration achieved target registration errors comparable to a marker-based baseline (median 1.8 mm vs. 1.7 mm; maximum < 4 mm). In a margin relocalization task, AR guidance reduced error from verbal guidance (median 14.2 mm) to a few millimeters (median 3.2 mm), with all AR localizations within 5 mm error. These results support the feasibility of markerless AR margin guidance for more precise intraoperative re-excision.
Read the full abstract ▾
All authors Yue Yang, Matthieu Chabanas, Carrie Reale, Annie Benson, Jason Slagle, Matthew Weinger, Michael Topf, Jie Ying Wu
Computer Vision
VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference
Anmin Liu, Ruixuan Yang, Huiqiang Jiang et al. (+5 more)
Long-context
video understanding and generation pose a significant computational challenge for
Transformer-based
video models due to the quadratic complexity of self-
attention. While existing sparse
attention methods employ coarse-grained patterns to improve efficiency, they typically incur redundant computation and suboptimal performance. To address this issue, in this paper, we propose VecAttention, a novel framework of vector-wise sparse
attention that achieves superior accuracy-efficiency trade-offs for
video models. We observe that
video attention maps exhibit a strong vertical-vector sparse pattern, and further demonstrate that this vertical-vector pattern offers consistently better accuracy-sparsity trade-offs compared with existing coarse-grained sparse patterns. Based on this observation, VecAttention dynamically selects and processes only informative vertical vectors through a lightweight important-vector selection that minimizes
memory access overhead and an optimized kernel of vector sparse
attention. Comprehensive evaluations on
video understanding (VideoMME, LongVideoBench, and VCRBench) and generation (VBench) tasks show that VecAttention delivers a 2.65$\times$ speedup over full
attention and a 1.83$\times$ speedup over state-of-the-art sparse
attention methods, with comparable accuracy to full
attention. Our code is available at https://github.com/anminliu/VecAttention.
Read the full abstract ▾
All authors Anmin Liu, Ruixuan Yang, Huiqiang Jiang, Bin Lin, Minmin Sun, Yong Li, Chen Zhang, Tao Xie
Computer Vision
Square Superpixel Generation and Representation Learning via Granular Ball Computing
Shuyin Xia, Meng Yang, Dawei Dai et al. (+6 more)
Superpixels provide a compact region-based representation that preserves object boundaries and local structures, and have therefore been widely used in a variety of
vision tasks to reduce computational cost. However, most existing superpixel
algorithms produce irregularly shaped regions, which are not well aligned with regular operators such as convolutions. Consequently, superpixels are often treated as an offline preprocessing step, limiting parallel implementation and hindering end-to-end optimization within deep learning pipelines. Motivated by the adaptive representation and coverage property of granular-ball computing, we develop a square superpixel generation approach. Specifically, we approximate superpixels using multi-scale square blocks to avoid the computational and implementation difficulties induced by irregular shapes, enabling efficient parallel processing and learnable feature extraction. For each block, a purity score is computed based on pixel-intensity similarity, and high-quality blocks are selected accordingly. The resulting square superpixels can be readily integrated as graph nodes in graph neural networks (GNNs) or as
tokens in
Vision Transformers (ViTs), facilitating multi-scale information aggregation and structured visual representation. Experimental results on downstream tasks demonstrate consistent performance improvements, validating the effectiveness of the proposed method.
Read the full abstract ▾
All authors Shuyin Xia, Meng Yang, Dawei Dai, Fan Chen, Shilin Zhao, Junwei Han, Xinbo Gao, Guoyin Wang, Wen Lu
Computer Vision
FedDBP: Enhancing Federated Prototype Learning with Dual-Branch Features and Personalized Global Fusion
Ningzhi Gao, Siquan Huang, Leyu Shi et al. (+1 more)
Federated prototype learning (FPL), as a solution to heterogeneous
federated learning (HFL), effectively alleviates the challenges of data and model heterogeneity.However, existing FPL methods fail to balance the fidelity and discriminability of the feature, and are limited by a single global prototype. In this paper, we propose FedDBP, a novel FPL method to address the above issues. On the client-side, we design a Dual-Branch feature projector that employs L2
alignment and contrastive learning simultaneously, thereby ensuring both the fidelity and discriminability of local features. On the server-side, we introduce a Personalized global prototype fusion approach that leverages Fisher information to identify the important channels of local prototypes. Extensive experiments demonstrate the superiority of FedDBP over ten existing advanced methods.
Read the full abstract ▾
All authors Ningzhi Gao, Siquan Huang, Leyu Shi, Ying Gao
Computer Vision
Few-shot Writer Adaptation via Multimodal In-Context Learning
Tom Simon, Stephane Nicolas, Pierrick Tranouez et al. (+2 more)
While state-of-the-art Handwritten Text Recognition (HTR) models perform well on standard
benchmarks, they frequently struggle with writers exhibiting highly specific styles that are underrepresented in the training data. To handle unseen and atypical writers, writer adaptation techniques personalize HTR models to individual handwriting styles. Leading writer adaptation methods require either offline fine-tuning or parameter updates at inference time, both involving gradient computation and backpropagation, which increase computational costs and demand careful hyperparameter tuning. In this work, we propose a novel context-driven HTR framework3 inspired by
multimodal in-context learning, enabling inference-time writer adaptation using only a few examples from the target writer without any parameter updates. We further demonstrate the impact of context length, design a compact 8M-parameter CNN-
Transformer that enables few-shot in-context adaptation, and show that combining context-driven and standard OCR training strategies leads to complementary improvements. Experiments on IAM and RIMES validate our approach with Character Error Rates of 3.92% and 2.34%, respectively, surpassing all writer-independent HTR models without requiring any parameter updates at inference time.
Read the full abstract ▾
All authors Tom Simon, Stephane Nicolas, Pierrick Tranouez, Clement Chatelain, Thierry Paquet
Computer Vision
NeoNet: An End-to-End 3D MRI-Based Deep Learning Framework for Non-Invasive Prediction of Perineural Invasion via Generation-Driven Classification
Youngung Han, Minkyung Cha, Kyeonghun Kim et al. (+12 more)
Minimizing invasive diagnostic procedures to reduce the risk of patient injury and infection is a central goal in medical imaging. And yet, noninvasive diagnosis of perineural invasion (PNI), a critical prognostic factor involving infiltration of tumor
cells along the surrounding nerve, still remains challenging, due to the lack of clear and consistent imaging criteria criteria for identifying PNI. To address this challenge, we present NeoNet, an integrated end-to-end 3D deep learning framework for PNI prediction in cholangiocarcinoma that does not rely on predefined image features. NeoNet integrates three modules: (1) NeoSeg, utilizing a Tumor-Localized ROI Crop (TLCR)
algorithm; (2) NeoGen, a 3D Latent
Diffusion Model (LDM) with ControlNet, conditioned on anatomical masks to generate synthetic image patches, specifically balancing the
dataset to a 1:1 ratio; and (3) NeoCls, the final prediction module. For NeoCls, we developed the PNI-
Attention Network (PattenNet), which uses the frozen LDM encoder and specialized 3D Dual
Attention Blocks (DAB) designed to detect subtle intensity variations and spatial patterns indicative of PNI. In 5-fold cross-validation, NeoNet outperformed baseline 3D models and achieved the highest performance with a maximum AUC of 0.7903.
Read the full abstract ▾
All authors Youngung Han, Minkyung Cha, Kyeonghun Kim, Induk Um, Myeongbin Sho, Joo Young Bae, Jaewon Jung, Jung Hyeok Park, Seojun Lee, Nam-Joon Kim, Woo Kyoung Jeong, Won Jae Lee, Pa Hong, Ken Ying-Kai Liao, Hyuk-Jae Lee
Computer Vision
EarthEmbeddingExplorer: A Web Application for Cross-Modal Retrieval of Global Satellite Images
Yijie Zheng, Weijie Wu, Zhuoyao He et al. (+5 more)
While the Earth observation community has witnessed a surge in high-impact foundation models and global Earth
embedding datasets, a significant barrier remains in translating these academic assets into freely accessible tools. This tutorial introduces EarthEmbeddingExplorer, an interactive web application designed to bridge this gap, transforming static research artifacts into dynamic, practical workflows for discovery. We will provide a comprehensive hands-on guide to the system, detailing its cloud-native software architecture, demonstrating cross-modal queries (natural
language, visual, and geolocation), and showcasing how to derive scientific insights from
retrieval results. By democratizing access to precomputed Earth
embeddings, this tutorial empowers researchers to seamlessly transition from state-of-the-art models and data archives to real-world application and analysis. The web application is available at https://modelscope.ai/studios/Major-TOM/EarthEmbeddingExplorer.
Read the full abstract ▾
All authors Yijie Zheng, Weijie Wu, Zhuoyao He, Bingyue Wu, Long Zhao, Guoqing Li, Mikolaj Czerkawski, Konstantin Klemmer
Computer Vision · Cross-listed from eess.IV
Polyhedral Unmixing: Bridging Semantic Segmentation with Hyperspectral Unmixing via Polyhedral-Cone Partitioning
Antoine Bottenmuller, Etienne Decencière and Petr Dokládal
Semantic
segmentation and hyperspectral unmixing are two central problems in spectral image analysis. The former assigns each pixel a discrete label corresponding to its material class, whereas the latter estimates pure material spectra, called endmembers, and, for each pixel, a vector representing material abundances in the observed scene. Despite their complementarity, these two problems are usually addressed independently. This paper aims to bridge these two lines of work by formally showing that, under the linear mixing model, pixel classification by dominant materials induces polyhedral-cone regions in the spectral space. We leverage this fundamental property to propose a direct
segmentation-to-unmixing pipeline that performs blind hyperspectral unmixing from any semantic
segmentation by constructing a polyhedral-cone partition of the space that best fits the labeled pixels. Signed distances from pixels to the estimated regions are then computed, linearly transformed via a change of basis in the distance space, and projected onto the probability simplex, yielding an initial abundance estimate. This estimate is used to extract endmembers and recover final abundances via
matrix pseudo-inversion. Because the
segmentation method can be freely chosen, the user gains explicit control over the unmixing process, while the rest of the pipeline remains essentially deterministic and lightweight. Beyond improving interpretability, experiments on three real
datasets demonstrate the effectiveness of the proposed approach when associated with appropriate clustering
algorithms, and show consistent improvements over recent deep and non-deep state-of-the-art methods. The code is available at: https://github.com/antoine-bottenmuller/polyhedral-unmixing
Read the full abstract ▾
Computer Vision
SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answering
Wenli Li, Kai Zhao, Haoran Jiang et al. (+3 more)
Vision-
language models (VLMs) have been widely adopted for 3D question answering (3D QA). In typical pipelines, visual
tokens extracted from multiple viewpoints are concatenated with
language tokens and jointly processed by a large
language model (LLM) for inference. However, aggregating multi-view observations inevitably introduces severe
token redundancy, leading to an overly large visual
token set that significantly hinders inference efficiency under constrained
token budgets. Visual
token pruning has emerged as a prevalent strategy to address this issue. Nevertheless, most existing pruners are primarily tailored to 2D inputs or rely on indirect
geometric cues, which limits their ability to explicitly retain semantically critical objects and maintain sufficient spatial coverage for robust 3D
reasoning. In this paper, we propose SeGPruner, a semantic-aware and geometry-guided
token reduction framework for efficient 3D QA with multi-view images. Specifically, SeGPruner first preserves semantically salient
tokens through an
attention-based importance module (Saliency-aware
Token Selector), ensuring that object-critical evidence is retained. It then complements these
tokens with spatially diverse ones via a geometry-guided selector (Geometry-aware
Token Diversifier), which jointly considers semantic relevance and 3D
geometric distance. This cooperation between saliency preservation and geometry-guided diversification balances object-level evidence and global scene coverage under aggressive
token reduction. Extensive experiments on ScanQA and OpenEQA demonstrate that SeGPruner substantially improves inference efficiency, reducing the visual
token budget by 91% and inference latency by 86%, while maintaining competitive performance in 3D
reasoning tasks.
Read the full abstract ▾
All authors Wenli Li, Kai Zhao, Haoran Jiang, Enquan Yang, Yi Su, Dan Zeng
Computer Vision
Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions
Xuesong Wang and Harry Wang
Vision-
language models (VLMs) exhibit a systematic bias when confronted with classic optical illusions: they overwhelmingly predict the illusion as "real" regardless of whether the image has been counterfactually modified. We present a tool-guided inference framework for the DataCV 2026 Challenge (Tasks I and II) that addresses this failure mode without any model training. An off-the-shelf
vision-
language model is given access to a small set of generic image manipulation tools: line drawing, region cropping, side-by-side comparison, and channel isolation, together with an illusion-type-routing system
prompt that prescribes which tools to invoke for each perceptual question category. Critically, every tool call produces a new, immutable image resource appended to a persistent registry, so the model can reference and compose any prior annotated view throughout its
reasoning chain. Rather than hard-coding illusion-specific modules, this generic-tool-plus-routing design yields strong cross-structural generalization: performance remained consistent from the validation set to a test set containing structurally unfamiliar illusion variants (e.g., Mach Bands rotated from vertical to horizontal stacking). We further report three empirical observations that we believe warrant additional investigation: (i) a strong positive-detection bias likely rooted in imbalanced illusion training data, (ii) a striking dissociation between pixel-accurate spatial
reasoning and logical inference over self-generated annotations, and (iii) pronounced sensitivity to image compression artifacts that compounds false positives.
Read the full abstract ▾
Computer Vision
A2BFR: Attribute-Aware Blind Face Restoration
Chenxin Zhu, Yushun Fang, Lu Liu et al. (+5 more)
Blind face restoration (BFR) aims to recover high-quality facial images from degraded inputs, yet its inherently ill-posed nature leads to ambiguous and uncontrollable solutions. Recent
diffusion-based BFR methods improve perceptual quality but remain uncontrollable, whereas text-guided face editing enables attribute manipulation without reliable restoration. To address these issues, we propose A$^2$BFR, an attribute-aware blind face restoration framework that unifies high-fidelity reconstruction with
prompt-controllable generation. Built upon a
Diffusion Transformer backbone with unified image-text cross-modal
attention, A$^2$BFR jointly conditions the denoising trajectory on both degraded inputs and textual
prompts. To inject semantic priors, we introduce attribute-aware learning, which supervises denoising latents using facial attribute
embeddings extracted by an attribute-aware encoder. To further enhance
prompt controllability, we introduce semantic dual-training, which leverages the pairwise attribute variations in our newly curated AttrFace-90K
dataset to enforce attribute discrimination while preserving fidelity. Extensive experiments demonstrate that A$^2$BFR achieves state-of-the-art performance in both restoration fidelity and instruction adherence, outperforming
diffusion-based BFR baselines by -0.0467 LPIPS and +52.58% attribute accuracy, while enabling fine-grained,
prompt-controllable restoration even under severe degradations.
Read the full abstract ▾
All authors Chenxin Zhu, Yushun Fang, Lu Liu, Shibo Yin, Xiaohong Liu, Xiaoyun Zhang, Qiang Hu, Guangtao Zhai
Computer Vision
Multimodal Models Meet Presentation Attack Detection on ID Documents
Marina Villanueva, Juan M. Espin and Juan E. Tapia
The integration of
multimodal models into Presentation Attack Detection (PAD) for ID Documents represents a significant advancement in biometric security. Traditional PAD systems rely solely on visual features, which often fail to detect sophisticated spoofing attacks. This study explores the combination of visual and textual modalities by utilizing pre-trained
multimodal models, such as Paligemma, Llava, and Qwen, to enhance the detection of presentation attacks on ID Documents. This approach merges deep visual
embeddings with contextual metadata (e.g., document type, issuer, and date). However, experimental results indicate that these models struggle to accurately detect PAD on ID Documents.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment
Qiyuan Zhuang, He-Yang Xu, Yijun Wang et al. (+3 more)
Understanding object affordances is essential for enabling
robots to perform purposeful and fine-grained interactions in diverse and unstructured environments. However, existing approaches either rely on
retrieval, which is fragile due to sparsity and coverage gaps, or on large-scale models, which frequently mislocalize contact points and mispredict post-contact actions when applied to unseen categories, thereby hindering robust generalization. We introduce
Retrieval-Augmented Affordance Prediction (RAAP), a framework that unifies affordance
retrieval with
alignment-based learning. By decoupling static contact localization and dynamic action direction, RAAP transfers contact points via dense correspondence and predicts action directions through a
retrieval-augmented
alignment model that consolidates multiple references with dual-weighted
attention. Trained on compact subsets of DROID and HOI4D with as few as tens of samples per task, RAAP achieves consistent performance across unseen objects and categories, and enables zero-shot robotic manipulation in both simulation and the real world. Project website: https://github.com/SEU-VIPGroup/RAAP.
Read the full abstract ▾
All authors Qiyuan Zhuang, He-Yang Xu, Yijun Wang, Xin-Yang Zhao, Yang-Yang Li, Xiu-Shen Wei
Computer Vision
Covert Visual Prompt Injection against Commercial Multimodal Large Language Models
Meiwen Ding, Song Xia, Chenqi Kong et al. (+1 more)
Although
multimodal large
language models (MLLMs) are increasingly deployed in real-world applications, their instruction-following behavior leaves them vulnerable to
prompt injection attacks. Existing
prompt injection methods predominantly rely on textual
prompts or perceptible visual
prompts that are observable by human users. In this work, we study imperceptible visual
prompt injection against powerful closed-source MLLMs, where
adversarial instructions are embedded in the visual modality. Our method adaptively embeds the malicious
prompt into the input image via a bounded text overlay to provide semantic guidance. Meanwhile, the imperceptible visual perturbation is iteratively optimized to align the feature representation of the attacked image with those of the malicious visual and textual targets at both coarse- and fine-grained levels. Specifically, the visual target is instantiated as a text-rendered image and progressively refined during optimization to more faithfully represent the desired semantics and improve transferability. Extensive experiments on two
multimodal understanding tasks across multiple closed-source MLLMs demonstrate the superior performance of our approach compared to existing methods.
Read the full abstract ▾
All authors Meiwen Ding, Song Xia, Chenqi Kong, Xudong Jiang
Computer Vision
Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations
Ni Ou, Zhuo Chen, Xinru Zhang et al. (+1 more)
Accurate camera-
LiDAR fusion relies on precise extrinsic calibration, which fundamentally depends on establishing reliable cross-modal correspondences under potentially large misalignments. Existing learning-based methods typically project
LiDAR points into depth maps for feature fusion, which distorts 3D geometry and degrades performance when the extrinsic initialization is far from the ground truth. To address this issue, we propose an extrinsic-aware cross-
attention framework that directly aligns image patches and
LiDAR point groups in their native domains. The proposed
attention mechanism explicitly injects extrinsic parameter hypotheses into the correspondence modeling process, enabling geometry-consistent cross-modal interaction without relying on projected 2D depth maps. Extensive experiments on the KITTI and nuScenes
benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches in both accuracy and robustness. Under large extrinsic perturbations, our approach achieves accurate calibration in 88% of KITTI cases and 99% of nuScenes cases, substantially surpassing the second-best baseline. We have open sourced our code on https://github.com/gitouni/ProjFusion to benefit the community.
Read the full abstract ▾
All authors Ni Ou, Zhuo Chen, Xinru Zhang, Junzheng Wang
Computer Vision
Yubo Cui, Xianchao Guan, Zijun Xiong et al. (+1 more)
Pre-trained
vision-
language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to
adversarial perturbations. Existing classification-guided
adversarial fine-tuning methods often disrupt pre-trained cross-modal
alignment, weakening visual-textual correspondence and degrading zero-shot performance. In this paper, we propose an
Alignment-Guided Fine-Tuning (AGFT) framework that enhances zero-shot
adversarial robustness while preserving the cross-modal semantic structure. Unlike label-based methods that rely on hard labels and fail to maintain the relative relationships between image and text, AGFT leverages the probabilistic predictions of the original model for text-guided
adversarial training, which aligns
adversarial visual features with textual
embeddings via soft
alignment distributions, improving zero-shot
adversarial robustness. To address structural discrepancies introduced by fine-tuning, we introduce a distribution consistency calibration mechanism that adjusts the robust model output to match a temperature-scaled version of the pre-trained model predictions. Extensive experiments across multiple zero-shot
benchmarks demonstrate that AGFT outperforms state-of-the-art methods while significantly improving zero-shot
adversarial robustness.
Read the full abstract ▾
All authors Yubo Cui, Xianchao Guan, Zijun Xiong, Zheng Zhang
Computer Vision
Hallucination-aware intermediate representation edit in large vision-language models
Wei Suo, Hanzu Zhang, Lijun Zhang et al. (+3 more)
Large
Vision-
Language Models have demonstrated exceptional performance in
multimodal reasoning and complex scene understanding. However, these models still face significant
hallucination issues, where outputs contradict visual facts. Recent research on
hallucination mitigation has focused on retraining methods and Contrastive Decoding (CD) methods. While both methods perform well, retraining methods require substantial training resources, and CD methods introduce dual inference overhead. These factors hinder their practical applicability. To address the above issue, we propose a framework for dynamically detecting
hallucination representations and performing
hallucination-eliminating edits on these representations. With minimal additional computational cost, we achieve state-of-the-art performance on existing
benchmarks. Extensive experiments demonstrate the effectiveness of our approach, highlighting its efficient and robust
hallucination elimination capability and its powerful controllability over
hallucinations. Code is available at https://github.com/ASGO-MM/HIRE
Read the full abstract ▾
All authors Wei Suo, Hanzu Zhang, Lijun Zhang, Ji Ma, Peng Wang, Yanning Zhang
Computer Vision
AA-Splat: Anti-Aliased Feed-forward Gaussian Splatting
Taewoo Suh, Sungpyo Kim, Jongmin Park et al. (+1 more)
Feed-forward 3D Gaussian Splatting (FF-3DGS) emerges as a fast and robust solution for sparse-view 3D reconstruction and novel view synthesis (NVS). However, existing FF-3DGS methods are built on incorrect screen-space dilation filters, causing severe rendering artifacts when rendering at out-of-distribution sampling rates. We firstly propose an FF-3DGS model, called AA-Splat, to enable robust anti-aliased rendering at any resolution. AA-Splat utilizes an opacity-balanced band-limiting (OBBL) design, which combines two components: a 3D band-limiting post-filter integrates multi-view maximal frequency bounds into the feed-forward reconstruction pipeline, effectively band-limiting the resulting 3D scene representations and eliminating degenerate Gaussians; an Opacity Balancing (OB) to seamlessly integrate all pixel-aligned Gaussian primitives into the rendering process, compensating for the increased overlap between expanded Gaussian primitives. AA-Splat demonstrates drastic improvements with average 5.4$\sim$7.5dB PSNR gains on NVS performance over a state-of-the-art (SOTA) baseline, DepthSplat, at all resolutions, between $4\times$ and $1/4\times$. Code will be made available.
Read the full abstract ▾
All authors Taewoo Suh, Sungpyo Kim, Jongmin Park, Munchurl Kim
Computer Vision
Extend3D: Town-Scale 3D Generation
Seungwoo Yoon, Jinmo Kim and Jaesik Park
In this paper, we propose Extend3D, a training-free pipeline for 3D scene generation from a single image, built upon an object-centric 3D generative model. To overcome the limitations of fixed-size latent spaces in object-centric models for representing wide scenes, we extend the latent space in the $x$ and $y$ directions. Then, by dividing the extended latent space into overlapping patches, we apply the object-centric 3D generative model to each patch and couple them at each time step. Since patch-wise 3D generation with image conditioning requires strict spatial
alignment between image and latent patches, we initialize the scene using a point cloud prior from a
monocular depth
estimator and iteratively refine occluded regions through SDEdit. We discovered that treating the incompleteness of 3D structure as noise during 3D refinement enables 3D completion via a concept, which we term under-noising. Furthermore, to address the sub-optimality of object-centric models for sub-scene generation, we optimize the extended latent during denoising, ensuring that the denoising trajectories remain consistent with the sub-scene dynamics. To this end, we introduce 3D-aware optimization objectives for improved
geometric structure and texture fidelity. We demonstrate that our method yields better results than prior methods, as evidenced by human preference and quantitative experiments.
Read the full abstract ▾
Computer Vision
PromptForge-350k: A Large-Scale Dataset and Contrastive Framework for Prompt-Based AI Image Forgery Localization
Jianpeng Wang, Haoyu Wang, Baoying Chen et al. (+4 more)
The rapid democratization of
prompt-based AI image editing has recently exacerbated the risks associated with malicious content fabrication and misinformation. However, forgery localization methods targeting these emerging editing techniques remain significantly under-explored. To bridge this gap, we first introduce a fully automated mask annotating framework that leverages keypoint
alignment and semantic space similarity to generate precise ground-truth masks for edited regions. Based on this framework, we construct PromptForge-350k, a large-scale forgery localization
dataset covering four state-of-the-art
prompt-based AI image editing models, thereby mitigating the data scarcity in this domain. Furthermore, we propose ICL-Net, an effective forgery localization network featuring a triple-stream backbone and intra-image contrastive learning. This design enables the model to capture highly robust and generalizable forensic features. Extensive experiments demonstrate that our method achieves an IoU of 62.5% on PromptForge-350k, outperforming SOTA methods by 5.1%. Additionally, it exhibits strong robustness against common degradations with an IoU drop of less than 1%, and shows promising generalization capabilities on unseen editing models, achieving an average IoU of 41.5%.
Read the full abstract ▾
All authors Jianpeng Wang, Haoyu Wang, Baoying Chen, Jishen Zeng, Yiming Qin, Yiqi Yang, Zhongjie Ba
Computer Vision
Assessing Multimodal Chronic Wound Embeddings with Expert Triplet Agreement
Fabian Kabus, Julia Hindel, Jelena Bratulić et al. (+6 more)
Recessive dystrophic epidermolysis bullosa (RDEB) is a rare genetic skin disorder for which clinicians greatly benefit from finding similar cases using images and clinical text. However, off-the-shelf foundation models do not reliably capture clinically meaningful features for this heterogeneous, long-tail disease, and structured measurement of agreement with experts is challenging. To address these gaps, we propose evaluating
embedding spaces with expert ordinal comparisons (triplet judgments), which are fast to collect and encode implicit clinical similarity knowledge. We further introduce TriDerm, a
multimodal framework that learns interpretable wound representations from small cohorts by integrating wound imagery, boundary masks, and expert reports. On the
vision side, TriDerm adapts visual foundation models to RDEB using wound-level
attention pooling and non-contrastive representation learning. For text, we
prompt large
language models with comparison queries and recover medically meaningful representations via soft ordinal
embeddings (SOE). We show that visual and textual modalities capture complementary aspects of wound phenotype, and that fusing both modalities yields 73.5% agreement with experts, outperforming the best off-the-shelf single-modality foundation model by over 5.6 percentage points. We make the expert annotation tool, model code and representative
dataset samples publicly available.
Read the full abstract ▾
All authors Fabian Kabus, Julia Hindel, Jelena Bratulić, Meropi Karakioulaki, Ayush Gupta, Cristina Has, Thomas Brox, Abhinav Valada, Harald Binder