Computer Vision · Cross-listed from cs.IR
MARQUIS: A Three-Stage Pipeline for Video Retrieval-Augmented Generation
Debashish Chakraborty, Dengjia Zhang, Jialiang Jin et al. (+7 more)
Retrieval-augmented generation from
videos requires systems to retrieve relevant audiovisual evidence from large corpora and synthesize it into coherent, attributed text. Current approaches struggle at both ends:
retrieval methods fail on complex, multi-faceted queries that cannot be captured by a single
embedding, while generation methods lack the high-level
reasoning needed to synthesize across multiple
videos and face
memory constraints over long, multi-
video contexts. We present MARQUIS: a three-stage pipeline that addresses these limitations through (1) query expansion, fusion, and reranking, (2) calibrated structured evidence extraction, and (3) article generation from extracted evidence, optionally controlled by an RLM. On the MAGMaR2026 shared task, we improve
retrieval performance from 0.195 to 0.759 (nDCG@10). For article generation, ITER-QA-BASE improves average human score from 3.09 to 3.83 over the CAG baseline, while MARQUIS-RLM achieves a human score of 3.30 and the strongest citation recall among non-QA systems.
Read the full abstract ▾
All authors Debashish Chakraborty, Dengjia Zhang, Jialiang Jin, Hanting Liu, Katherine Guerrerio, Hanxiang Qin, Tyler Skow, Alexander Martin, Reno Kriz, Benjamin Van Durme
Computer Vision
TouchMap-OR: Multi-View 3D Mapping of Hand-Surface Contacts
Sophokles Ktistakis, Rui Wang, Bastian Grande et al. (+1 more)
Hand-surface interactions between clinicians, patients, and medical equipment play a central role in pathogen transmission during medical procedures. However, these interactions remain largely unobserved, as current infection-prevention practices rely on manual observation and cannot reconstruct detailed contact histories. In this work we formulate the problem of identity-resolved hand-surface interaction reconstruction in operating rooms and introduce TouchMap-OR, a multi-view RGB-D
vision system that models clinicians, articulated hand geometry, and the semantic structure of the clinical environment to infer when and where contacts occur. The system reconstructs globally consistent multi-person 3D skeleton tracks across cameras while estimating articulated MANO hand meshes from RGB observations aligned to depth data. Multi-view hand reconstructions are fused and associated with tracked clinicians to obtain consistent left and right hand trajectories. A semantic 3D model of the operating room is built from multi-view
segmentation and depth fusion, enabling reconstructed hand trajectories to be mapped to specific surfaces, including medical equipment, movable objects, and patient body sites. Temporal hand-surface proximity is used to infer contact episodes describing which clinician touched which surface and when. We evaluate TouchMap-OR on recordings from three real anesthesia inductions with manually annotated contact events. TouchMap-OR achieves 0.75 binary contact F1, outperforming tracking-based baselines while maintaining comparable multi-person tracking accuracy and achieving 0.96 identity attribution accuracy.
Read the full abstract ▾
All authors Sophokles Ktistakis, Rui Wang, Bastian Grande, Hugo Sax
Computer Vision
SparseSAM: Structured Sparsification of Activations in Segment Anything Models
Hoai-Chau Tran, Chi H. Nguyen, Duy M. H. Nguyen et al. (+3 more)
The Segment Anything Model (SAM) achieves strong open-vocabulary
segmentation, but its ViT-based image encoders dominate inference latency and
memory. Existing activation compression methods, such as
token merging, reduce the
token length to process, yet introduce non-trivial runtime overhead and encounter catastrophic quality drop under high compression. Other methods applying Sparse
Attention focus on
attention alone, leaving the MLP fully dense and capping achievable speedup. We propose SparseSAM, a (i) training-free structured sparsification framework that jointly accelerates
attention and MLP layers while preserving
token identity. SparseSAM introduces (ii) Stripe-Sort
Attention, which uses a deterministic Z-order permutation to transform dense
attention into static hardware-friendly sparse patterns, eliminating dynamic masking overhead. SparseSAM further introduces a (iii) Residual-Consistency MLP that routes only informative
tokens through the MLP while propagating remaining
tokens through the residual pathway. Across four
segmentation benchmarks, SparseSAM loses only 0.004 mIoU at a 0.4 density and 0.021 mIoU at 0.3, a 2.10x reduction in accuracy loss versus
token merging advances, while achieving 2x faster inference and 2.8x
memory reduction.
Read the full abstract ▾
All authors Hoai-Chau Tran, Chi H. Nguyen, Duy M. H. Nguyen, Mathias Niepert, Fan Lai, Khoa D. Doan
Computer Vision
SegRAG: Retrieval Augmented Spatial Prompting for Open Vocabulary Semantic Segmentation
Abderrahmene Boudiaf, Irfan Hussain and Sajid Javed
Frozen
segmentation foundation models often fail when the target class appears in a form that is weakly represented during pretraining. To address this problem, we introduce SegRAG, a
retrieval-augmented inference-time spatial prompting pipeline for open-vocabulary semantic
segmentation that uses frozen models without updating their weights. SegRAG builds a compact class-indexed
memory from annotated references. When multiple references are available, Intra-Class Cohesion
Distillation (ICCD) filters DINOv3 patch descriptors by cross-image foreground agreement. With one reference, foreground descriptors are retained directly without ICCD. Topographic Similarity Grounding (TSG) then turns high-similarity query regions into point
prompts for SAM 3. In the matched five-shot comparison, SegRAG exceeds recent exemplar- and
retrieval-based baselines on ADE20K-150, Cityscapes, and PC-59. It also improves over the SAM 3 text-only baseline by 1.15 to 3.92 mean Intersection over Union (mIoU) points. In the up-to-30-shot AgML agricultural domain-transfer evaluation, SegRAG raises mIoU from 25.27 to 59.24. It also recovers text-only failures, including cauliflower from 0.00 to 95.36 IoU and sugarbeet weed from 0.00 to 80.22 IoU. Controlled ablations show complementary contributions from ICCD, TSG point selection, and joint text-and-point prompting. SegRAG therefore formulates
segmentation adaptation as an information organization and
retrieval problem by maintaining annotated visual knowledge as an external, class-indexed
memory that can guide a frozen
segmentation model without weight updates. Code: https://github.com/boudiafA/SegRAG.
Read the full abstract ▾
Computer Vision
Multi-task learning on partially labeled datasets via invariant/equivariant semi-supervised learning
Miquel Martí i Rabadán, Alessandro Pieropan, Hossein Azizpour et al. (+1 more)
We investigate the potential of invariant and equivariant semi-supervised learning for addressing the challenges of training multi-task models on partially labeled
datasets with differently structured output tasks. Specifically, we use the popular FixMatch method for invariant semi-supervised learning and its equivariant extension Dense FixMatch. We evaluate their performance on the Cityscapes and BDD100K
datasets in the context of the prevalent object detection and semantic
segmentation tasks in computer
vision. We consider varying sizes of the subsets annotated for each task and different overlaps among them. Our results for both invariant and equivariant semi-supervised learning outperform supervised baselines in most situations, with the most significant improvements observed when fewer labeled samples are available for a task and generally better results for the latter approach. Our study suggests that invariant/equivariant learning is a promising general direction for multi-task learning from limited labeled data.
Read the full abstract ▾
All authors Miquel Martí i Rabadán, Alessandro Pieropan, Hossein Azizpour, Atsuto Maki
Computer Vision
SafeLens: Deliberate and Efficient Video Guardrails with Fast-and-Slow Screening
Shahriar Kabir Nahin, Hadi Askari, Muhao Chen et al. (+1 more)
The rapid growth of online
video platforms and AI-generated content has made reliable
video guardrails a key challenge for safety and real-world deployment. While most
videos can be screened through fast pattern recognition, a small subset requires deeper
reasoning over temporally complex content and nuanced policy constraints. Existing approaches typically rely on large
vision-
language models applied uniformly across all inputs, resulting in high inference costs and inefficient allocation of computation. We propose SafeLens, a
video guardrail framework that introduces a fast-and-slow inference architecture for efficient and accurate content moderation with variable computational cost across inputs. Additionally, we construct a high-quality
dataset by applying influence-guided filtering to the SafeWatch
Dataset, retaining only 2.4% of the original data. To further address limitations of training-time scaling, we enable test-time
reasoning by augmenting the filtered data with structured Chain-of-Thought traces. Across real-world and AI-generated
video benchmarks, SafeLens achieves state-of-the-art performance, outperforming strong open-source
video guardrails (e.g., SafeWatch-8B, OmniGuard-7B) and closed-source models (e.g., GPT-5.4, Gemini-3.1-pro) while significantly reducing inference cost, demonstrating that efficient design serves to be more effective than scaling data or model size alone.
Read the full abstract ▾
All authors Shahriar Kabir Nahin, Hadi Askari, Muhao Chen, Anshuman Chhabra
Computer Vision · Cross-listed from cs.AI
AutoRubric-T2I: Robust Rule-Based Reward Model for Text-to-Image Alignment
Kuei-Chun Kao, Daixuan Huo, Yuanhao Ban et al. (+1 more)
Aligning Text-to-Image (T2I) generation models with human preferences increasingly relies on image reward models that score or rank generated images according to
prompt alignment and perceptual quality. Existing reward models are commonly trained as Bradley-Terry (BT) preference models on large-scale human preference corpora, making them costly to train, difficult to adapt, and opaque in their evaluation criteria. Meanwhile,
Vision-
Language Model (VLM) judges can provide more fine-grained assessments through textual rubrics, but their manually designed or heuristically generated scoring rules may fail to reliably reflect human preferences. In this paper, we propose AutoRubric-T2I, the first rubric learning framework in T2I that automatically synthesizes and selects explicit rubrics for guiding VLM judges. AutoRubric-T2I first synthesizes
reasoning traces from preference pairs into candidate rubrics, then uses a VLM judge to score paired images under each rubric, producing pairwise rubric-score differences for preference learning. To remove noisy and redundant rules, we further employ a $\ell_1$-Regularized Logistic Regression Refiner, which selects the Top-$N$ most discriminative rubrics. Extensive evaluations show that AutoRubric-T2I produces high-quality, interpretable reward signals using less than 0.01% of the annotated preference data, substantially reducing the need for large-scale reward-model training. On image reward
benchmarks such as MMRB2, AutoRubric-T2I outperforms strong reward model baselines. We further validate AutoRubric-T2I as an RL reward on downstream T2I tasks, including TIIF and UniGenBench++, where it improves generation quality over scalar reward models using the Flow-GRPO pipeline on
diffusion models.
Read the full abstract ▾
All authors Kuei-Chun Kao, Daixuan Huo, Yuanhao Ban, Cho-Jui Hsieh
Computer Vision
Error-Decomposed Class-Conditional Fusion for Statistically Guaranteed Hard-Category Robust Perception
Guowei Luo, Ziqi Shi and Zhao Xie
Aggregate object detection metrics inherently mask catastrophic and repeatable failures in operationally critical, long-tail minority classes. This paper formally defines this pervasive vulnerability as the Hard-Category Reliability Problem (HCRP): the fundamental architectural challenge of strictly rectifying vulnerable categories without compromising the performance boundaries of stable classes under stringent protocols. To systematically dismantle this limitation, we propose Error-Decomposed Class-Conditional Fusion (ED-CCF), an elegant decision-layer inference framework. Diverging from heuristic global post-processing, ED-CCF projects predictions into a sophisticated quad-state error taxonomy, dynamically activating calibration pathways exclusively upon rigorous empirical justification. On a highly constrained 600-image validation
benchmark, isolating cz as the critical vulnerability (HCEC=0.86, BSR=0.14), our framework achieves a targeted breakthrough: it elevates cz mAP50 from 0.089343 to 0.109353 (a massive +22.4% relative surge) while flawlessly preserving the Pareto optimality of global stability (raising all mAP50 from 0.581925 to 0.584864). Backed by exhaustive validation across 50 paired subset trials demonstrating an overwhelming 96% win rate and strict Bonferroni-corrected Wilcoxon significance (p<0.05), this work fundamentally redefines output-level fusion as an auditable, statistically guaranteed paradigm for safety-critical visual perception.
Read the full abstract ▾
Computer Vision
MSIQ: Moment-based Scale-Invariant Quality Measure for Single Image Super-Resolution
Leonid Bedratyuk
Assessing the quality of single image super-resolution (SISR) results remains an open methodological problem. Common full-reference metrics (PSNR, SSIM, LPIPS) do not explicitly evaluate the preservation of the
geometric structure of images, which is critical for the correctness of scale-based reconstruction. In addition, they require the forced
alignment of images to the same size (forced resizing), which introduces an external interpolation error into the evaluation process. This paper proposes a diagnostic scale-invariant quality measure, MSIQ (Moment-based Scale-Invariant Quality), based on the comparison of normalized central
geometric moments of two images. MSIQ enables direct comparison of images with different spatial resolutions without resizing, is mathematically deterministic (model-free), and has an analytical form. To provide a theoretical basis for the approach, we introduce a conceptual distinction between the ability of metrics to monotonically track degradation (tracking ability) and their
geometric selectivity (
geometric specificity). The experimental validation confirmed the stability of MSIQ under uniform scaling and, at the same time, revealed the high sensitivity of traditional metrics to the choice of interpolation method. The results show that MSIQ has pronounced
geometric selectivity: the proposed measure effectively separates
geometric deformations from non-
geometric artifacts, in particular JPEG compression, unlike pixel-based and perceptual metrics. It is also shown that the response of MSIQ to structural perturbations remains stable across different classes of SR
algorithms, including DNN models with different architectures. The proposed measure is a complementary diagnostic tool for domains where
geometric fidelity has priority, in particular medical imaging and remote sensing.
Read the full abstract ▾
Computer Vision
Xin Wang, Yixu Wang, Jiaming Zhang et al. (+6 more)
Large-scale pre-trained
Vision-
Language models (VLMs), such as CLIP, exhibit strong zero-shot generalization, yet remain highly vulnerable to imperceptible
adversarial perturbations, raising serious safety concerns for open-world deployment. To enhance robustness without requiring downstream task-specific retraining, we propose TAME, a novel test-time defense. Building upon our prior Test-Time
Adversarial Prompt Tuning (TAPT), TAME introduces an architectural reformulation by replacing TAPT's single adaptive
prompt with an input-conditioned Mixture-of-Experts (MoE) framework, enabling more expressive and adaptive defense. Specifically, TAME maintains a bank of learnable expert
prompts and employs an input-dependent routing mechanism to aggregate a customized
prompt mixture for each unlabeled test sample at inference time. This test-time defense mechanism is driven by three unsupervised objectives: (1) multi-view prediction
entropy minimization, (2) layer-wise
alignment of visual
token statistics to precomputed clean and
adversarial reference distributions, and (3) MoE regularization for balanced expert utilization and
prompt diversity. We evaluated TAME on 11
benchmark datasets, including ImageNet and 10 additional zero-shot
datasets. The results show that TAME improves the zero-shot
adversarial robustness of the original CLIP by at least 49.1% under AutoAttack while largely preserving generalization on clean samples. TAME also consistently outperforms existing
adversarial prompt tuning methods across multiple
prompt designs, yielding an average robustness gain of at least 30.2%.
Read the full abstract ▾
All authors Xin Wang, Yixu Wang, Jiaming Zhang, Ruofan Wang, Jiaqi Yu, Kai Chen, Jingjing Chen, Xingjun Ma, Yu-Gang Jiang
Computer Vision
Deepfake Detection in Social Media: A Temporal Artifact Analysis Using 3D Convolutional Neural Networks
Mohammadreza Rashidi, Raja Hashim Ali and Sami Ur Rahman
Synthetic facial
videos have proliferated across social media faster than platform moderation can respond, raising the cost of disinformation and identity-based attacks. Frame-level deepfake detectors degrade sharply as generator quality increases; high-quality 128x128 GAN output cuts spatial-only accuracy by five percentage points while leaving temporal inconsistencies largely intact. We address this gap with a 3D Convolutional Neural Network detector based on R3D-18, trained with a composite loss that combines binary cross-
entropy with a temporal-consistency regularizer. The model processes 16-frame clips from the DeepfakeTIMIT
dataset and is initialized from Kinetics-400 action-recognition weights. We report 92.8% accuracy on intra-
dataset evaluation at 128x128 resolution; cross-
dataset transfer to FaceForensics++ without fine-tuning reaches 76.4%, rising after minimal fine-tuning. Ablation studies show that transfer learning contributes 7.2 percentage points and face tracking adds 3.5 points, while temporal consistency regularization provides additional gains on high-quality fakes. The results establish that temporal artifacts generalize more broadly than spatial ones, providing a detection signal that survives social-media re-encoding.
Read the full abstract ▾
Computer Vision
Stable Routing for Mixture-of-Experts in Class-Incremental Learning
Zirui Guo, Quan Cheng, Da-Wei Zhou et al. (+1 more)
Class-incremental learning (CIL) requires models to learn new classes sequentially while preserving prior knowledge. Recently, approaches that combine pre-trained models with mixture-of-experts (MoE) have received increasing
attention in CIL: they typically expand experts during learning and employ a router to assign weights across experts. However, existing MoE methods often overlook routing drift induced by expert expansion. Once new experts are introduced, the router may reassign samples from earlier classes to newly added experts, thereby perturbing previously established expert compositions and causing interference even when old experts remain frozen. We argue that expandable MoE in CIL requires two complementary properties: stable old-class routing for knowledge preservation and sufficient capacity utilization for new-class adaptation. To this end, we propose Stable Routing for MoE (
StaR-MoE), a routing-level framework for expandable MoE in CIL. By incorporating sensitivity-aware routing
alignment,
StaR-MoE aligns current old-class routing behavior with historical routing distributions through sensitivity-guided constraints. Complementarily,
StaR-MoE introduces asymmetric capacity regularization to encourage effective utilization of the expanded expert pool without compromising class-specific routing specialization. Extensive experiments across four standard CIL
benchmarks demonstrate that
StaR-MoE consistently improves both average and last accuracy over state-of-the-art methods, highlighting the importance of stable routing.
Read the full abstract ▾
All authors Zirui Guo, Quan Cheng, Da-Wei Zhou, Lijun Zhang
Computer Vision
Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework
Yumeng Yao, Jingzhi Dong, Haowen Gu et al. (+4 more)
With the rapid progress of
multimodal foundation models and predictive pre-training, an important open question is how to equip 3D point clouds with a pre-training paradigm that is better aligned with next-
token and next-
embedding learning. Existing point-cloud self-supervised methods are largely built on masked reconstruction or explicit
geometric generation, and thus remain tied to input recovery rather than predictive dependency modeling. In this paper, we introduce PointNTP, which reformulates point cloud pre-training as a fully causal, decoder-free latent Next-
Token Prediction problem. Specifically, each point cloud is first partitioned into local patches and serialized into a structured 3D
token sequence according to patch-center geometry. The resulting sequence is then modeled by a causal
Transformer under prefix-only conditioning, and trained with a shift-based prediction objective stabilized by stop-gradient targets. This design enables the model to learn structural dependencies directly in latent space, without reconstruction decoders or explicit
geometric recovery. Extensive experiments demonstrate that the proposed PointNTP is highly competitive across multiple downstream tasks: it achieves 93.8%(+0.5%), 92.6%(+0.3%), and 89.3%(+1.1%) on OBJ_BG, OBJ_ONLY, and PB_T50_RS of ScanObjectNN, respectively; obtains 85.0%(+0.1%) in Cls.mIoU on ShapeNetPart; and reaches 71.1% mAcc on S3DIS Area 5. Overall, decoder-free causal latent prediction provides a simple, scalable, and potentially modality-agnostic paradigm for point-cloud self-supervised learning, offering a new 3D perspective on foundation-style predictive learning for 3D data.
Read the full abstract ▾
All authors Yumeng Yao, Jingzhi Dong, Haowen Gu, Tao Chen, Zonghan Wu, Xiaoshui Huang, Yazhou Yao
Computer Vision
A Conditional U-Net Pipeline with Pre- and Post-Processing for Aerial RGB-to-Thermal Image Translation
Tseten Sherpa, Sikandar Ali, Shubham Parab et al. (+5 more)
Paired RGB-thermal data has shown significant utility across a range of applications, including image fusion, object tracking, and anomaly detection; however, its broader adoption is constrained by the limited availability of aligned RGB-thermal image pairs. RGB-to-thermal (and vice versa) image translation has emerged as a practical solution to this challenge. Prior approaches including conditional generative
adversarial networks (cGANs) such as ThermalGAN and Scalable Interpolant
Transformer (SiT)-based architectures such as ThermalGen have demonstrated strong potential for aerial-to-thermal image translation. In this work, we explore alternative architectures that prioritize simplicity while maintaining performance. Specifically, we propose a conditional U-Net that incorporates weather data at the bottleneck layer, complemented by targeted preprocessing and post-processing techniques applied within the Pix2Pix GAN architecture. We utilize a training set of 612 paired RGB and thermal images, and evaluate over 5-fold cross-validation, ultimately testing on a held-out test set. Our conditional U-Net model performed best, with a peak signal-to-noise ratio (PSNR) of 14.5485, structural similarity index measure (SSIM) of 0.8095, and learned perceptual image patch similarity (LPIPS) of 0.1666. These results outperformed the base ThermalGen model, which attained PSNR, SSIM, and LPIPS scores of 7.56, 0.2444, and 0.6317 respectively. We find that while saturation boost and contrast enhancement for preprocessing and Gaussian blur for post-processing provide observable improvements, the incorporation of conditioning data was most effective. Our findings cement the potential of integrating auxiliary metadata into thermal image generation, suggesting that such information can serve as a proxy for environmental conditions critical to accurate thermal reconstruction.
Read the full abstract ▾
All authors Tseten Sherpa, Sikandar Ali, Shubham Parab, Haoyun Feng, Matthew Dennis, Keenan Gibbons, Verrah Otiende, Geoffrey H. Siwo
Computer Vision · Cross-listed from cs.GR
Real-Time Neural Hair G-Buffer Anti-Aliasing
Chenghao Wu, Yuefan Shen, Tao Huang et al. (+3 more)
We propose a lightweight real-time method for reconstructing strand-based hair G-Buffers from severely undersampled rasterized inputs. Our pipeline first applies neural spatial reconstruction and temporal accumulation to recover hair coverage, i.e., fractional hair visibility within a pixel, and tangent. It then uses a tangent-guided reconstruction step to complete the position, which is subsequently used for physically based deferred hair shading. We evaluate our method across a diverse set of hairstyles, including straight, wavy, afro, and ponytail styles, under both static and dynamic scenarios. Our method achieves higher hair reconstruction quality than general industrial neural reconstruction solutions such as DLSS and FSR.
Read the full abstract ▾
All authors Chenghao Wu, Yuefan Shen, Tao Huang, Kai Yan, Zahra Montazeri, Kui Wu
Computer Vision · Cross-listed from cs.LG
PFlow-T: A Persistence-Driven Forward Process for Topology-Controlled Generation
Snigdha Chandan Khilar
Current
topology aware
diffusion models face an architectural mismatch by using Gaussian noise for corruption while recovering structural features through conditional side channels To fix this we introduce PFlow T a generative model that bases its forward process entirely on persistent homology In PFlow T time measures the destruction of H1 topological features like holes rather than Gaussian noise injection This forward process eliminates features based on their persistence The reverse network then directly inverts this structured corruption to predict the clean state in one step Tests on MNIST digits zero one and eight show PFlow T significantly outperforms a baseline model in generating requested Betti numbers and handling out of distribution tasks PFlow T is the first generative architecture using persistent homology for the forward process although we note it is currently limited to low resolution pixel space proxies
Read the full abstract ▾
Computer Vision
HL-OutPaint: Coarse-to-Fine Video Outpainting for High-Resolution Long-Range Videos
Jeongeun Park, Janghyeok Han, Geonung Kim et al. (+4 more)
Video outpainting generates plausible visual content beyond the original spatial extent of a
video, playing a key role in adapting
videos to diverse display formats. To support such use cases, it must enable large spatial extrapolation over long sequences. However, most existing methods address only one of these challenges or lack explicit mechanisms for ensuring global spatio-temporal consistency, leading to notable limitations. In this paper, we propose HL-OutPaint, a high-resolution
video outpainting framework for long sequences. Our approach follows a coarse-to-fine strategy with a two-stage pipeline. We first construct Global Coarse Guidance (GCG), a low-resolution representation that captures global structure and dominant motion across the
video. Unlike naive downsampling, GCG is built via a novel global-local frame swapping mechanism that couples sparse global keyframes with local temporal windows and exchanges information during sampling. This enables GCG to encode both long-term structural consistency and short-term temporal dynamics in a unified representation. Guided by this representation, HL-OutPaint then performs high-resolution outpainting to generate spatially detailed and temporally consistent content. By separating global structure modeling from fine-grained synthesis, our framework achieves stable, coherent generation for large spatial expansion and long
video sequences. Extensive experiments show that HL-OutPaint outperforms existing methods in challenging scenarios involving wide spatial extrapolation and long
video sequences.
Read the full abstract ▾
All authors Jeongeun Park, Janghyeok Han, Geonung Kim, Hyun-Seung Lee, Kyuha Choi, Youngseok Han, Sunghyun Cho
Computer Vision
Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification
Yuting Yang, Haichao Jiang, Tianming Liang et al. (+2 more)
Referring
segmentation aims to segment the target objects in images or
videos based on the textual query. Despite remarkable progress over the past years, existing works always assume that the user-provided queries are already precise and clear. However, this assumption is impractical. In real-world scenarios, it is unrealistic to expect all users to thoroughly review their visual content and carefully ensure their queries are unique and unambiguous. When encountering such cases, existing
segmentation models tend to arbitrarily guess the user preferences, often resulting in undesired outcomes. To address this limitation, we propose IC-Seg, a novel agentic framework that proactively clarifies user intent through multi-turn conversation before
segmentation. To effectively incentivize this capability, we further introduce Hi-GRPO, a new hierarchical optimization strategy that injects dense and informative supervision signals at the trajectory, turn, and step levels. This strategy encourages efficient intent clarification, effectively eliminating redundant interactions and improving overall dialogue quality. For evaluation, we establish Ambi-RVOS, a referring
video object
segmentation benchmark with ambiguous user queries. Extensive experiments demonstrate that IC-Seg not only outperforms existing methods by a large margin in resolving ambiguous queries, but also maintains state-of-the-art performance on standard
reasoning segmentation benchmarks. Code and data will be released at https://github.com/iSEE-Laboratory/IC-Seg.
Read the full abstract ▾
All authors Yuting Yang, Haichao Jiang, Tianming Liang, Quan Zhang, Jian-Fang Hu
Computer Vision
Designing streetscapes from street-view imagery using diffusion models
Yuzhou Chen, Yuebing Liang, Lingqian Hu et al. (+4 more)
Street-view imagery (SVI) is widely used to quantify key indicators of urban environment, such as green- ery, sky, or road view indices. However, existing studies largely focus on measuring current streetscapes and rarely support the generation of alternative and non-existing urban scenarios, which is a core task in geospatial disciplines such as urban planning and design. To address this gap, we propose a gener- ative
multimodal AI framework that synthesizes alternative streetscapes conditioned on targeted visual metrics, enabling direct visual exploration of urban scenarios. We first construct a
multimodal dataset that aligns SVIs with textual descriptions,
segmentation maps, road masks, and quantitative metrics of visual elements in Chicago and Orlando. Using this
dataset, we demonstrate that
diffusion models can produce realistic and semantically consistent streetscape imagery while responding to both textual and imagery controls. Our quantitative evaluations show that incorporating visual controls can improve semantic consistency, reducing the LPIPS index by approximately 6% while maintaining global visual realism. In addition, overall semantic consistency increases by 23.7% in Orlando and 46.4% in Chicago, as measured by the mIoU index, with class-wise gains exceeding even 100% improvement for building view indices. Streetscape generation can be controlled in a fine-grained manner by both visual and textual
prompts, and when textual and visual controls conflict, imagery controls consistently dominate, indicating a clear control hierarchy and the importance of further developing visual controls for urban scene generation. Overall, this work establishes an important
benchmark for streetscape generation us- ing SVIs and
diffusion models, and illustrates how generative AI can serve as a practical, scalable, and controllable approach for urban scenario exploration.
Read the full abstract ▾
All authors Yuzhou Chen, Yuebing Liang, Lingqian Hu, Kailai Sun, Qingqi Song, Chang Zhao, Shenhao Wang
Computer Vision
Degradation Frequency Curve: An Explicit Frequency-Quantified Representation for All-in-One Image Restoration
Xinghua Huang, Zhixiong Yang, Chen Wu et al. (+6 more)
A fundamental difficulty in all-in-one blind image restoration is that degradation is usually treated as an implicit factor hidden in degraded-to-clean mapping, rather than as an explicit object that can be measured and manipulated. This limitation becomes more pronounced under mixed, compound, or unseen degradation conditions, where degradation effects are hard to assign to predefined labels or task-specific parameters. We propose the Degradation Frequency Curve (DFC), a structured spectral representation that quantifies degradation responses by measuring band-wise residual-to-degraded energy ratios in the frequency domain. DFC converts visually entangled and hard-to-describe degradation effects into a measurable degradation coordinate space. Moreover, DFC can be adaptively decomposed into band-wise spectral
tokens, allowing local degradation responses to be represented as reusable restoration priors. Based on this representation, we develop the DFC-guided Image Restorer (DFC-IR), a
token-conditioned multi-scale framework that progressively estimates DFCs from intermediate restorations and uses the resulting spectral
tokens to guide degradation-aware restoration in a coarse-to-fine manner. Extensive experiments on standard, composite, unseen, and real-world degradation
benchmarks show that DFC provides an effective representation basis for all-in-one restoration, leading to state-of-the-art performance and improved generalization under complex degradation profiles.
Read the full abstract ▾
All authors Xinghua Huang, Zhixiong Yang, Chen Wu, Shengxi Li, Shuaifeng Zhi, Yue Zhang, Qibin Hou, Xin Deng, Jingyuan Xia
Computer Vision
A Distributional View for Visual Mechanistic Interpretability: KL-Minimal Soft-Constraint Principle
Guancheng Zhou, Yisi Luo, Zhengfu He et al. (+5 more)
Most current paradigms in visual mechanistic interpretability (MI) remain confined to interpreting internal units of the
vision model via heuristic methods (e.g., top-$K$ activation
retrieval or optimization with regularization). In this work, we establish a theoretical distributional view for visual MI, which models the influence of a feature activation on the natural image distribution, thereby formulating a Kullback-Leibler (KL)-minimal optimization problem to model the MI task. Under this framework, statistical biases are identified within previous MI paradigms, which reveal that they may either be perceptually uninterpretable to humans (i.e., deviate from the natural image distribution), or mechanistically unfaithful to the
vision models (i.e., unable to activate model features). To resolve the biases under the distributional view, we propose a model with a KL-minimal soft-constraint principle for visual MI that theoretically balances interpretability and faithfulness. We realize this principle via energy-guided
diffusion posterior sampling. Extensive experiments validate the theoretical soundness of the proposed distributional view and demonstrate the practical effectiveness of our paradigm on the DINOv3
vision model.
Read the full abstract ▾
All authors Guancheng Zhou, Yisi Luo, Zhengfu He, Zhenyu Jin, Xuyang Ge, Wentao Shu, Deyu Meng, Xipeng Qiu
Computer Vision · Cross-listed from cs.LG
The Silent Brush: Evaluating Artistic Style Leakage in AI Art Generation
Ninad Joshi, Ashutosh Ranjan, Vivek Srivastava et al. (+1 more)
Generative text-to-image models are typically trained on large-scale web-scraped
datasets that include diverse visual content such as copyrighted and stylistically distinctive artworks, raising concerns about ownership, attribution, and the unintended reuse of protected visual expressions. A key issue is that models can learn stylistic patterns from this data and reproduce them in generated outputs without any explicit reference in the
prompt. We refer to this phenomenon as The Silent Brush, where such learned styles reappear even when they are not requested. Existing evaluation methods mainly focus on near-duplicate
retrieval or membership inference and do not account for this form of unintended stylistic resurfacing across
prompts. To address these gaps, we first formulate guiding principles for evaluation of The Silent Brush. We then introduce Art Arena, an evaluation protocol that measures how strongly artworks are encoded, how they interact, and how frequently their stylistic traits reappear in generated outputs without explicit mention in
prompts. We evaluate Art Arena on widely used text-to-image
diffusion models, including Stable
Diffusion v1.5, Stable
Diffusion XL (SDXL), and SANA-1.5, and design it to generalize across text-to-image generative systems. Our results show that The Silent Brush arises from differences in representational strength and interaction dynamics between artworks, leading to asymmetric blending in model generations. Code and evaluation resources are available at: https://anonymous.4open.science/r/ArtArena-EBE4.
Read the full abstract ▾
All authors Ninad Joshi, Ashutosh Ranjan, Vivek Srivastava, Shirish Karande
Computer Vision · Cross-listed from cs.LG
Beyond Linear Superposition: Discovering Climate Features in AI Weather Models with KAN-SAE
Minjong Cheon
Deep learning weather prediction models achieve remarkable predictive skill yet remain largely opaque: we know little about how they represent physical
climate phenomena internally. Mechanistic interpretability through Sparse Autoencoders (SAEs) offers a principled route to decomposing these representations, but existing SAEs assume strictly linear feature superposition - a constraint ill-suited for the highly nonlinear atmospheric dynamics encoded in modern
transformers. We introduce KAN-SAE, a sparse autoencoder whose encoder replaces the standard ReLU with learnable per-feature B-spline activations drawn from Kolmogorov-Arnold Networks (KANs), allowing each latent dimension to develop its own nonlinear gating profile. Applied to Sonny, KAN-SAE discovers 975 alive features (vs. 566 for a linear baseline, a 72% improvement) with 20% lower inter-feature redundancy and comparable reconstruction fidelity. Without any
climate supervision, KAN-SAE identifies an interpretable European heatwave feature spatially concentrated over western Europe, and a western Pacific typhoon tracker confirmed by causal steering experiments. Our results demonstrate that nonlinear activations are essential for mechanistic interpretability of deep learning weather prediction models, recovering
climate features that remain invisible to linear baselines.
Read the full abstract ▾
Computer Vision
Employing Vision-Language Models for Face Image Quality Assessment
Erdi Sar{\i}taş, Eren Onaran, Vitomir Štruc et al. (+1 more)
Face Image Quality Assessment (FIQA) is a crucial control step in biometric pipelines. It ensures only reliable samples are processed to maintain system accuracy. State-of-the-art FIQA methods achieve high utility but typically operate as "black boxes." They produce scalar scores without human-interpretable justifications. This lack of transparency limits their effectiveness in human-in-the-loop scenarios, such as automated border control, where actionable feedback is essential. In this paper, we investigate the potential of off-the-shelf
Vision-
Language Models (VLMs) to bridge this gap by performing FIQA in a zero-shot setting. We present a comprehensive evaluation framework for assessing VLM performance. This involves benchmarking traditional FIQA methods through error-versus-reject curves. Additionally, using a diverse set of
datasets, ranging from surveillance-oriented to synthetically generated, we analyzed their interpretability, consistency, and robustness to
prompt changes. Our results show biometric utility performance depends significantly on architecture, not merely on parameter count. Most VLMs' outputs align with those of traditional methods. We also find that VLM ranking performance and the generated scores may vary across
prompts. Our synthetic ablation study shows that while increasing the parameter count can improve internal consistency, it yields worse degradation-detection performance than smaller models. These findings suggest that zero-shot FIQA score estimation using VLMs is promising and could effectively complement conventional FIQA pipelines as an interpretability module. The codes are available at https://github.com/ThEnded32/VLM4FIQA.git.
Read the full abstract ▾
All authors Erdi Sar{\i}taş, Eren Onaran, Vitomir Štruc, Haz{\i}m Kemal Ekenel
Computer Vision
Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generation
Yuheng Chen, Qingdong He, Teng Hu et al. (+4 more)
The landscape of joint audio and
video generation has been fundamentally transformed by the advent of powerful foundation models. Despite these strides, achieving cohesive
multimodal customization for the simultaneous preservation of visual identities and vocal timbres across multiple interacting subjects remains largely underexplored. To bridge this gap, we present Omni-Customizer, an end-to-end framework targeted at the precise binding and seamless fusion of
multimodal identity information. Specifically, we introduce an Omni-Context Fusion (OCF) module that effectively enriches the base textual
prompt with dense,
multimodal identity cues, along with a Masked TTS Cross-
Attention (MTP-CA) mechanism explicitly designed to prevent the severe "
speech leakage" problem. Within this architecture, we propose Semantic-Anchored
Multimodal RoPE (SA-MRoPE) to anchor visual and audio reference
tokens, along with TTS
embeddings, to their corresponding semantic descriptions, enabling structured
multimodal fusion and robust identity binding. Furthermore, we devise a comprehensive training strategy that incorporates interleaved audio-
video scheduling to rapidly adapt the audio branch to
multilingual scenarios without degrading foundational priors, and a progressive in-pair to cross-pair curriculum to facilitate the learning of high-level and robust identity features. Extensive experiments demonstrate that Omni-Customizer achieves state-of-the-art performance in dual-modal customized generation, excelling across visual identity similarity, timbre consistency, precise audio-
video synchronization, and overall
video-audio fidelity.
Read the full abstract ▾
All authors Yuheng Chen, Qingdong He, Teng Hu, Yuji Wang, Yabiao Wang, Lizhuang Ma, Jiangning Zhang
Computer Vision
On Applicability of Synthetic Datasets for Facial Expression Recognition
Ali Azmoudeh, Erdi Sar{\i}taş, Ömer Y{\i}ld{\i}r{\i}m et al. (+1 more)
Facial Expression Recognition faces two core challenges. The first is class imbalance in public
datasets, which skews the learning process and weakens generalization. The second is related to
privacy and data collection constraints, which limit the sharing of facial images and restrict the creation of large, balanced
datasets. To address these issues, we examine three complementary strategies for constructing
privacy-preserving FER
datasets in the standard seven discrete facial expression classes setting. Our strategies are: (i) pseudo-labeling large unlabeled face collections with a teacher model under a confidence-thresholding scheme, (ii)
prompt-driven synthesis using
diffusion models conditioned on demographic attributes, and (iii) task-aware GAN-based expression editing that modifies facial expression while preserving identity and realism. For training and evaluation, we employed widely adopted
datasets, including AffectNet, RAF-DB, and FER2013. We utilized the synthetic
datasets DigiFace, DCFace, and EmoNet-Face BIG as unlabeled sources for pseudo-labeling. Additionally, we utilized the FFHQ
dataset as the source for generative synthesis. The main experiments are conducted using a classic CNN backbone, IR50, and we also explore a more complex architecture, POSTERv1, to assess its feasibility and robustness. Using cross-
dataset evaluations, we analyze the trade-offs each strategy presents in curated
datasets. The findings demonstrate how synthetic data can effectively substitute or be combined with real
datasets to mitigate imbalance and
privacy limitations. Code and generated
datasets:https://www.github.com/AliAZ98/SyntFER
Read the full abstract ▾
All authors Ali Azmoudeh, Erdi Sar{\i}taş, Ömer Y{\i}ld{\i}r{\i}m, Haz{\i}m Kemal Ekenel
Computer Vision
Mamba-VGGT: Persistent Long-Sequence Video Geometry Grounded Transformer via External Sliding Window Mamba Memory
Tianchen Deng, Zhenxiang Xiong, Nailin Wang et al. (+4 more)
Visual Geometry Grounded
Transformers (VGGT) have set new
benchmarks in high-fidelity 3D scene reconstruction. However, as the sequence length increases, these models suffer from catastrophic
geometric forgetting and accumulation drift, primarily due to the quadratic complexity of global
attention which necessitates truncated temporal windows. To overcome the resulting
geometric drift, we present Mamba-VGGT, an enhanced VGGT framework capable of persistent long-range
reasoning. Our key contribution is a Sliding Window Mamba (SWM)
memory module that maintains an explicit external
memory token across temporal windows. This module leverages selective state-space modeling to distill and propagate global
geometric priors, effectively bypassing the
memory constraints of traditional
transformers. To integrate these long-term temporal cues without disrupting the highly optimized spatial features of the pre-trained VGGT, we propose a Zero-Init Spatial
Memory Injector. Utilizing zero-convolutional layers, this injector adaptively fuses persistent
memory into the patch
token stream, ensuring structural stability and seamless feature
alignment. Extensive experiments demonstrate that our approach significantly outperforms existing VGGT-based methods in maintaining spatial consistency and reducing trajectory accumulation errors. Our work provides a scalable, linear-complexity solution for geometry-grounded world modeling in extensive 3D environments.
Read the full abstract ▾
All authors Tianchen Deng, Zhenxiang Xiong, Nailin Wang, Fangjinhua Wang, Jiuming Liu, Jianfei Yang, Hesheng Wang
Computer Vision
Weighted Reverse Convolution for Feature Upsampling
Wentong Li, Zhiyuan Qi, Zichen Zhao et al. (+2 more)
Pre-trained
vision foundation models (VFMs) provide strong semantic representations, yet their patch-level features are inherently coarse, limiting their effectiveness on tasks requiring fine-grained localization, dense prediction, and point-wise correspondence. In this work, we revisit feature upsampling for VFMs from the perspective of \textbf{inverse problem} and propose Weighted Reverse Convolution (WRC), a spatially adaptive inverse operator for densifying high-level visual descriptors. Specifically, we formulate feature upsampling as a weighted Tikhonov-regularized least-squares problem, where spatially varying weights modulate both data fidelity and prior strength at each spatial location. This allows WRC to adapt the reconstruction to spatially varying feature characteristics, thereby preserving critical structures while mitigating over-smoothing. Moreover, WRC retains an efficient, fully differentiable closed-form FFT solution, making it a practical drop-in upsampling operator. Integrated into a lightweight self-supervised densification framework, WRC consistently improves dense feature quality across various downstream
benchmarks, including
segmentation, depth estimation,
video object
segmentation, object discovery, and keypoint correspondence, while maintaining high computational efficiency.
Read the full abstract ▾
All authors Wentong Li, Zhiyuan Qi, Zichen Zhao, Kai Zhang, Lei Zhang
Computer Vision
EchoSR: Efficient Context Harnessing for Lightweight Image Super-Resolution
Hanli Zhao, Binhao Wang, Shihao Zhao et al. (+3 more)
Image super-resolution (SR) aims to reconstruct high-quality, high-resolution (HR) images from low-resolution (LR) inputs and plays a critical role in various downstream applications. Despite recent advancements, balancing reconstruction fidelity and computational efficiency remains a fundamental challenge, particularly in resource-constrained scenarios. While existing lightweight methods attempt to expand receptive fields, many of them either incur substantial computational overhead, naively scale up kernel sizes, or lack mechanisms for coherent multi-scale integration, limiting their overall effectiveness and scalability. To address these limitations, we propose EchoSR, an efficient context-harnessing framework for lightweight image super-resolution, which unifies multi-scale receptive field modeling and hierarchical context fusion. EchoSR decouples feature learning into disentangled local, multi-scale, and global modeling stages through an efficient context-harnessing strategy, and further promotes seamless cross-scale integration via a cross-scale overlapping fusion mechanism. Extensive experiments have shown that EchoSR consistently outperforms state-of-the-art lightweight super-resolution methods across multiple
benchmarks, while also achieving a faster speed $(\sim 2\times)$. The source code is available at https://github.com/funnyWang-Echoes/EchoSR.
Read the full abstract ▾
All authors Hanli Zhao, Binhao Wang, Shihao Zhao, Tao Wang, Kaihao Zhang, Wanglong Lu
Computer Vision
GCE-MIL: Faithful and Recoverable Evidence for Multiple Instance Learning in Whole-Slide Imaging
Xiangyu Li and Ran Su
Multiple instance learning (MIL) is the standard approach for whole-slide image (WSI) classification and survival prediction, where
attention-based models ag gregate patch features into slide-level predictions. These models treat
attention weights as evidence for their predictions, but
attention is optimized for classi fication, not for identifying which patches actually support the diagnosis. This conflation leads to three failures: selected patches are insufficient (keeping them alone drops Macro-F1 by 0.078), unnecessary (removing them barely changes the prediction), and unrecoverable (continuous
attention scores disagree with discrete patch subsets used at inference). The central premise is that evidence quality should be optimized directly through explicit criteria- Sufficiency, Necessity, and Recov erability (S/N/R)- rather than inherited as a byproduct of classification. GCE-MIL is a backbone-agnostic wrapper implemented through three injection modes and three evidence components: a grounding mechanism that aligns selection with domain-specific concepts, noisy-OR coverage that acts as a differentiable proxy for interventional evidence search, and threshold-plus-repair recovery that converts continuous selectors into discrete subsets through marginal-guided repair. Across 9 backbones and 9
datasets (81 configurations), GCE-MIL improves average Macro-F1 by 0.024 and C-index by 0.014, reduces the continuous-discrete gap by 4-7, and increases complement degradation by 2-4. With optional tile prefiltering after discrete recovery, inference runs up to 5 faster while retaining 0.989 full-bag utility.
Read the full abstract ▾