Computer Vision · Cross-listed from cs.CL
From Pixels to Pairs: A Comprehensive Benchmark of LLM-Driven Key-Value Extraction in Noisy Document Settings
Zahra Anvari
Large
language models (LLMs) have demonstrated strong capabilities in document key-value pair (KVP) extraction, yet controlled evaluations of their robustness to optical character recognition (OCR) output remain limited. This leaves an important gap in understanding their reliability in real-world OCR-to-LLM pipelines. Unlike end-to-end
Vision-
Language Models (VLMs), which jointly perform visual perception and semantic extraction, modular pipelines allow these stages and their errors to be isolated and audited. We introduce a controlled
benchmark that distinguishes downstream LLM extraction behavior from upstream OCR degradation. It evaluates 136 experimental configurations and 17,688 document-level inferences generated with deterministic decoding across five instruction-tuned open-weight LLMs (2B-8B parameters), three
datasets, and four text-quality conditions. The evaluation combines a full zero-shot comparison, targeted one- to three-shot experiments, and a sensitivity analysis of 40 configurations across 20 frozen demonstration sets. By separating Key Recall (annotated-field recovery) from Exact Match and Value F1 (exact and partial value recovery, respectively), we test whether OCR degradation affects field identification and value reproduction differently across models. Our findings challenge three practical assumptions: (1) clean-text performance reliably predicts real-world robustness, (2) model rankings remain consistent across annotation-derived Gold and OCR-derived text, and (3) additional few-shot demonstrations monotonically improve extraction accuracy. The observed model-ranking reversals and unstable few-shot gains expose important reliability risks under noisy document conditions. We release the benchmarking framework,
dataset splits, and evaluation scripts to support reproducible research.
Read the full abstract ▾
Computer Vision
xperception – Making Robotic Grasping Easier
Matteo Bortolon, Andrea Caraffa, Alice Fasoli et al. (+1 more)
The transition toward high-mix low-volume manufacturing demands flexibility in robotic manipulation. However, conventional
vision systems remain a bottleneck, requiring extensive data collection and model retraining whenever a new object is introduced to the production line. To overcome this
rigidity, we present xperception, a zero-shot 6D pose estimation technology that eliminates the need for object-specific fine-tuning and laborious data annotation. By directly utilizing typical CAD models and integrating the rich semantic features of foundation models (e.g. DINOv2, GeDi), xperception achieves millimeter-accurate 6D pose estimation. xperception showed robustness against severe occlusions in industrial tasks like bin picking and is engineered for deployment on industrial edge hardware, such as NVIDIA Jetson Thor. Validated at a TRL of 6, the core methodology behind xperception is based on the FreeZe
algorithm, which won the international BOP Challenge 2024, paving the way for scalable, plug-and-play robotic automation in unstructured high-mix low-volume manufacturing industries.
Read the full abstract ▾
All authors Matteo Bortolon, Andrea Caraffa, Alice Fasoli, Fabio Poiesi
Computer Vision
Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
Jingyu Sun, Jiachen Tu, Yuyang Xue et al. (+8 more)
Vision-
language models (VLMs) often answer visual questions using learned
language and category priors rather than grounding their predictions in the image itself. Counterfactual images provide a natural diagnostic setting for this failure mode: when visible evidence contradicts what is usually true, a grounded model should answer from the pixels, while a prior-following model will produce a canonical but visually incorrect response. However, existing counterfactual
benchmarks mainly ask whether such prior-following behavior exists. In this paper, we ask a further question motivated by the rise of tool-augmented and agentic
vision systems: can additional visual evidence views help VLMs reason against their priors? We introduce PriVE-Bench, a Prior-vs-Visual Evidence
Benchmark that uses paired original and counterfactual images to distinguish visually grounded answers from prior-consistent errors. We further introduce PriVE-Tools, a controlled agentic-
vision-inspired extension that evaluates whether tool-derived visual evidence – including bounding boxes, crops, zoom panels, and contours – improves grounding under the same counterfactual conflicts. Across open- and closed-source VLMs, we compare raw, paired-image, and tool-conditioned inputs using accuracy, prior-following error rate, and other-response rate. Our results show that visual evidence tools can help in some settings, especially when models can use localized evidence effectively, but they are not a universal remedy: several models continue to follow
language and category priors even when relevant visual evidence is explicitly provided.
Read the full abstract ▾
All authors Jingyu Sun, Jiachen Tu, Yuyang Xue, Yaoxin Jiang, Guoyi Xu, Zhengtao Yao, Rui Qian, Yizheng Sun, Hongpeng Zhou, Jingyuan Sun, Yan Lin
Computer Vision
SaaF: Scene-Specific Ambiguity-Aware 3D Language Fields towards Interactive Real-World Object Retrieval
Yuga Yano, Daiju Kanaoka, Hakaru Tamukoh et al. (+1 more)
We propose Scene-specific Ambiguity-aware 3D
Language Fields (SaaF), a novel Gaussian Splatting-based 3D
language field designed for interactive object
retrieval in a given real-world scene. Interactive object
retrieval using natural
language is a crucial capability for service
robots operating in complex real-world environments. While recent 3D
language field methods for object
retrieval establish associations between rendered pixels and autoencoder-compressed CLIP features, they suffer from two limitations: (1) reduced discriminability among similar objects due to feature compression, and (2) poor handling of ambiguous queries, often resulting in unstable or incorrect
retrieval. To address these limitations, SaaF introduces a metric learning strategy to construct a unified feature space that is both instance-discriminative and ambiguity-aware. (i) To enhance instance-level visual discrimination, SaaF employs metric learning that pulls image features from multiple viewpoints of the same object closer together in the feature space. (ii) To establish ambiguity awareness, the model jointly trains on multiple text labels generated by the proposed method from each tracked object image sequence, including ambiguous descriptions, to learn the semantic relationships between ambiguous and specific features in a target scene. This feature space enables fine-grained visual understanding while allowing the system to estimate query ambiguity and interactively request clarification when needed. Experimental results demonstrate that SaaF not only improves
retrieval accuracy over previous methods but also robustly detects and handles ambiguity in the user text queries under open-vocabulary settings.
Read the full abstract ▾
All authors Yuga Yano, Daiju Kanaoka, Hakaru Tamukoh, Yasutomo Kawanishi
Computer Vision
DAUPNet: Domain-Aware Uncertainty Modeling for Reliable Prototype Discrimination in Cross-Domain Few-Shot Semantic Segmentation
Lei Yuan, Zhongxu Hu, Jingyi Wen et al. (+1 more)
Cross-domain few-shot semantic
segmentation (CD-FSS) has predominantly been formulated as learning domain-invariant representations or improving support-query correspondence. Nevertheless, large domain shifts still make prototype matching unreliable: inconsistent hierarchical responses corrupt the support representation, deterministic prototypes cannot express boundary and appearance ambiguity, and treating prototypes with different reliability equally during optimization weakens foreground-background separation. We therefore propose DAUPNet, a unified framework that reformulates cross-domain prototype matching as uncertainty-aware prototype discrimination. DAUPNet first harmonizes hierarchical support-query features to provide stable evidence, then represents foreground and background prototypes probabilistically, and finally uses their estimated uncertainty to regulate contrastive optimization. On four standard target domains, DAUPNet achieves 72.6% and 76.7% average mIoU in the 1-shot and 5-shot settings, respectively, including substantial gains on the two medical domains. These results demonstrate that modeling prototype uncertainty and incorporating it into optimization provides a robust and interpretable approach to CD-FSS under severe domain shift. The code is available at https://github.com/madness-Lei/DAUPNet
Read the full abstract ▾
All authors Lei Yuan, Zhongxu Hu, Jingyi Wen, Pengxing Yi
Computer Vision
CoBind: Stage-Aware Compositional Binding for Training-Free Text-to-Image Generation
Kaijie Chen, Ethan Caldwell, Mira Vossen et al. (+3 more)
Diffusion-based text-to-image models often fail on complex
prompts involving multiple entities, attributes, and relations, producing object omissions, incorrect attribute assignments, or reversed spatial layouts. Existing training-free methods mainly strengthen
token-level
attention, but do not explicitly model which attributes belong to which entities or when different constraints should be enforced during denoising. We introduce CoBind, a training-free framework for stage-aware compositional binding. CoBind parses a
prompt into a composition graph of entities, attributes, and relations. It first establishes the global layout using entity-completeness and relation constraints, then binds attributes to their target entities through contrastive cross-entity optimization. Structural guidance is gradually relaxed in later denoising steps to preserve textures and visual details. CoBind also adapts the guidance strength according to the current satisfaction of each constraint, reducing unnecessary latent updates. CoBind requires no retraining or additional annotations. Experiments on T2I-CompBench++, GenEval, and multiple
diffusion backbones show consistent improvements in attribute binding, spatial relations, and complex compositional generation while maintaining competitive visual quality.
Read the full abstract ▾
All authors Kaijie Chen, Ethan Caldwell, Mira Vossen, Julian Hartwell, Serena Whitlock, Adrian Bellamy
Computer Vision
Zeyu Xu, Xingzhong Hou, Pengkai Guo et al. (+6 more)
Vision-
Language Models (VLMs) have achieved strong progress in
multimodal understanding. However, scaling dense or sparse Mixture-of-Experts (MoE) models to improve performance limits deployment in resource-constrained environments due to the trade-off between high
memory usage from full loading and increased latency from on-demand loading. Recently, the Per-Layer
Embedding (PLE) architecture addresses this by scaling models with large external
embedding tables stored in read-only
memory (ROM) and performing lightweight lookup to retrieve relevant
embeddings to enhance
token representations. Nevertheless, existing PLE-style methods are primarily designed for text
embeddings due to the convenience of ID-based
retrieval, limiting their effectiveness in VLMs where
multimodal embeddings contain richer information for visual tasks. In this paper, we propose LookME, the first framework that enables lookup-based enhancement for
multimodal embeddings in VLMs while supporting partitioned storage and on-demand loading. To efficiently lookup arbitrary continuous
multimodal embeddings from large-scale
embedding tables, we propose a hierarchical two-level lookup method employing a coarse-to-fine strategy that performs lookups from the scene-level to the intra-scene primitive-level. Furthermore, we integrate the lookup method with a sparse injection strategy, which adaptively prioritizes critical
embeddings over voluminous
multimodal embeddings within layers, and facilitates
embedding table reuse across neighboring layers, improving the trade-off among efficiency, model size, and performance. Experiments on multiple visual
benchmarks show that LookME outperforms text-only PLE-style methods, validating the effectiveness of lookup-based
multimodal embedding enhancement.
Read the full abstract ▾
All authors Zeyu Xu, Xingzhong Hou, Pengkai Guo, Siling Lin, Xiao Xu, Menghua Zhai, Haoyu Chen, Yunke Zhang, Fei Huang
Computer Vision
Dual-Domain Self-Supervised Artifact Removal Framework for Photoacoustic Computed Tomography
Yucheng Zhou, Shuang Li, Yu Zhang et al. (+4 more)
Photoacoustic Computed
Tomography (PACT) often faces severe challenges from reconstruction artifacts due to sparse detection conditions. In this work, based on the distinct differences in artifact patterns between back-projection-based and
Fourier-based reconstruction
algorithms, we propose a self-supervised artifact removal framework that employs a lightweight Siamese Neural Network and a composite loss function integrating cross-domain fidelity and uncertainty-weighted consistency, effectively decoupling dual-domain features and filtering artifacts. Comprehensive validations using simulations, phantoms, in vivo rat and human experimental data demonstrate that the proposed method can significantly suppress image artifacts. Furthermore, enabled by the acceleration of the spatial-domain and frequency-domain inverse operator, this end-to-end approach also achieves exceptional computational efficiency.
Read the full abstract ▾
All authors Yucheng Zhou, Shuang Li, Yu Zhang, Yibing Wang, Chulhong Kim, Seongwook Choi, Changhui Li
Computer Vision
Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillation
Yunhang Qian, Jiaquan Yu, Jiawei Liu et al. (+3 more)
Medical
Vision-
Language Models (Med-VLMs) require reliable
reasoning from fine-grained visual evidence, yet existing models can produce plausible clinical answers by relying on
language priors or medical templates rather than truly attending to diagnosis-critical regions. On-Policy
Distillation (OPD) offers dense
token-level supervision on student-generated trajectories and provides a
privacy-compatible means of capability transfer without requiring the redistribution of raw patient data. However, standard OPD uniformly distills all
tokens, causing sparse evidence-dependent
tokens to be diluted by abundant clinical narrative
tokens. Inspired by the success of OPD in the large
language model community, we propose Med-OPD, to our knowledge the first unified post-training framework that integrates on-policy
distillation with medical evidence-aware supervision for Med-VLMs. We introduce Medical Evidence Advantage (MEA), a teacher-grounded counterfactual signal that uses an answer-aware hint to focus teacher scoring on evidence supporting the target diagnosis, and measures each
token's dependence on medical visual evidence by comparing teacher likelihoods under the original and evidence-degraded imaging modalities. Based on MEA, Med-OPD redistributes the
distillation signal at both the
token and trajectory levels, emphasizing diagnosis-critical
tokens and evidence-reliant rollouts. Experiments on OmniMedVQA subsets show that Med-OPD consistently outperforms SFT and standard OPD across CT, MRI, Disease Diagnosis, and Lesion Grading. These results demonstrate that evidence-aware
distillation can better strengthen medical VLMs' reliance on key visual evidence and improve reliable
multimodal medical
reasoning. The source code and data is publicly available at: https://github.com/yunhang8658/MedOPD.git
Read the full abstract ▾
All authors Yunhang Qian, Jiaquan Yu, Jiawei Liu, Meng Wang, Hongwei Bran Li, Xiaobin Hu
Computer Vision · Cross-listed from cs.LG
QFireNet: A Quantum-Enhanced U-Net for Wildfire Segmentation from Sentinel-2 Imagery
Jaiman Munshi, Tanvi Tewary, Sawyer Bloom et al. (+10 more)
Wildfire detection from satellite imagery is a semantic image
segmentation problem that has proven to be difficult due to challenges such as class imbalance, feature complexity, and atmospheric interference. In this paper, we build on the foundational U-Net image
segmentation model to develop a
quantum-hybrid solution in hopes of more effectively modeling the high-dimensional spectral feature space of the Sen2Fire
dataset. We inject a variational
quantum circuit in the bottleneck portion of U-Net, specifically the QuFeX and QB-Net ansatzes. We test a classical Feature Pyramid Network (FPN) for further comparative analysis of the model, and we also explore classical improvements to the U-Net model and its training process, including a compression of parameters, alternative loss functions, and uniform mixing of input data. Our primary finding is that under matched conditions, both QB-Net (with an $F_1$ score of 31.18) and QuFeX ($F_1 = 30.79$) outperformed the classical U-Net baseline results ($F_1 = 28.71$). Additionally, the classical FPN achieved a comparable score of 31.13. A crucial finding was that data mixing removed a significant domain shift between the geographically-separated train and test sets, which boosted the classical FPN $F_1$ score to 39.76. We validate the architecture's robustness and generalizability to the wildfire detection problem via cross-
dataset transfer on the California Burned Areas (CaBuAr)
dataset. Overall, we find that
quantum machine learning has potential to provide an advantage in the problem of wildfire image
segmentation, and further experiments will continue to validate and expand upon this finding.
Read the full abstract ▾
All authors Jaiman Munshi, Tanvi Tewary, Sawyer Bloom, Aidan Chu, Chetan Maviti, Kyon Winston-Bey, Harshit Badjatia, Farhan Kittur, Vardhan Madhavarapu, Varun Kota, Joshua Kwon, Nazia Rangwala-Vohra, Franz Klein
Computer Vision · Cross-listed from cs.AI
ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capability
Weiting Liu, Jieyi Bi, Wanqi Zhou et al. (+4 more)
Tool-augmented large
language model
agents excel at long-horizon tasks, yet they are typically post-trained on fixed toolsets. When tasks demand new tools, these
agents struggle to incorporate them effectively, and retraining from scratch is often impractical. We identify the core obstacle in such toolset expansion problem as behavioral inertia: the tendency of
agents to fall back on familiar tools and established
reasoning patterns despite having access to new ones. We demonstrate that injecting counterfactual anchor contexts at critical decision points can break this inertia, recovering failed trajectories by eliciting suppressed
agent capabilities. To scale this insight, we propose ToolAnchor, a framework that uses teacher models to hypothesize these counterfactual contexts, verifies them via student rollouts, and internalizes the successful interventions through agentic post-training. Extensive evaluations across general AI assistant (
GAIA), textual search (BrowseComp), and visual search (VDR-Bench) tasks demonstrate that ToolAnchor consistently exhibits competitive performance under expanded toolsets. Our work bridges the gap between static post-training and dynamic adaptation, charting a new path for scalable agentic reinforcement learning.
Read the full abstract ▾
All authors Weiting Liu, Jieyi Bi, Wanqi Zhou, Jianfeng Feng, Yining Ma, Ai Han, Wenlian Lu
Computer Vision
Delving into the Temporal Challenges of Unified Video Protection Against Image-to-Video and Fine-Tuning-based Customization
Yuxin Huang, Ziming Hong, Mingming Gong et al. (+3 more)
Recent
diffusion-based
video generation models have enabled high-quality personalized
video customization through both tuning-based pipelines, which fine-tune a
video diffusion model, and reference-based pipelines such as image-to-
video generation. However, these capabilities raise serious concerns about personal
privacy, identity ownership and intellectual property protection. Existing anti-customization works focus on protecting images, while protection for
videos against both reference- and tuning-based customization remains largely underexplored. Protecting
videos in this setting raises three challenges: (i) Image-level perturbations, optimized frame by frame, cannot survive temporal compression by 3D
video VAE. (ii) A
video-level perturbation optimized on a single
video is vulnerable to temporal editing and fails to protect unseen
videos. (iii) Temporally inconsistent perturbations are not robust to temporal attacks. To address these challenges, we propose Temporally Consistent Universal
Adversarial Perturbations (TC-UAP), the first protection method against both reference- and tuning-based
video customization. TC-UAP optimizes an identity-level multi-frame UAP over sliding windows from multiple
videos, accounting for local temporal dependencies induced by temporal compression in
video VAE and enabling a single perturbation to protect unseen
videos of varying lengths. Moreover, we introduce intrinsic temporal modeling and an extrinsic surrogate temporal-attack loss, which make the perturbation temporally consistent and robust to unseen temporal attacks. Empirically, quantitative and qualitative results show that TC-UAP achieves the strongest identity protection compared with existing methods under both reference- and tuning-based
video customization, and remains robust under multiple unseen temporal attacks.
Read the full abstract ▾
All authors Yuxin Huang, Ziming Hong, Mingming Gong, Wanyu Wang, Jing Zhang, Tongliang Liu
Computer Vision
SARFA: Segment Anything with Radiomic Feature Alignment
Tyler Ward and Abdullah Imran
The Segment Anything Model (SAM) has demonstrated strong generalizability across a variety of
segmentation tasks. However, SAM often struggles in situations where the target to be segmented is ambiguous. This poses a problem in medical imaging, where accurate delineation of targets such as tumors is vital, but even expert radiologists can disagree on the appropriate boundary for a target. Addressing this, we propose SARFA (Segment Anything with Radiomic Feature
Alignment), a novel framework for improved medical image
segmentation. Via probabilistic prompting, SARFA generates a diverse set of plausible masks for each input image and optimizes them with a radiomics-driven training objective based on Fréchet Radiomic Distance (FRD) and Direct Preference Optimization (DPO). By minimizing the FRD between masked predicted and ground truth regions within each image, SARFA encourages
segmentation outputs whose anatomical and textural characteristics align with clinically meaningful ground truth representations, without relying solely on pixel-level overlap. Evaluated on computed
tomography (CT) and
magnetic resonance imaging (MRI)
benchmarks, SARFA outperforms existing ambiguous
segmentation methods, demonstrating the effectiveness of radiomic feature
alignment and DPO-style candidate mask ranking as a training objective. Our code is available at https://github.com/tbwa233/SARFA.
Read the full abstract ▾
Computer Vision
Reflecting Process Expertise in Procedural Material Generation
Kunal Gupta, Gaurav Joshi, Yen-Ru Chen et al. (+3 more)
Procedural material creation underpins applications in digital content creation, visual effects, and 3D asset design. Achieving high-quality results requires more than reproducing node graphs – it demands understanding the process by which experts construct materials. We formulate procedural material generation as
retrieval-time process
reasoning over expert demonstrations, elevating process to a first-class representation beyond graph-only synthesis. Concretely, we represent expert workflows as process traces: textual records of construction steps, parameters, and design intent. To instantiate this idea, we use a pretrained LLM-based ProcessSynthesizer to synthesize a process trace aligned with a user's intent and a pretrained LLM-based Compiler to ground the process trace into an executable Blender material graph. Because procedural expertise is most naturally conveyed through demonstrations, we leverage tutorial
videos as a source of process knowledge and extract textual, LLM-compatible traces using automated
video analysis tools. In an expert study with five Blender artists (avg. 7.5 years of experience), materials generated by reflecting expert demonstrations were found to produce workflows requiring fewer edits, and more closely match professional design strategies than methods operating solely on static artifacts. A user study with 150 participants further shows that our approach achieves superior generation and editing performance compared to prior procedural systems. All code, models, and data will be available at https://materialapprentice.github.io
Read the full abstract ▾
All authors Kunal Gupta, Gaurav Joshi, Yen-Ru Chen, Seemandhar Jain, Ishit Mehta, Manmohan Chandraker
Computer Vision
Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks
Jae Joong Lee
Benchmark accuracy in
video large
language models (LLMs) is often treated as evidence of visual understanding. We audit this assumption across twenty models spanning 2-78B parameters and ten architecture families. We introduce the Visual Dependency Gap (VDG), the difference in per-question correctness between original-
video and black-screen conditions. Paired McNemar tests on MVBench show that accuracy and visual dependency are separable: models differ on original
video (p = 0.0003) but not on black screens (p = 0.53). Across models, task-type rankings are stable: Attribute Perception is strongly visual, whereas Temporal
Reasoning approaches the
language-only baseline. A diagnostic ladder from black screen to single frame, shuffled frames, and original
video reveals that frame diversity supplies most of the visual benefit, while temporal order contributes near-zero accuracy across sixteen open-weight models. An ablation from 0.5 to 24 FPS rules out sparse sampling as the cause. H.264 experiments further show that stable aggregate accuracy conceals bidirectional question-level answer flips. The diagnostic also generalizes to four API-accessed models, whose VDG values range from 0.025 to 0.315. These results motivate VDG as a standard audit for whether
video benchmarks measure visually grounded capability. Code is available at https://github.com/JaeLee18/accuracy-without-grounding.
Read the full abstract ▾
Computer Vision
Improving Medical Image Generative Models with Fréchet Distance Loss
Andrew Marshall, Xuanang Xu, Xiaoran Zhang et al. (+3 more)
Diffusion generative models have demonstrated immense potential for synthetic medical image generation. However, these models often struggle to capture complex morphological characteristics of heterogeneous tumors with irregular boundaries, limiting their utility for downstream clinical tasks such as
segmentation. This limitation stems from the standard denoising objective: minimizing a per-pixel error, which smooths high-variance irregular structures characteristic of tumors. To address this, we propose finetuning these generative models with Fréchet Distance loss (FD-loss). FD-loss aligns the first and second order feature statistics of real and generated images in a pretrained encoder space, encouraging the generator to capture complex structural variations characteristic of heterogeneous tumors. We integrate FD-loss across diverse architectural settings, using both natural- and medical-image encoders on multiple liver and
brain cancer
datasets spanning CT and MRI modalities. Downstream
segmentation networks trained on our FD-regularized synthetic data consistently achieve superior performance, improving tumor DSC by $>$$5\%$ over unregularized synthetic augmentation alone. Qualitative analysis suggests these gains are associated with more faithful tumor synthesis and fewer
segmentation hallucinations. Our results show FD-loss as an effective regularizer for medical image generative models to improve clinical workflows.
Read the full abstract ▾
All authors Andrew Marshall, Xuanang Xu, Xiaoran Zhang, Rui Wang, Lawrence Staib, James Duncan
Computer Vision
FOLIO: Focused Semantic Memory for Streaming Video Understanding
Haoyang Fan, Dhruv Parikh, Anvitha Ramachandran et al. (+4 more)
In online streaming
video understanding, a
video stream continues to arrive and queries may be issued at any time. Because streaming frames grow without bound, the system must continuously compress and retain information from the observed
video prefix while future frames and future queries remain unknown. The core challenge is deciding what information to retain and how to organize the maintained history: as this history grows with the stream,
memory cost increases and many redundant visual details are retained, whereas later queries often depend on specific entities, actions, and their temporal changes. To address this challenge, we introduce FOLIO, a training-free focused semantic
memory system that records important parts of the stream in higher detail while keeping surrounding context compact. As the stream arrives, FOLIO updates
memory at the segment level, guided by a dynamic focus state, combining a short-term visual buffer with a long-term semantic
memory organized around observed entities and linked to a visual-evidence cache. At query time, lightweight hybrid
retrieval combines direct matching over the structured
memory with semantic query expansion. FOLIO achieves state-of-the-art performance, reaching 82.0/69.1 Perception/Backward accuracy on OVO-Bench with Qwen3-VL-8B and 74.5 overall accuracy on StreamingBench, while substantially reducing the cost of maintaining streaming
memory by reserving detailed records for focused entities and storing surrounding context compactly.
Read the full abstract ▾
All authors Haoyang Fan, Dhruv Parikh, Anvitha Ramachandran, Sameh Gobriel, Nilesh Jain, Rajgopal Kannan, Viktor Prasanna
Computer Vision
Differentiable Polarized Path Tracing
Pramod Rao, Jérémy Riviere, Xilong Zhou et al. (+6 more)
Physically based differentiable rendering has proven to be a powerful tool for inverse rendering problems (e.g., 3D reconstruction, reflectance estimation, lighting estimation). However, most existing methods operate solely on radiometric intensity, discarding valuable polarization cues that constrain scene geometry and material properties. While forward simulation of polarized light is well-defined via Mueller-Stokes calculus, extending reverse-mode differentiation to this domain presents significant challenges. The rank-deficient nature of common polarimetric operators, such as linear polarizers and diffuse reflections, violates the invertibility assumptions of standard gradient
estimators like path replay backpropagation and results in numerical instability. We address this by proposing a robust, polarization-aware differentiable path tracing method. Our approach estimates unbiased gradients through a combination of path replay and local caching. This formulation enables efficient and stable optimization of material and lighting parameters in complex scenes, broadening the applicability of physically based inverse rendering. Project page: https://vcai.mpi-inf.mpg.de/projects/DPPT/
Read the full abstract ▾
All authors Pramod Rao, Jérémy Riviere, Xilong Zhou, Abhijeet Ghosh, Abhimitra Meka, Thabo Beeler, Marc Habermann, Christian Theobalt, Delio Vicini
Computer Vision
AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow
Salah Eddine Bekhouche, Abdellah Zakaria Sellam, Fadi Dornaika et al. (+1 more)
We present AffectFlow-DINO, a multi-task learning system for the 11th ABAW challenge that extends a standard deterministic architecture with a conditional rectified-flow head to model the inherent ambiguity of in-the-wild facial behavior. Instead of predicting a single affect estimate, the model learns a conditional generative distribution, enabling uncertainty-aware one-to-many predictions through Monte Carlo sampling. The system jointly estimates continuous valence-arousal, classifies eight facial expressions, and detects twelve Action Units from static face images. Built on a frozen DINOv3 ViT-S/16 backbone, extensive ablation studies show that rectified-flow decoding consistently improves deterministic prediction, particularly for valence-arousal estimation (CCC-V $+0.058$). We further show that post-hoc threshold calibration effectively recovers performance on severely imbalanced rare classes (e.g., Fear: $3.8\% \rightarrow 33.1\%$) without retraining. Combined with backbone fine-tuning and flow retuning, the final model achieves $\mathbf{P_{MTL}=1.177}$, substantially outperforming the official challenge baseline of $P_{MTL}=0.45$.
Read the full abstract ▾
All authors Salah Eddine Bekhouche, Abdellah Zakaria Sellam, Fadi Dornaika, Abdenour Hadid
Computer Vision
Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Robotics
Yue Chang, Rufeng Chen, Yifan Tian et al. (+7 more)
While 3D Scene Graphs (3DSGs) provide crucial structured representations for
embodied agents, conventional Ahead-of-Time, "build-everything-then-filter" pipelines conflict with the real-time, low-latency demands of edge platforms, inducing a perceptual saturation effect via severe observation redundancy. To resolve this, we present JITOMA (Just-In-Time On-demand
Memory Activation), a closed-loop framework that unifies task
reasoning, perception, and
memory into a just-in-time growth process. Instead of exhaustively mapping the entire environment, JITOMA leverages a top-down task heatmap at the frontend to filter continuous observations, routing minimal streams to maintain a global foundation of low-cost, dormant anchors. Upon a cognitive query, the backend Large
Language Model (LLM) parses the robotic intent to dynamically awaken task-relevant anchors, triggering expensive semantic operations such as dense node captioning exclusively within the activated local subgraph. To evaluate these dynamic capabilities and study perceptual saturation trade offs, we introduce JITOMA-Bench, a
benchmark for long-horizon task switching and complex intent grounding. Across JITOMA-Bench, JITOMA maintains only 1–6 active semantic objects and 0.25–0.28 s/frame across all tiers, showing that semantic computation remains bounded by current task demand rather than accumulated scene complexity.
Read the full abstract ▾
All authors Yue Chang, Rufeng Chen, Yifan Tian, Dazhi Huang, Zhaofan Zhang, Yi Chen, Wenze Zhang, Li Chen, Hui Xiong, Sihong Xie
Computer Vision
Active Learning for Efficient Annotation of Surgical Videos with Weak Supervision
Manasa Dendukuri, Matjaz Jogan, Daniel A. Hashimoto et al. (+1 more)
Precise spatial-temporal annotation of laparoscopic
videos is time-consuming and requires expert knowledge. We propose a human-in-the-loop knowledge acquisition framework that combines active learning with dual-loss optimization to significantly reduce the annotation effort needed for automatic localization and
segmentation of objects in the
surgical field. Our method employs a foundation model to generate temporally consistent class activation maps (CAMs) from
video using two complementary training objectives: a weak supervision loss on
video-level tool presence labels for weakly annotated data, and an image-level mask loss on human-corrected annotations obtained through active learning. Rather than requiring dense pixel-level annotation upfront, our pipeline iteratively proposes pseudo-masks that guide the expert annotator to refine the knowledge previously captured by the model. We demonstrate that our framework reduces the effort of
surgical video annotation by 50% by the end of training in comparison to fully manual annotation. Through eliminating the need for large, fully annotated
datasets from the start, this framework enables scalability to the development of
surgical tool
segmentation models. This iterative human-in-the-loop refinement supports efficient knowledge acquisition with minimal expert input, providing a practical and deployable strategy for expanding tool
segmentation to larger, more diverse
datasets and real-world clinical settings.
Read the full abstract ▾
All authors Manasa Dendukuri, Matjaz Jogan, Daniel A. Hashimoto, Guiqiu Liao
Computer Vision
Continuously Evolving Deepfake Detection: An Architecture and Public-Benchmark Evaluation of a Dynamic Detection System
Ken Jon Miyachi and Dylan Uys
Deepfake detectors that achieve near-perfect scores on academic
benchmarks collapse on real-world content: recent in-the-wild evaluations report AUC drops of 45-50% for state-of-the-art open-source models. We argue this gap is structural: static detectors are trained once against a moving generative frontier. We present BitMind Forensics (BMF), trained through Bittensor SN34, an open
adversarial competition that continually refreshes the training distribution. We evaluate one dated export comprising image, general-
video, and human-
video checkpoints across nineteen public
datasets: the canonical face-swap suites (FaceForensics++, Celeb-DF v1/v2/++, DFDC, DFD, UADFV, DF40) and recent in-the-wild and AI-generated-media
benchmarks (Sumsub, Deepfake-Eval-2024, WildRF, Community Forensics, AIGCDetectBench, GenImage, AI-GenBench, AIGIBench, RAID, GenVidBench, GenVideo-100K). BMF reaches 0.936 AUC on Sumsub's original images and 0.872 pooled AUC over its full four-condition manipulation battery (1.4M images), staying robust under perturbation (0.855 JPEG, 0.799 downscaled), while GPEN enhancement improves detection (0.996). On Deepfake-Eval-2024, it matches the best commercial detector on images (0.915 vs 0.90) and exceeds it on
video (0.822 vs 0.79), far above the best open-source detectors (0.56 and 0.63). It reaches 0.991 AUC on a 21-generator AI-image panel and 0.918 on GenVidBench, and exceeds the FF++-trained frontier on DFDC (0.947 vs 0.843) and Celeb-DF v2 (0.9985 vs 0.956), both contamination-audited, with statistical parity on Celeb-DF++. In a temporal study, successive dated exports improve on held-out media from generators absent from the static baseline's training (image 0.842 to 0.902;
video 0.864 to 0.936). Our evaluation harness is public, and at publication the production API serves the exact evaluated snapshot for independent verification.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Efficient Computing for Medical Image Acquisition and Reconstruction
Xiao Wang, Jayasai Rajagopal, Md Safaiat Hossain et al. (+4 more)
Medical imaging systems such as CT, MRI, PET, and SPECT do not directly acquire images. Instead, they measure physical signals that encode anatomical or physiological information, and image reconstruction recovers the underlying image by solving an inverse problem. Although these imaging modalities are governed by different imaging physics, they share a common computational framework that naturally connects medical physics, linear
algebra, probability, numerical optimization, and efficient computing. As medical imaging systems acquire increasingly large and higher-dimensional
datasets, image reconstruction has become one of the primary computational bottlenecks in modern medical imaging. Advanced reconstruction methods, including analytical reconstruction, iterative optimization, and statistical model-based reconstruction, substantially improve image quality while reducing radiation dose or scan time, but at significantly increased computational cost. Efficient computing has therefore become essential for achieving clinically practical reconstruction times. This chapter presents a unified computational perspective on medical image acquisition and reconstruction across CT, MRI, PET, and SPECT. It first reviews the imaging physics and data acquisition process for each modality and derives a generalized mathematical framework for image reconstruction. Building on this framework, the chapter discusses analytical, iterative, and statistical reconstruction methods together with their computational characteristics. Finally, it examines efficient computing considerations, including optimization
algorithms, physics-aware forward operators,
memory-efficient implementations, and parallel computing strategies. Together, these topics demonstrate how the integration of imaging physics, mathematical modeling, and efficient computing enables accurate and scalable medical image reconstruction.
Read the full abstract ▾
All authors Xiao Wang, Jayasai Rajagopal, Md Safaiat Hossain, Peng Chen, Mohamed Wahib, Enzhi Zhang, Emma J. Reid
Computer Vision
Self-Supervised Visual Representation Learning: Pretrain-Finetuning or Joint Training?
Nusrat Munia, Tyler Ward, Nishat Nayla et al. (+2 more)
Self-supervision is a powerful technique for learning visual representations from unlabeled data. Existing techniques primarily adopt a two-stage approach for self-supervised learning (SSL): a pretraining stage on unlabeled data followed by a finetuning stage on labeled data. While this pipeline has demonstrated extreme effectiveness, the interaction between self-supervised and supervised learning objectives remains insufficiently understood. In this work, we systematically investigate whether jointly optimizing the self-supervised and supervised objectives during training provides a better alternative. We compare two training paradigms: (1) the aforementioned pretraining followed by finetuning (PFT) and (2) joint training (JT), where self-supervised and supervised losses are optimized simultaneously in the same network. Across eight representative SSL methods and diverse computer
vision tasks on natural, medical, crisis response, and remote sensing data, we evaluate performance under varying percentages of labeled data. Our results reveal that the relative effectiveness of PFT and JT depends strongly on the task at hand, the availability of labeled data, and the complexity of the domain. We find that JT consistently improves data and training efficiency while being robust in low-label settings, while PFT is more reliable in more specialized domains. We further analyze representation quality, robustness, and cross-domain generalization, providing new insights into how self-supervised and supervised objectives interact during optimization. We establish a comprehensive empirical
benchmark for hybrid SSL-based semi-supervised learning and offer practical guidance for selecting appropriate training strategies across diverse
vision applications.
Read the full abstract ▾
All authors Nusrat Munia, Tyler Ward, Nishat Nayla, Matthew A. Massey, Abdullah-Al-Zubaer Imran
Computer Vision
MGFace: Mask-Gated Face Matching via Conditional Similarity Routing
Huy Che, Hoang-Minh Trinh, Dinh-Duy Phan et al. (+1 more)
Face identification has achieved remarkable performance under normal conditions. Yet, its accuracy often degrades significantly when query faces are partially occluded, especially by facial masks. Existing re-ranking approaches improve robustness by exploiting patch-level similarities. Still, they often rely on costly, fine-grained matching mechanisms, which limit their efficiency in large-scale
retrieval scenarios. In this paper, we propose MGFace, a mask-gated face identification pipeline that predicts the mask status of a query face and conditionally routes the similarity computation accordingly. Specifically, MGFace distinguishes between masked and unmasked queries, applies global
embedding matching to unmasked queries, and activates mask-aware patch-level re-ranking only for masked queries. This design focuses on reliable upper-face regions while avoiding unnecessary fine-grained computation. Experiments on the extended LFW-Mask
dataset show that MGFace achieves over 80% identification accuracy with the FaceNet backbone and over 90% with the ArcFace backbone. Compared with a previous EMD-based re-ranking method, MGFace achieves better identification performance while reducing query time by approximately 20x. These results demonstrate the effectiveness of MGFace in improving masked-face identification accuracy with low computational overhead. The source code is available at https://github.com/chequanghuy/MGFace.
Read the full abstract ▾
All authors Huy Che, Hoang-Minh Trinh, Dinh-Duy Phan, Duc-Lung Vu
Computer Vision
A Masked Autoencoder Approach to Unsupervised Steel Surface Defect Recognition
Shrey Patel
Automated visual inspection of steel surface defects is a recurring quality control task in which labeled defect data is scarce and costly to obtain, while unlabeled surface images are abundant, which motivates self supervised methods that learn useful representations without class labels. A
Transformer based Masked Autoencoder is used here to learn representations of steel surface defects for unsupervised grouping. During pretraining, 75% of the input image patches are randomly masked, and a lightweight decoder reconstructs the masked regions from the visible 25%. The encoder is trained jointly with an auxiliary defect localization objective, used only as a training signal and not evaluated as a detector. The decoder reaches a structural similarity score of 0.92 and a mean squared error of 0.47. Features from the pretrained encoder are then clustered using UMAP for dimensionality reduction and Agglomerative clustering, reaching a Hungarian matched accuracy of 91.3% against the six known defect categories.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Text2Sign: A Single-GPU Diffusion Baseline for Text-to-Sign Language Video Generation
Ruize Xia
Sign
language is a primary communication channel for millions of Deaf and hard-of-hearing people, yet text-to-signer
video generation remains costly because
video diffusion models are expensive to train and evaluate. This paper presents Text2Sign, a text-conditioned
diffusion model for short sign-
language clips that runs on a single NVIDIA L4 GPU. It combines a frozen
vision-
language text encoder with a 3D encoder-decoder and factorized spatiotemporal
attention to reduce the cost of full-
video attention while preserving motion coherence. We compare convolution-only and
transformer-style backbones, frozen pretrained and task-specific text encoders, and factorized versus full
attention. On a signer-disjoint How2Sign split, the best short-run ablation reaches a validation loss of 0.0648, while a longer-run checkpoint reaches 0.00999. On a compact evaluation slice, the latter achieves an SSIM of $0.2403 \pm 0.0238$, a PSNR of $15.11 \pm 0.42$ dB, and temporal consistency of $1.0000 \pm 0.0000$ using 8-step DDIM sampling with a guidance scale of 5.0. It generates a 32-frame, $64 \times 64$ clip in 12.60 seconds, or 2.54 frames per second, with peak inference
memory of 3.12 GB. A held-out denoising audit shows only weak
prompt sensitivity: removing text increases late-timestep loss from 0.9875 to 0.9891, while shuffled
prompts perform similarly to correct
prompts. Frozen text conditioning therefore improves short-budget validation loss, but
prompt-specific separation remains limited. The system is restricted to low-resolution, short clips and lacks expert linguistic evaluation, so it should be viewed as a single-GPU research baseline rather than a complete sign-
language production system. Code is available at https://github.com/xiaruize0911/text2sign.
Read the full abstract ▾