Computer Vision · Cross-listed from cs.RO
Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration
Xinghao Zhu, Zixi Liu, Shalin Jain et al. (+18 more)
Dexterous
robot manipulation can benefit from the abundance of human demonstrations, but transferring such demonstrations to
robot policies remains challenging. We present Contact Wrench Guidance from Human Demonstration in Robotic Dexterous Manipulation (CHORD), a framework for long-horizon manipulation of rigid and articulated objects with reinforcement learning. The key idea is object-centric contact wrench space guidance: we represent human and
robot motions by the forces and torques they can induce on the object, enabling similarity to be measured by the induced instantaneous motions. This guidance makes reinforcement learning more scalable for contact-rich dexterous manipulation. We further introduce a large-scale simulation
benchmark with 4,739 bimanual dexterous manipulation tasks, constructed from motion-capture
datasets and reconstructed in-house
videos. Evaluated on 1,831
benchmark tasks, CHORD achieves an average success rate of 82.12%, demonstrating strong scalability. CHORD also generalizes to whole-body manipulation from hand-only and third-person demonstrations, achieving a 90.77% success rate, and the learned policies transfer to the real world in both open-loop and closed-loop settings.
Read the full abstract ▾
All authors Xinghao Zhu, Zixi Liu, Shalin Jain, Chenran Li, Milad Noori, Michael Andres Lin, Huihua Zhao, John Welsh, Mrinal Verghese, Wei Liu, Tingwu Wang, Xingye Da, Zhengyi Luo, Vishal Kulkarni, Naema Bhatti, Yuke Zhu, Linxi Fan, Bowen Wen, Danfei Xu, Soha Pouya, Yan Chang
Computer Vision
Data Provenance for Image Auto-Regressive Generation
Bihe Zhao, Louis Kerner, Michel Meintz et al. (+3 more)
Image autoregressive models (IARs) have recently demonstrated remarkable capabilities in visual content generation, achieving photorealistic quality and rapid synthesis through the next-
token prediction paradigm adapted from large
language models. As these models become widely accessible, robust data provenance is required to reliably trace IAR-generated images to the source model that synthesized them. This is critical to prevent the spread of misinformation, detect fraud, and attribute harmful content. We find that although IAR-generated images often appear visually identical to real images, their generation process introduces characteristic patterns in their outputs, which serves as a reliable provenance signal for the generated images. Leveraging this, we present a post-hoc framework that enables the robust detection of such patterns for provenance tracing. Notably, our framework does not require modifications of the generative process or outputs. Thereby, it is applicable in contexts where prior watermarking methods cannot be used, such as for generated content that is already published without additional marks and for models that do not integrate watermarking. We demonstrate the effectiveness of our approach across a wide range of IARs, highlighting its high potential for robust data provenance tracing in autoregressive image generation.
Read the full abstract ▾
All authors Bihe Zhao, Louis Kerner, Michel Meintz, Tameem Bakr, Franziska Boenisch, Adam Dziedzic
Computer Vision
SEMIR: Topology-Preserving Graph Minors for Thin-Structure Segmentation
Luke James Miller and Yugyung Lee
Thin-structure
segmentation–power lines, cracks, lane markings at 1-3 pixel width–requires preserving connectivity that standard representations preclude: patching severs continuous structures and conventional superpixels merge thin targets into background before classification.
Topology-aware losses penalize connectivity breaks at the objective level but cannot recover what the representation has already destroyed. We propose SEMIR, a framework that replaces the pixel
lattice with a parameterized graph minor whose contraction map preserves thin-structure connectivity under the contraction criterion. The minor collapses millions of pixels into tens or hundreds of boundary-aligned supernodes, enabling full-resolution inference without patching at scales demonstrated up to 21 MP in this paper; a lightweight GNN classifies the reduced graph and an exact map lifts predictions to pixel resolution. One pipeline–identical architecture, features, loss, and GNN hyperparameters across all
dataset–matches or exceeds domain-specific baselines on TTPLA (power lines), CrackSeg9k (pavement cracks), and SkyScapes Lane (aerial markings) on Dice, IoU, and Boundary F1 while reducing mask fragmentation by at least 4.6x relative to SLIC at matched inference.
Read the full abstract ▾
Computer Vision
Saar Huberman, Ron Mokady, Or Patashnik et al. (+1 more)
In modern generative models, images are specified and controlled through text
prompts. In practice, images are generated from sequences of
tokens derived from these
prompts. However, the space of
token sequences lacks a consistent accessible structure: semantically similar images may correspond to sequences that differ in wording, ordering, and placement of concepts, while similar
token sequences may encode very different semantics. This apparent lack of structure makes it difficult to perform smooth transitions in this space, hindering applications such as image blending and continuous control of edits. We argue that this limitation stems not from the absence of semantic structure, but from misalignment between representations. To address this misalignment, we introduce
Token-to-
Token alignment, a framework that establishes explicit semantic correspondence between
tokens across
prompts. Our approach transforms
prompts into a structured representation in which semantically corresponding concepts are mapped to consistent positions across
prompts, and then aligns their
token embeddings based on semantic similarity. Concretely, the method consists of two stages: a structural
alignment that rephrases
prompts into a shared structured form, followed by an
embedding-level
alignment that matches
token representations across
prompts. With this
alignment in place, simple linear interpolation becomes a meaningful operation, producing smooth and coherent semantic transitions and enabling applications such as blending and continuous editing. Our results show that text
embedding spaces in text-to-image models implicitly encode a continuous semantic structure that becomes accessible once representations are properly aligned, suggesting that semantic control can be achieved by organizing existing representations rather than modifying the generative model.
Read the full abstract ▾
All authors Saar Huberman, Ron Mokady, Or Patashnik, Daniel Cohen-Or
Computer Vision · Cross-listed from cs.LG
Cyclic Denoising Reveals Ultrastable Memories in Diffusion Models
Rishabh Sharma and Stefano Martiniani
We introduce cyclic denoising – repeated forward and reverse
diffusion at controlled noise amplitudes – as an extraction attack for image
diffusion models. Inspired by random organization in disordered solids, cyclic denoising exposes regions of the learned distribution that are largely inaccessible to standard sampling. The dynamics drive samples toward attractors with a broad stability spectrum. The deepest attractors are ultrastable: they regenerate after near-total corruption and persist through thousands of noising-denoising cycles. Many of these attractors correspond to memorized training images, including stock photographs, brand watermarks, and web-crawl artifacts. The attack requires only sampler-level control, with no gradients, weight inspection,
prompts, captions, or prior knowledge of the training data. Unlike generate-and-filter attacks, which rely on large-scale prompted generation and post-hoc similarity or membership-inference filtering, our main protocol is fully unconditioned. We demonstrate the phenomenon in Stable
Diffusion v1.4 and in a pixel-space DDPM, showing consistent behavior across latent- and pixel-space
diffusion models. Across noise amplitudes, we observe a yielding-like transition: low-amplitude cycling produces trivial absorbing fixed points or limit cycles, while larger amplitudes induce rearrangements, basin hopping, and long-lived trapping in structured memorized attractor basins. We also observe hierarchical partial absorption,
prompt-stabilized basins, and cross-initial-condition universality of the recovered attractor set. Our results therefore show that cyclic denoising is both a physics-inspired probe of generative landscapes and a practical tool for memorization
auditing, with implications for
privacy, copyright compliance, and model fingerprinting.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy
Amirhossein Kardoost, Lion Gleiter, Tingying Peng et al. (+1 more)
Self-supervised learning in fluorescence microscopy often relies on 2D projections, despite the inherently three-dimensional nature of
cells. We present a systematic comparison of 2D and 3D masked autoencoders (MAE-2D vs. MAE-3D) on volumetric microscopy data. Under matched architectures and training protocols, MAE-3D consistently outperforms 2D max-projection and slice-based variants on downstream single-
cell tasks. We further align visual representations with a pretrained
protein language model (ESM2) and show that cross-modal supervision yields larger gains for volumetric models. Channel cross-
attention and frequency-domain regularization are critical for leveraging 3D spatial context. On
protein–
protein interaction prediction, our best model achieves a ROC–AUC of 0.86, while on
protein localization it reaches an AUC$_{\text{micro}}$ of 0.95 and an F1$_{\text{micro}}$ of 0.74, demonstrating competitive performance on both tasks. Overall, our findings highlight the potential of volumetric modeling and
multimodal alignment for representation learning in single-
cell microscopy.
Read the full abstract ▾
All authors Amirhossein Kardoost, Lion Gleiter, Tingying Peng, Carsten Marr
Computer Vision
DivRL: Disentangled Self-Similarity Rewards for Diverse Subject-Driven Generation
Qian Wang, Zhenyu Li, Abdelrahman Eldesokey et al. (+1 more)
Subject-driven image generation faces an "Identity-Diversity Paradox", where strong identity preservation often leads to rigid and low-diversity outputs. We propose a post-training framework called DivRL that jointly optimizes identity consistency and structural diversity simultaneously by leveraging disentangled visual features from a robust similarity model. Specifically, we introduce a Negative Self-Similarity Measure (nSSM) to quantify structural diversity, and Visual Semantic Matching (VSM) to evaluate identity consistency. We propose an "Explore-and-Suppress" strategy that treats VSM as a gated constraint: the model freely explores structurally diverse configurations, and only samples that violate the identity threshold are penalized via a quadratic hinge loss. This converts identity preservation from a competing objective into a feasibility constraint, allowing nSSM and VSM to improve jointly. Experiments demonstrate that our method effectively pushes the model to generate both consistent and diverse images and improves structural diversity while maintaining comparable identity consistency through a gated optimization formulation.
Read the full abstract ▾
All authors Qian Wang, Zhenyu Li, Abdelrahman Eldesokey, Peter Wonka
Computer Vision
Trustworthy Image Authentication using Forensic Knowledge Graphs
Tai D. Nguyen and Matthew C. Stamm
Advances in generative AI have made image falsification highly realistic, demanding trustworthy authentication systems. Existing forensic detectors can target certain forgery types but lack interpretability, while
vision-
language models (VLMs) provide explanations but cannot exploit forensic traces for reliable detection. We propose Forensic Knowledge Graphs (FKGs), a unified framework that integrates forensic evidence extraction, structured
reasoning, and human-interpretable explanation. Our FKG structure encodes forensic traces along with their causal dependencies and links to scene content. To generate accurate FKGs, we introduce a novel forensic authentication network and an Iterative Context Refinement strategy that guides VLMs to produce faithful, grounded explanations. We also present FKG-50K, a
dataset of 50,000 realistic forgeries with ground-truth FKGs. Experiments demonstrate that FKG outperforms both forensic detectors and VLMs in detection, forgery identification and localization, and forensic justification.
Read the full abstract ▾
Computer Vision
Ahmad Algadhi, Ahmed Alzuhair, Omar Alkhulaif et al. (+1 more)
Prompt distillation compresses large
vision-
language models (VLMs) such as CLIP into lightweight student models by matching teacher predictions on unlabeled domain images. PromptKD (CVPR 2024) established this paradigm with a single PromptSRC-finetuned ViT-L/14 teacher and a ViT-B/16 student. We propose TheProfessor, a multi-teacher extension that distills from a fixed two-teacher ensemble: a domain-finetuned PromptSRC ViT-L/14 teacher and a zero-shot EVA-CLIP-L/14 teacher whose logits are pre-computed per
dataset. We evaluate single-teacher PromptKD, equal-probability ensembling, and confidence-weighted ensembling on four base-to-novel
datasets: Caltech-101, DTD, UCF101, and EuroSAT. In a 12-run single-seed sweep, confidence-weighted ensembling improves average HM from 87.52 to 89.28 (+1.77 points), while equal averaging improves average HM to 88.88 (+1.37 points). Gains are
dataset dependent: they are negligible on Caltech-101 (+0.16 HM for confidence weighting), modest on UCF101 (+0.62), and largest on domain-shifted EuroSAT (+5.78). These results update our earlier Caltech-only analysis and show that multi-teacher
prompt distillation is most useful when the second teacher contributes complementary supervision under domain shift.
Read the full abstract ▾
All authors Ahmad Algadhi, Ahmed Alzuhair, Omar Alkhulaif, Muzammil Behzad
Computer Vision
REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs
Yifei Zhao, Qian Lou and Mengxin Zheng
Vision-
language models (VLMs) are increasingly used as perception-
reasoning backbones for
embodied intelligence in safety-critical physical systems, where perception or
reasoning errors can lead to unsafe decisions or actions. Although many red-teaming methods have been developed to probe VLM vulnerabilities, their evaluation remains fragmented across
datasets, metrics, and threat models, making direct comparison difficult and obscuring whether observed differences arise from stronger attacks, more vulnerable models, or incompatible evaluation settings. Existing chatbot-centric red-teaming
benchmarks mainly standardize jailbreak and content-safety evaluation, but they do not systematically capture physically grounded functional failures or cover red-teaming methods that target physical-world VLMs. This raises the key challenge of comparing diverse attack methods under a unified protocol while targeting the same scenario-specific failures. We introduce REALM, to our knowledge the first unified red-teaming
benchmark for physical-world VLMs. REALM integrates 12 red-teaming methods, 3 model-agnostic defenses, and 13 VLMs under a practical black-box threat model with shared
datasets and metrics. To align
adversarial objectives across attack families, REALM introduces an agentic target-generation pipeline that constructs shared, scenario-specific, and physically grounded attack objectives for each scene, enabling fair comparison of diverse red-teaming methods under aligned
adversarial goals. Our evaluation shows that text and typographic injection attacks induce the most failures,
multimodal co-optimization yields the strongest visual-perturbation transfer, single-pass attacks approach iterative methods at much lower cost, and model scale alone does not confer
adversarial robustness. Code is available at https://github.com/UCF-ML-Research/REALM.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
E-MRL: Cross-view Aligned Evidence-driven Multimodal Reinforcement Learning for Reliable 3D Tumor Analysis
Sijing Li, Zhongwei Qiu, Zhuoya Wang et al. (+6 more)
While
Vision-
Language Models (VLMs) show great promise in volumetric medical report generation, they frequently suffer from visual
hallucinations and a lack of grounding in 3D CT data. Current Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) strategies typically optimize text fidelity alone, essentially rewarding correct diagnoses derived from
language priors rather than genuine visual perception. To address this, we propose cross-view aligned Evidence-driven
Multimodal Reinforcement Learning (Evidence-MRL, noted as E-MRL), a reliable RL
reasoning framework that formulates the generation process as a Markov Decision Process of "diagnosis-localization-verification". Unlike standard approaches, our model is explicitly trained to identify a "key evidence slice" alongside the global diagnostic report, grounding its findings in verifiable visual evidence. Crucially, we introduce a novel cross-view consistency reward, which validates the semantic
alignment between the golden-standard report and a local visual re-query of the selected key slice, providing additional rewards for correctly-localized
reasoning. Experiments on large-scale 3D CT tumor
datasets demonstrate that E-MRL significantly reduces
hallucinations and improves diagnostic accuracy compared to SFT and RL baselines, offering a clinically interpretable solution for visually-grounded and tumor analysis.
Read the full abstract ▾
All authors Sijing Li, Zhongwei Qiu, Zhuoya Wang, Boxiang Yun, Zhenyu Yi, Jianwei Xu, Wenqiao Zhang, Yingda Xia, Ling Zhang
Computer Vision
Mind the Heads: Topological Representation Alignment for Multimodal LLMs
Davide Caffagni, Alberto Compagnoni, Federico Melis et al. (+5 more)
Representation
alignment has emerged as an effective approach to improve
Multimodal Large
Language Models (MLLMs) by regularizing their internal representations toward those of an external
vision encoder. However, existing methods typically align a fixed layer of the
language backbone, overlooking the fine-grained structure of
Transformer models. In this work, we propose Head-Wise Representation
Alignment (HeRA), a method that enforces cross-modal
alignment at the level of individual
attention heads. Our approach is grounded in the Platonic Representation Hypothesis, focusing on preserving the topological structure of representations (i.e., their local neighborhood relationships) across modalities. Following the Mutual K-Nearest Neighbor (MKNN)
alignment metric, we introduce a contrastive objective that acts as a differentiable proxy for matching local structures. HeRA applies this objective during
multimodal training to specific
attention heads in the LLM, selected by their
alignment score according to the MKNN metric. Counterintuitively, we find that aligning the least aligned heads yields the largest gains. Extensive evaluations across multiple MLLMs and 18
benchmarks demonstrate that HeRA consistently improves performance on challenging
vision-centric tasks and serves as an effective regularizer against visual
hallucinations by naturally curbing the over-reliance on linguistic priors. Our code is publicly released.
Read the full abstract ▾
All authors Davide Caffagni, Alberto Compagnoni, Federico Melis, Sara Sarto, Pier Luigi Dovesi, Mark Granroth-Wilding, Marcella Cornia, Lorenzo Baraldi
Computer Vision · Cross-listed from cs.CL
Ground Then Rank: Revisiting Knowledge-Based VQA with Training-Free Entity Identification
Qian Ma, Qiong Wu, Zhengyi Zhou et al. (+1 more)
Knowledge-Based Visual Question Answering (KB-VQA) requires grounding visual queries to external knowledge beyond directly observable content in images. While recent multi modal large
language models (MLLMs) show strong perceptual abilities, they struggle on KB-VQA tasks requiring groundings from both fine-grained entity and evidence levels. Most existing multi-modal
retrieval augmented generation (MM-RAG) methods tightly couple entity discrimination and section-level evidence ranking into a single re-ranking stage, leading to high cost and limited generalization. In this work, we revisit existing MM-RAG solutions from a workflow perspective and argue both entity-level and fact-level groundings are key bottlenecks. We observe that although MLLMs often fail under open-ended entity naming, they can better identify the correct entity when selecting from a small set of candidate names. Based on this insight, we propose a simple and training-free identify-before-answer IBA framework that decouples entity identification from section-level re-ranking. Our approach
prompts an MLLM to select high-confidence entities using only candidate names, followed by an off-the-shelf textual re-ranker for evidence selection. Experiments on Encyclopedic-VQA and InfoSeek show that our method consistently outperforms fine-tuned multi-modal re-ranking baselines while reducing training and inference complexity. Additional analyses reveal that the improvements arise not only from better entity identification, but also from selecting more informative evidence once correct entity is fixed. Our implementation is made public to ease reproducibility.
Read the full abstract ▾
All authors Qian Ma, Qiong Wu, Zhengyi Zhou, Yao Ma
Computer Vision · Cross-listed from cs.LG
Machine Learning Modeling for Real-Time Melt Pool Monitoring in Laser Powder Bed Fusion Additive Manufacturing: A Hybrid Approach
Inioluwa Emmanuel, Zhuo Yang, Ho Yeung et al. (+1 more)
This work investigates the implementation of artificial intelligence and machine learning (AI/ML) for real-time monitoring in
laser powder bed fusion (LPBF) additive manufacturing. We developed a binary image classification framework for distinguishing normal and abnormal melt pool images using a balanced
dataset of 1,200 images collected from Nickel superalloy 625 on the NIST AMMT platform. The study evaluates accuracy and inference time based on control requirements and hardware limitations of open-architecture LPBF machines. We
benchmark three transfer learning architectures (ResNet50, EfficientNetB0, and MobileNetV2) against two Random Forest approaches: one trained on EfficientNetB0 feature
embeddings (hybrid) and one trained on raw pixel features (baseline). Images are stratified into 80/20 train-test splits, with a further 90/10 validation split on the training set, and undergo standardized resizing, normalization, and label-preserving data augmentation to emulate realistic process variability. Each model is evaluated using accuracy, precision, recall, F1 score, and area under the receiver operating characteristic curve (AUC), along with training time, inference latency, and CPU & GPU usage to capture deployability constraints relevant to factory-floor monitoring. The hybrid EfficientNetB0-plus-Random Forest approach achieves the best performance on the held-out test set, with an F1 score of 0.9451, accuracy of 0.9458, and AUC of 0.9904, while maintaining sub-millisecond per-image inference (1.15 ms). In contrast, purely deep learning models exhibit significantly higher inference times with lower accuracy. These results demonstrate that combining pre-trained convolutional features with classical ensemble methods provides a robust, computationally efficient route to real-time melt pool anomaly detection in data-limited additive manufacturing environments.
Read the full abstract ▾
All authors Inioluwa Emmanuel, Zhuo Yang, Ho Yeung, Xinyao Zhang
Computer Vision
Hoang-Bao Le, Aiden Durrant, Thai Son Mai et al. (+3 more)
Vision-
language models (VLMs) achieve strong cross-modal
alignment but remain brittle to negation, often relying on shallow word associations rather than compositional
reasoning. Fine-tuning on negation-specific data can also compromise their general purpose capabilities through catastrophic forgetting. We introduce HANCLIP (
Hyperbolic, Angular, and Negation), a geometry-aware framework that improves negation sensitivity while preserving the structure of the pretrained joint
embedding space. HANCLIP combines a
hyperbolic contrastive objective, which models hierarchical relations and semantic asymmetries, with an angular triplet loss that separates negated descriptions from their affirmative counterparts. Using only 20,000 image-text quadruplets, HANCLIP consistently improves performance across CLIP, LongCLIP, and SmartCLIP backbones on the NegBench
benchmark, while maintaining or improving zero-shot classification and image-text
retrieval performance. These results show that lightweight, geometry-guided objectives can enhance negation understanding without large-scale retraining.
Read the full abstract ▾
All authors Hoang-Bao Le, Aiden Durrant, Thai Son Mai, Binh T. Nguyen, Liting Zhou, Cathal Gurrin
Computer Vision
ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation
Anindya Mondal, Sauradip Nag and Anjan Dutta
We present ABACUS, a unified
vision-
language model that jointly addresses object counting, crowd counting, referring-expression counting, and count-faithful image generation within a single 3B-parameter model. ABACUS introduces three contributions: density-aware adaptive zooming paired with an objectness map from multi-head self-
attention decomposition to spatially ground count predictions; a boundary-aware count policy trained via GRPO with nested local, boundary, and global rewards to eliminate over- and undercounting at crop boundaries; and a cycle-consistent GRPO strategy in which the frozen understanding branch scores generated candidates on count-deviation and aesthetic quality, closing the understanding-generation synergy gap without any external critic or annotation. ABACUS achieves state-of-the-art results across seven
benchmarks spanning object counting (FSC-147, CARPK), crowd counting (ShanghaiTech A/B), referring-expression counting (REC-8K), count-faithful generation (CoCoCount, T2I-CompBench, GenEval), and count
reasoning (CountQA), surpassing both task-specific specialists and larger generalist models. Project page is at https://mondalanindya.github.io/pages/ABACUS.
Read the full abstract ▾
Computer Vision
From Spatial to Spectral: An Efficient, Frequency-Guided Feature Representation Learner for Small Object Detection
Yuhan Rui, Shihan Qiao, Yibin Lou et al. (+7 more)
Efficient small object detection is bottlenecked by the inherent feature scarcity of tiny targets, which is further aggravated by operations of spatial-domain detectors that indiscriminately discard critical high-frequency details. Recovering these fragile cues within the spatial domain is notoriously difficult, as it often requires computationally expensive architectural upscaling that inadvertently amplifies background noise. To bridge this gap, we propose a paradigm shift from spatial to spectral feature processing, introducing a holistic solution with the following novelty: (1) A versatile Frequency-Guided Feature Representation framework that generalizes across diverse detector architectures (both CNN and
Transformer-based), offering a robust alternative to spatial-only feature extraction; (2) The unified Decompose–Enhance–Reconstruct (DER) operator, instantiated via three lightweight, plug-and-play modules – Wavelet-Difference Gate (WDG), Log-Gabor Enhancer (LGE), and Frequency-Driven Head (FDHead) – to systematically inject frequency-aware modulation into the backbone, neck, and head. This mechanism decouples feature modeling from resolution reduction, capturing discriminative high-frequency components to enable accurate localization with significantly reduced parameter redundancy; (3) Extensive validation on multi-domain
benchmarks (VisDrone2019, UAVDT, TinyPerson, DOTAv1) demonstrating consistent gains. Notably, our proposed DERNet series outperforms YOLOv11 models under the same scale while requiring only 1/6 of the parameters, backed by rigorous spectral diagnostics and error decomposition analysis.
Read the full abstract ▾
All authors Yuhan Rui, Shihan Qiao, Yibin Lou, Mingxi Yu, Yutong Wan, Yanqiao Chen, Dongsheng Hou, Zhen Cao, Athena Zhuoming Zhong, Qi Hao
Computer Vision
Listening makes Vision Clear for VLMs
Yiyang Chen, Yixin Tan and Binrui Shen
Recent work typically assesses
vision–
language consistency using
attention distributions of answer-side
tokens. However, we observe that highest
attention regions are not always consistent with the intended semantic
token. This probably stems from decoding drift, where
language priors from previously generated answer
tokens accumulate and mismatch with visual
attention. Besides the priors from previous answer
tokens, we find that structural
tokens, e.g., modality boundary markers, may encompass the entire context and generate high
attention to areas unrelated to the target. To avoid these distortions and provide consistency evaluation for large VLMs, we adopt
prompt-side semantics and propose
Prompt-
Vision Token Activation Map (PV-TAM). PV-TAM further incorporates a filter to remove systematic bias induced by modality boundary markers. Unlike traditional methods that evaluate overlap solely through masks while ignoring activation intensity, our metrics leverage the peak distribution of
attention to measure the
alignment between
prompts and visual regions. In experiments, PV-TAM consistently improves both
attention-based and IoU-style localization metrics over answer-side baselines on various
datasets.
Read the full abstract ▾
Computer Vision
Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild
Yehonathan Litman, Xiaoxuan Ma, Manan Shah et al. (+4 more)
Reconstructing dynamic non-rigid objects from
monocular video requires integrating visual cues from direct observations with data-driven priors over geometry and appearance. Prior approaches either learn to directly predict 4D representations from visual input or initialize a 3D representation that is subsequently deformed and refined based on
video evidence. However, the former are constrained by the scarcity of 4D training data, while the latter leverage priors only for the initial reconstruction and rely solely on
video supervision thereafter; neither handles complex in-the-wild scenarios with large deformations and occlusions well. We present Lift4D, a test-time optimization framework that addresses both limitations. First, we adapt an existing single-view 3D reconstruction model to yield temporally consistent per-frame predictions via causal latent conditioning, providing a coherent initialization for a deformable 3D Gaussian Splatting representation. We then “sculpt” this representation to match the input
video through an occlusion-aware optimization that faithfully recovers visible surface details while completing unobserved regions using a view-conditioned
diffusion prior. We demonstrate that Lift4D clearly improves over prior 4D reconstruction methods, particularly on challenging in-the-wild sequences with severe occlusions and non-rigid motion.
Read the full abstract ▾
All authors Yehonathan Litman, Xiaoxuan Ma, Manan Shah, Nicolas Ugrinovic, Kris Kitani, Fernando De la Torre, Shubham Tulsiani
Computer Vision
Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping
Rishubh Parihar, Ayush Raina, R. Venkatesh Babu et al. (+1 more)
Reference-based
diffusion models enable highly controllable image generation by leveraging elements from input images to guide
prompt-driven synthesis. However, these models are computationally expensive in runtime, and their cost scales severely with the number of input references. While the efficiency of
diffusion models has been extensively studied in the context of
prompt-driven generation, it remains largely under-explored in the realm of reference-based models. This setting presents unique challenges not addressed by methods focusing solely on generation. In particular, the wasteful representation of references as dense
token grids offers significant opportunities for improvement. In this work, we present Sparse Context, a method for constructing sparse reference representations by retaining only a reduced subset of reference
tokens. We observe that even without modifying the model, dropping a significant portion of reference
tokens at inference time largely preserves its generation capabilities. To fully realize this potential, we fine-tune the model with random
token dropping at varying ratios, encouraging robustness to partial reference representations. Crucially, this training strategy decouples the model from any specific
token selection rule, allowing flexible control at inference time. At inference time, instead of random dropping, we apply task-aware
token selection strategies that prioritize the most informative regions of the reference images, adapting the
token budget to the input and task requirements. Extensive experiments show our method achieves a 4x increase in inference speed for multi-reference generation and an 2x for single reference generation. Importantly, this efficiency is achieved without compromising visual quality across both spatially-aligned editing and subject-driven generation.
Read the full abstract ▾
All authors Rishubh Parihar, Ayush Raina, R. Venkatesh Babu, Or Patashnik
Computer Vision
Semantic Browsing: Controllable Diversity for Image Generation
Sara Dorfman, Maya Vishnevsky, Omer Dahary et al. (+2 more)
Modern text-to-image models excel in visual fidelity and
prompt adherence. However, this strict adherence comes at the cost of diversity: generated samples tend to collapse into a single visual interpretation. Existing methods to improve diversity produce outputs driven by incidental variations rather than meaningful design choices. This motivates a new variant of the diversity task where structure is enforced on the generated samples. We introduce a method for controlled diversity that enables Semantic Browsing, where users can navigate structured image galleries and experience creative exploration through a systematic traversal of meaningful, interpretable axes of variation. Achieving this level of semantic control requires a deep understanding of the scene. We exploit the fact that recent text-to-image models are trained on elaborated captions, effectively decoupling semantic decision-making from pixel generation. This enables a paradigm shift: instead of relying on
stochastic variation within the text-to-image model, we induce diversity directly at the text level. By leveraging rich textual representations, we allow a
Vision Language Model (VLM) to operate on the full scene context. To overcome the generic outputs typical of standard VLMs, we employ an agentic workflow that explicitly enforces structured variation attuned to the original
prompt. We demonstrate that our method produces diverse and navigable design spaces where every variation corresponds to a specific, user-understandable semantic decision.
Read the full abstract ▾
All authors Sara Dorfman, Maya Vishnevsky, Omer Dahary, Or Patashnik, Daniel Cohen-Or
Computer Vision
AIR: Adaptive Interleaved Reasoning with Code in MLLMs
Cong Han, Xiaohan Lan, Haibo Qiu et al. (+1 more)
Following the paradigm shift initiated by OpenAI o3, interleaved
reasoning with code to enhance
multimodal large
language models (MLLMs) has become a pivotal research frontier. The existing literature focuses primarily on tool-use within
vision-perception tasks. However, such approaches typically rely on predefined heuristics for visual manipulation and are inherently incapable of addressing numerical computation problems due to their exclusive focus on visual operations. This paper empowers MLLMs with adaptive interleaved
reasoning capabilities through extended reinforcement learning training on code-augmented complex numerical computation tasks. To this end, we propose a comprehensive three-component solution consisting of: a two-stage cold-start data construction pipeline, data filtering strategies for RL
dataset curation, and an adaptive tool-invocation strategy leveraging a group-constrained reward function for interleaved
reasoning trajectories. Extensive experiments demonstrate that after Reinforcement Learning training with the group-constrained reward function, performance improves by an average of 6.1 percentage points (pp) on evaluation
benchmarks. Specifically, the accuracy for interleaved
reasoning samples increases by 9.9 pp, and the overall success rate of tool-use exceeds 95%. Our data and code are available at: https://github.com/CongHan0808/AIR.git.
Read the full abstract ▾
All authors Cong Han, Xiaohan Lan, Haibo Qiu, Yujie Zhong
Computer Vision
IMAGIN-4D: Image-Guided Controllable Interaction Generation
Sai Kumar Dwivedi, Federica Bogo, Buğra Tekin et al. (+6 more)
Generating human-object interactions (HOI) is central to character animation, robotics, AR/VR, and
embodied AI. Recent HOI generation methods synthesize motion from text, object geometry, and sparse waypoints, controlling action semantics and object trajectories. However, these signals underspecify interaction: the same
prompt and trajectory can produce different grasps, approach directions, body poses, object poses, contacts, and body-object layouts. We address this ambiguity with a reference image as a visual specification of the desired interaction snapshot. However, a single global image representation conflates distinct cues and conditions all frames on identical visual evidence. We therefore introduce IMAGIN-4D, a
diffusion-based HOI generator that decomposes image conditioning spatio-temporally. For spatial conditioning, IMAGIN-4D extracts supervised interaction-state
tokens for body pose, object pose, body-object contact, and spatial relationships at the depicted frame. For temporal conditioning, it computes frame-aware
tokens by querying image patches per generated frame, allowing sequence segments to attend to different visual cues from the same image. To balance image, text, and waypoint cues, IMAGIN-4D uses role-aware conditioning: text, waypoints, and interaction-state
tokens use separate AdaLN streams, while frame-aware visual
tokens cross-attend with motion
tokens. Since HOI motion
datasets lack paired images, we build a synthetic motion-to-image rendering pipeline from FullBodyManipulation (FBM) and introduce an image-adherence metric to evaluate whether generated motions match the reference snapshot. Experiments on FBM and BEHAVE show that IMAGIN-4D improves fine-grained interaction control over single-
token and uniformly image-conditioned baselines while preserving waypoint-following and motion quality. Code and models will be released at https://imagin4d.github.io.
Read the full abstract ▾
All authors Sai Kumar Dwivedi, Federica Bogo, Buğra Tekin, Chenhongyi Yang, Nadine Bertsch, Tomas Hodan, Michael J. Black, Dimitrios Tzionas, Shreyas Hampali
Computer Vision
GeoFidelity-Bench: Evaluating Block-Conditioned Geographic Fidelity in Text-to-Image Street-View Generation
Kaizhen Tan
Text-to-image models can generate plausible street scenes from
prompts such as "a street in New York City," but it remains unclear whether these images resemble the requested place or show a generic city appearance. Commonly used evaluation metrics mainly measure image quality or text
alignment and do not directly assess geographic fidelity. We introduce GeoFidelity-Bench, a
benchmark for evaluating whether generated street-view images match a target location at the level of named street blocks. The
benchmark contains 112 street blocks from 25 cities across six continents, each paired with a panel of real Mapillary images and geographic metadata. From this metadata we build
prompts with the city only, with street and neighborhood names, and with these names plus raw GPS coordinates. Across six open-weight text-to-image models, street and neighborhood names improve similarity to the target location, while raw GPS coordinates add only a small and uneven gain. Control
prompts that swap in the names of other blocks while keeping the city fixed reduce
retrieval accuracy, with larger decreases when the neighborhood name or both names are replaced. Generated images also distinguish their target blocks from local alternatives less reliably than held-out real images. CLIP
alignment between generated images and their
prompts is nearly uninformative about whether the images achieve block-level geographic fidelity. GeoFidelity-Bench provides a controlled testbed for studying location-conditioned generation and for measuring progress beyond generic city-level plausibility.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.AS
PHAST-Net: Attention-Guided, Physics-Informed Network for Estimation of Ideal Time-Frequency Representations
James M. Cozens and Simon J. Godsill
We introduce PHAST-Net, an
attention-guided, physics-informed network for unified estimation of Ideal Time-Frequency Representations (ITFRs), spanning spectral, tempo-based, metrical, and harmonic representations such as Spectrograms, Tempograms, and Metrograms. PHAST-Net learns a mapping from a constellation of wavelet transforms, specifically the proposed Continuous Log-frequency Adaptive Wavelet Transform (CLAWT), to high-resolution, cross-term-suppressed time-frequency representations. The proposed constellation of CLAWTs is strategically selected through Wigner-Ville kernel analysis to provide orientation-diverse coverage in a log-frequency time-frequency plane tailored to harmonic signal structure. PHAST-Net further incorporates a proposed physics-informed auxiliary reprojection loss designed to match the CLAWT-domain reprojections of the predicted and target ITFRs using the corresponding CLAWT kernels during training. This auxiliary objective promotes transform and energy-distribution consistency, mitigates target sparsity, and regularises optimisation.
Attention layers further promote effective cross-term suppression across the input constellation. The log-frequency formulation also enables Harmonic PHAST-Net, which estimates a Harmonic ITFR that isolates fundamental structure, supporting robust fundamental-only representations for
speech and music, such as derived fundamental Tempograms and Metrograms. We further introduce Spline-PHAST-Net, which parameterises detected and associated time-frequency ridges as continuous spline trajectories, enabling arbitrary-grid re-rendering and signal reconstruction. Trained on a procedurally generated
dataset, PHAST-Net demonstrates improved accuracy over representative time-frequency analysis methods, providing a unified framework for high-resolution, cross-term-robust analysis of
speech, music, and broader nonstationary signals.
Read the full abstract ▾
Computer Vision
Lightweight Neural Framework for Robust 3D Volume and Surface Estimation from Multi-View Images
Diego E. Farchione, Ramzi Idoughi and Peter Wonka
Accurate volume and surface area estimation is critical for diverse applications, from marine ecology to medical diagnostics. However, existing methods often suffer from high computational costs and poor performance with sparse and noisy data. We propose a fully feed-forward framework that regresses scale-normalized volume and surface area and their associated uncertainties directly from multi-view images. By fusing 3D point cloud reconstructions with view-aligned 2D features through a graph-based decoder, our model bypasses iterative optimization, ensuring exceptional scalability and rapid inference. Experimental results demonstrate that our approach outperforms state-of-the-art methods, particularly when operating with a low number of input images. Validated across coral monitoring, dietary analysis, and anthropometry, our proposed framework provides a robust, adaptable solution for quantitative shape analysis. This architecture provides a high-speed, scalable alternative for precise
geometric estimation from visual data, maintaining high performance even in resource-constrained or sparse-view scenarios.
Read the full abstract ▾