Computer Vision · Cross-listed from cs.HC
Adaptive Prompt Elicitation for Text-to-Image Generation
Xinyi Wen, Lena Hegemann, Xiaofu Jin et al. (+2 more)
Aligning text-to-image generation with user intent remains challenging, as users frequently provide ambiguous inputs and struggle with model idiosyncrasies. We propose Adaptive
Prompt Elicitation (APE), a technique that adaptively poses visual queries to help users refine
prompts without extensive writing. Our technical contribution is a formulation of interactive intent inference under an information-theoretic framework. APE represents latent user intent as interpretable feature requirements using
language model priors, adaptively generates visual queries, and compiles elicited requirements into effective
prompts. Evaluation on IDEA-Bench and DesignBench shows that APE achieves stronger
alignment with improved efficiency. A user study with 128 participants on user-defined tasks demonstrates 19.8% higher perceived
alignment without increased workload. Our work contributes a principled approach to prompting that offers an effective and efficient complement to the prevailing
prompt-based interaction paradigm with text-to-image models.
Read the full abstract ▾
All authors Xinyi Wen, Lena Hegemann, Xiaofu Jin, Shuai Ma, Antti Oulasvirta
Computer Vision
SAR-RAG: ATR Visual Question Answering by Semantic Search, Retrieval, and MLLM Generation
David F. Ramirez, Tim Overman, Kristen Jaskie et al. (+2 more)
We present a visual-context image-
retrieval-augmented generation (ImageRAG)- assisted AI
agent for automatic target recognition (ATR) of synthetic aperture radar (SAR) imagery. SAR is a remote sensing method used in defense and security applications to detect and monitor the positions of military vehicles, which may appear indistinguishable in images. Researchers have extensively studied SAR ATR to improve the differentiation and identification of vehicle types, characteristics, and measurements. Test examples can be compared with known vehicle target types to improve recognition tasks. New methods enhance the capabilities of neural networks,
transformer attention, and
multimodal large
language models. An agentic AI method may be developed to utilize a defined set of tools, such as searching through a library of similar examples. Our proposed method, SAR
Retrieval-Augmented Generation (SAR-RAG), combines a
multimodal large
language model (MLLM) with a vector database of semantic
embeddings to support contextual search for image exemplars with known qualities. By recovering past image examples of known true target types, our SAR-RAG system can compare similar vehicle categories, thereby improving ATR prediction accuracy. We evaluate this through search and
retrieval metrics, categorical classification accuracy, and numeric regression of vehicle dimensions. These metrics all show improvements when SAR-RAG is added to an MLLM baseline method as an attached ATR
memory bank.
Read the full abstract ▾
All authors David F. Ramirez, Tim Overman, Kristen Jaskie, Joe Marvin, Andreas Spanias
Computer Vision
Annotation Free Spacecraft Detection and Segmentation using Vision Language Models
Samet Hicsonmez, Jose Sosa, Dan Pineau et al. (+4 more)
Vision Language Models (VLMs) have demonstrated remarkable performance in open-world zero-shot visual recognition. However, their potential in space-related applications remains largely unexplored. In the space domain, accurate manual annotation is particularly challenging due to factors such as low visibility, illumination variations, and object blending with planetary backgrounds. Developing methods that can detect and segment spacecraft and orbital targets without requiring extensive manual labeling is therefore of critical importance. In this work, we propose an annotation-free detection and
segmentation pipeline for space targets using VLMs. Our approach begins by automatically generating pseudo-labels for a small subset of unlabeled real data with a pre-trained VLM. These pseudo-labels are then leveraged in a teacher-student label
distillation framework to train lightweight models. Despite the inherent noise in the pseudo-labels, the
distillation process leads to substantial performance gains over direct zero-shot VLM inference. Experimental evaluations on the SPARK-2024, SPEED+, and TANGO
datasets on
segmentation tasks demonstrate consistent improvements in average precision (AP) by up to 10 points. Code and models are available at https://github.com/giddyyupp/annotation-free-spacecraft-
segmentation.
Read the full abstract ▾
All authors Samet Hicsonmez, Jose Sosa, Dan Pineau, Inder Pal Singh, Arunkumar Rathinam, Abd El Rahman Shabayek, Djamila Aouada
Computer Vision
DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking
Sijia Chen, Shaofeng Liang, Lijuan Ma et al. (+7 more)
Referring Multi-Object Tracking (RMOT) aims to track specific targets based on
language descriptions, playing a vital role in interactive AI systems such as robotics and autonomous driving. However, existing RMOT methods rely exclusively on 2D RGB observations. Since similar 2D projections can correspond to different 3D configurations, the available observations may not uniquely determine the referred targets. This creates a fundamental mismatch between the task requirements and the available input. To advance research on the DRMOT task, we construct DRSet, a
dataset tailored for the DRMOT task. DRSet comprises 2,076 referring expressions over 186 RGB-D source sequences. It features synchronized RGB images and depth maps paired with
language descriptions, including explicit depth-dependent descriptions, to systematically evaluate spatial-semantic detection and tracking capabilities. Furthermore, we propose DRTrack, an MLLM-guided depth-referring tracking framework. DRTrack leverages a
Geometric-Aware GRPO fine-tuned
Multimodal Large
Language Model (MLLM) to achieve precise spatial-semantic detection from joint L+RGB+D inputs, while incorporating explicit depth cues to enforce robust trajectory association. Extensive experiments on the DRSet
dataset demonstrate the effectiveness of our framework. Our
dataset and code will be available at https://github.com/chen-si-jia/DRMOT.
Read the full abstract ▾
All authors Sijia Chen, Shaofeng Liang, Lijuan Ma, Yanqiu Yu, Nan Li, Wenshuo Chen, Runwei Guan, Dongming Wu, En Yu, Yutao Yue
Computer Vision · Cross-listed from cs.CL
Investigating Disability Representations in Text-to-Image Models
Yang Tian, Yu Fan, Liudmila Zavolokina et al. (+1 more)
Text-to-image generative models have made remarkable progress in producing high-quality visual content from textual descriptions, yet concerns remain about how they represent social groups. While characteristics like gender and race have received increasing
attention, disability representations remain underexplored. This study investigates how people with disabilities are represented in AI-generated images by analyzing outputs from Stable
Diffusion XL and DALL-E 3 using a structured
prompt design. We analyze disability representations by comparing image similarities between generic disability
prompts and
prompts referring to specific disability categories. Moreover, we evaluate how mitigation strategies influence disability portrayals, with a focus on assessing affective framing through sentiment polarity analysis, combining both automatic and human evaluation. Our findings reveal persistent representational imbalances and highlight the need for continuous evaluation and refinement of generative models to foster more diverse and inclusive portrayals of disability.
Read the full abstract ▾
All authors Yang Tian, Yu Fan, Liudmila Zavolokina, Sarah Ebling
Computer Vision · Cross-listed from cs.LG
REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency
Ondrej Tybl and Lukas Neumann
Knowledge
Distillation (KD) transfers knowledge from a large teacher model to a smaller student by aligning their predictive distributions. However, conventional KD formulations - typically based on Kullback-Leibler divergence - assume that the teacher provides reliable soft targets. In practice, teacher predictions are often noisy or overconfident, and existing correction-based approaches rely on ad-hoc heuristics and extensive hyper-parameter tuning, which hinders generalization. We introduce REDistill (Robust
Estimator Distillation), a simple yet principled framework grounded in robust statistics. REDistill replaces the standard KD objective with a power divergence loss, a generalization of KL divergence that adaptively downweights unreliable teacher output while preserving informative logit relationships. This formulation provides a unified and interpretable treatment of teacher noise, requires only logits, integrates seamlessly into existing KD pipelines, and incurs negligible computational overhead. Extensive experiments on CIFAR-100 and ImageNet-1k demonstrate that REDistill consistently improves student accuracy in diverse teacher-student architectures. Remarkably, it achieves these gains without model-specific hyper-parameter tuning, underscoring its robustness and strong generalization to unseen teacher-student pairs.
Read the full abstract ▾
Computer Vision
AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation
Jin-Chuan Shi, Binhong Ye, Tao Liu et al. (+6 more)
Reconstructing dynamic hand-object interactions from
monocular videos is critical for dexterous manipulation data collection and creating realistic digital twins for robotics and VR. However, current methods face two prohibitive barriers: (1) reliance on neural rendering often yields fragmented, non-simulation-ready geometries under heavy occlusion, and (2) dependence on brittle Structure-from-Motion (SfM) initialization leads to frequent failures on in-the-wild footage. To overcome these limitations, we introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning. First, we employ an agentic pipeline where a
Vision-
Language Model (VLM) guides a generative model to synthesize a complete, watertight object mesh with high-fidelity texture, independent of
video occlusions. Second, bypassing fragile SfM entirely, we propose a robust anchor-and-track strategy. We initialize the object pose at a single interaction onset frame using a foundation model and propagate it temporally by leveraging the strong visual similarity between our generated asset and
video observations. Finally, a contact-aware optimization integrates semantic,
geometric, and interaction stability constraints to enforce physical plausibility. Extensive experiments on HO3D, DexYCB, ARCTIC, and in-the-wild
videos reveal that AGILE outperforms baselines in global
geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior arts frequently collapse. By prioritizing physical validity, our method produces simulation-ready assets validated via real-to-sim retargeting for robotic applications. Project page: https://agile-hoi.github.io.
Read the full abstract ▾
All authors Jin-Chuan Shi, Binhong Ye, Tao Liu, Junzhe He, Yangjinhui Xu, Xiaoyang Liu, Zeju Li, Hao Chen, Chunhua Shen
Computer Vision
TRIO: Token Reduction via Inference-Objective Guidance for Efficient Vision-Language Models
Haokui Zhang, Congyang Ou, Dawei Yan et al. (+5 more)
Recently, reducing redundant visual
tokens in
vision-
language models (VLMs) to accelerate VLM inference has emerged as a hot topic. However, most existing methods rely on heuristics constructed based on inter-visual-
token similarity or cross-modal visual-text similarity, which gives rise to certain limitations in compression performance and practical deployment. In contrast, we propose TRIO from the perspective of inference objectives, which transforms visual
token compression into preserving output result invariance and selects
tokens primarily by their importance to this goal. Specifically,
vision tokens are reordered with the guidance of
token-level gradient saliency generated by our designed layer-local proxy loss, a coarse constraint from the current layer to the final result. Then the most valuable
vision tokens are selected following the non-maximum suppression (NMS) principle.The proposed TRIO is training-free and compatible with FlashAttention, friendly to practical application and deployment. It can be deployed independently as an encoder-free method, or combined with encoder compression approaches like VisionZip for use as an encoder-involved method. On LLaVA-Next-7B, TRIO retains just 11.1% of visual
tokens but maintains 97.2% of the original performance, with a 2.75$\times$ prefill speedup, 2.14$\times$ inference speedup, 6.22$\times$ lower FLOPs, and 6.05$\times$ reduced KV Cache overhead.Our code is available at https://github.com/ocy1/TRIO.
Read the full abstract ▾
All authors Haokui Zhang, Congyang Ou, Dawei Yan, Peng Wang, Qingsen Yan, Yu Zhang, Ying Li, Rong Xiao
Computer Vision
A labeled dataset of simulated phlebotomy procedures for medical AI: polygon annotations for object detection and human-object interaction
Raúl Jiménez Cruz, César Torres-Huitzil, Marco Franceschetti et al. (+3 more)
This data article presents a
dataset of 11,884 labeled images documenting a simulated blood extraction (phlebotomy) procedure performed on a training arm. Images were extracted from high-definition
videos recorded under controlled conditions and curated to reduce redundancy using Structural Similarity Index Measure (SSIM) filtering. An automated face-anonymization step was applied to all
videos prior to frame selection. Each image contains polygon annotations for five medically relevant classes: syringe, rubber band, disinfectant wipe, gloves, and training arm. The annotations were exported in a
segmentation format compatible with modern object detection frameworks (e.g., YOLOv8), ensuring broad usability. This
dataset is partitioned into training (70%), validation (15%), and test (15%) subsets and is designed to advance research in medical training automation and human-object interaction. It enables multiple applications, including phlebotomy tool detection, procedural step recognition, workflow analysis, conformance checking, and the development of educational systems that provide structured feedback to medical trainees. The data and accompanying label files are publicly available on Zenodo.
Read the full abstract ▾
All authors Raúl Jiménez Cruz, César Torres-Huitzil, Marco Franceschetti, Ronny Seiger, Luciano García-Bañuelos, Barbara Weber
Computer Vision
ImmuVis: Hyperconvolutional Foundation Model for Imaging Mass Cytometry
Dawid Uchal, Marcin Możejko, Krzysztof Gogolewski et al. (+12 more)
We present ImmuVis, a family of efficient foundation models for imaging mass cytometry (IMC), a high-throughput multiplex imaging technology that handles molecular marker measurements as image channels and enables large-scale spatial tissue profiling. Unlike natural images, multiplex imaging lacks a fixed channel space, as real-world marker sets vary across studies, violating a core assumption of standard
vision backbones. To address this, ImmuVis introduces marker-adaptive hyperconvolutions that generate convolutional kernels from learned marker
embeddings, enabling a single model to operate on arbitrary measured marker subsets without retraining. We pretrain ImmuVis on the largest
dataset to date, IMC17M (28 cohorts, 24,405 images, 265 markers, over 17M patches), using self-supervised masked reconstruction. ImmuVis outperforms state-of-the-art baselines and ablations in virtual staining and downstream classification tasks at substantially lower compute cost than
transformer-based alternatives, and is the sole model that provides calibrated uncertainty via a heteroscedastic likelihood objective. These results position ImmuVis as a practical framework for real-world IMC modeling.
Read the full abstract ▾
All authors Dawid Uchal, Marcin Możejko, Krzysztof Gogolewski, Piotr Kupidura, Szymon Łukasik, Jakub Giezgała, Tomasz Nocoń, Kacper Pietrzyk, Robert Pieniuta, Mateusz Sulimowicz, Michal Orzyłowski, Tomasz Siłkowski, Karol Zagródka, Eike Staub, Ewa Szczurek
Computer Vision
SalFormer360: a transformer-based saliency estimation model for 360-degree videos
Mahmoud Z. A. Wahba, Francesco Barbato, Sara Baldoni et al. (+1 more)
Saliency estimation has received growing
attention in recent years due to its importance in a wide range of applications. In the context of 360-degree
video, it has been particularly valuable for tasks such as viewport prediction and immersive content optimization. In this paper, we propose SalFormer360, a novel saliency estimation model for 360-degree
videos built on a
transformer-based architecture. Our approach is based on the combination of an existing encoder architecture, SegFormer, and a custom decoder. The SegFormer model was originally developed for 2D
segmentation tasks, and it has been fine-tuned to adapt it to 360-degree content. To further enhance prediction accuracy in our model, we incorporated Viewing Center Bias to reflect user
attention in 360-degree environments. Extensive experiments on the three largest
benchmark datasets for saliency estimation demonstrate that SalFormer360 outperforms existing state-of-the-art methods. In terms of Pearson Correlation Coefficient, our model achieves 8.4% higher performance on Sport360, 2.5% on PVS-HM, and 18.6% on VR-EyeTracking compared to previous state-of-the-art.
Read the full abstract ▾
All authors Mahmoud Z. A. Wahba, Francesco Barbato, Sara Baldoni, Federica Battisti
Computer Vision
PEPR: Privileged Event-based Predictive Regularization for Domain Generalization
Gabriele Magrini, Federico Becattini, Niccolò Biondi et al. (+1 more)
Deep neural networks for visual perception are highly susceptible to domain shift, which poses a critical challenge for real-world deployment under conditions that differ from the training data. To address this domain generalization challenge, we propose a cross-modal framework under the learning using privileged information (LUPI) paradigm for training a robust, single-modality RGB model. We leverage event cameras as a source of privileged information, available only during training. The two modalities exhibit complementary characteristics: the RGB stream is semantically dense but domain-dependent, whereas the event stream is sparse yet more domain-invariant. Direct feature
alignment between them is therefore suboptimal, as it forces the RGB encoder to mimic the sparse event representation, thereby losing semantic detail. To overcome this, we introduce Privileged Event-based Predictive Regularization (PEPR), which reframes LUPI as a predictive problem in a shared latent space. Instead of enforcing direct cross-modal
alignment, we train the RGB encoder with PEPR to predict event-based latent features, distilling robustness without sacrificing semantic richness. The resulting standalone RGB model consistently improves robustness to day-to-night and other domain shifts, outperforming
alignment-based baselines across object detection and semantic
segmentation.
Read the full abstract ▾
All authors Gabriele Magrini, Federico Becattini, Niccolò Biondi, Pietro Pala
Computer Vision
Understanding Degradation with Vision Language Model
Guanzhou Lan, Chenyi Liao, Yuqi Yang et al. (+5 more)
Understanding visual degradations is a critical yet challenging problem in computer
vision. While recent
Vision-
Language Models (VLMs) excel at qualitative description, they often fall short in understanding the parametric physics underlying image degradations. In this work, we redefine degradation understanding as a hierarchical structured prediction task, necessitating the concurrent estimation of degradation types, parameter keys, and their continuous physical values. Although these sub-tasks operate in disparate spaces, we prove that they can be unified under one autoregressive next-
token prediction paradigm, whose error is bounded by the value-space quantization grid. Building on this insight, we introduce DU-VLM, a
multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning using structured rewards. Furthermore, we show that DU-VLM can serve as a zero-shot controller for pre-trained
diffusion models, enabling high-fidelity image restoration without fine-tuning the generative backbone. We also introduce DU-110k, a large-scale
dataset comprising 110,000 clean-degraded pairs with grounded physical annotations. Extensive experiments demonstrate that our approach significantly outperforms generalist baselines in both accuracy and robustness, exhibiting generalization to unseen distributions.
Read the full abstract ▾
All authors Guanzhou Lan, Chenyi Liao, Yuqi Yang, Qianli Ma, Zhigang Wang, Dong Wang, Bin Zhao, Xuelong Li
Computer Vision
Nix and Fix: Targeting 1000x Compression of 3D Gaussian Splatting with Diffusion Models
Cem Eteke and Enzo Tartaglione
3D Gaussian Splatting (3DGS) revolutionized novel view rendering. Instead of inferring from dense spatial points, as implicit representations do, 3DGS uses sparse Gaussians. This enables real-time performance but increases space requirements, hindering rate-constrained applications. 3DGS compression emerged as a field aimed at alleviating this issue. While impressive progress has been made, at low rates, compression introduces artifacts that degrade visual quality significantly. We introduce NiFi, a method for extreme 3DGS compression through restoration via artifact-aware,
diffusion-based one-step
distillation. We show that our method achieves state-of-the-art perceptual quality at extremely low rates, down to 0.1 MB, and towards 1000x rate improvement over 3DGS at comparable perceptual performance. Code is available at: https://github.com/ceteke/nifi
Read the full abstract ▾
Computer Vision
OmniRad: A Radiological Foundation Model for Multi-Task Medical Image Analysis
Luca Zedda, Andrea Loddo and Cecilia Di Ruberto
Radiological analysis increasingly benefits from pretrained visual representations that can support heterogeneous downstream tasks across imaging modalities. In this work, we introduce OmniRad, a self-supervised radiological foundation model pretrained on 1.2 million medical images, designed with radiology-inspired principles emphasizing representation reuse and cross-task transferability. We evaluate the pretrained encoder under multiple downstream adaptation regimes, including lightweight task-specific adapters with a frozen backbone as well as full end-to-end fine-tuning for classification, allowing us to assess both representation quality and task-specific performance. OmniRad is evaluated on a broad suite of public
benchmarks spanning classification and
segmentation across multiple modalities. On the MedMNISTv2 collection, OmniRad improves classification F1 by up to 2.05% over competing foundation models. For dense prediction, OmniRad attains mean Dice score improvements across six MedSegBench
datasets when using frozen representations. Qualitative analyses and latent-space visualizations suggest improved feature clustering and modality-related separation.
Read the full abstract ▾
Computer Vision
SLUM-i: Semi-supervised Learning for Urban Mapping of Informal Settlements and Data Quality Benchmarking
Muhammad Taha Mukhtar, Syed Musa Ali Kazmi, Khola Naseem et al. (+5 more)
Very-high-resolution remote-sensing imagery provides a scalable basis for delineating informal settlements, but sparse annotations, severe imbalance between informal-settlement and background pixels, and cross-city heterogeneity in urban morphology and image–mask correspondence complicate model development. We present SLUM-i, a semi-supervised semantic
segmentation framework together with a geographically diverse seven-city Earth observation
benchmark spanning three continents. The
benchmark combines a newly annotated Lahore
dataset and companion Karachi and Mumbai
datasets with four publicly released city
datasets, totaling 14,458 RGB image–mask tiles. We quantify cross-city heterogeneity using class composition, boundary morphology, grayscale separability, correspondence between mask boundaries and image edges, and divergence between measured feature distributions. For label-efficient mapping, SLUM-i combines representation-guided unlabeled-pool curation, using
embeddings from a
vision foundation model (DINOv2-Small) to remove the least-similar tiles, and Class-Aware Adaptive Thresholding, which adapts pseudo-label acceptance by class through global mean-confidence and per-class mean-softmax exponential moving averages. Experiments at 10%, 20%, and 30% label budgets, using convolutional and
vision-
transformer backbones and five random seeds, demonstrate improvements over the corresponding UniMatch baselines in multiple city–budget settings, reaching +5.9 percentage points in mean intersection-over-union. At the 30% budget, the ResNet-101 configuration matches or exceeds its corresponding fully labeled supervised baseline in four of seven cities. Both components operate only during training and add no inference overhead.
Read the full abstract ▾
All authors Muhammad Taha Mukhtar, Syed Musa Ali Kazmi, Khola Naseem, Muhammad Ali Chattha, Andreas Dengel, Sheraz Ahmed, Muhammad Naseer Bajwa, Muhammad Imran Malik
Computer Vision
S-MUSt3R: Sliding Multi-view 3D Reconstruction
Leonid Antsfeld, Boris Chidlovskii, Yohann Cabon et al. (+2 more)
The recent paradigm shift in 3D
vision led to the rise of foundation models with remarkable capabilities in 3D perception from uncalibrated images. However, extending these models to large-scale RGB stream 3D reconstruction remains challenging due to
memory limitations. This work proposes S-MUSt3R, a simple and efficient pipeline that extends the limits of foundation models for
monocular 3D reconstruction. Our approach addresses the scalability bottleneck of foundation models through a simple strategy of sequence
segmentation followed by segment
alignment and lightweight loop closure optimization. Without model retraining, we benefit from remarkable 3D reconstruction capacities of MUSt3R model and achieve trajectory and reconstruction performance comparable to traditional methods with more complex architecture. We evaluate S-MUSt3R on TUM, 7-Scenes and proprietary
robot navigation datasets and show that S-MUSt3R runs successfully on long RGB sequences and produces accurate and consistent 3D reconstruction. Our results highlight the potential of leveraging the MUSt3R model for scalable
monocular 3D scene in real-world settings, with an important advantage of making predictions directly in the metric space.
Read the full abstract ▾
All authors Leonid Antsfeld, Boris Chidlovskii, Yohann Cabon, Vincent Leroy, Jerome Revaud
Computer Vision · Cross-listed from cs.RO
EgoActor: Grounding Task Planning into Spatial-aware Egocentric Actions for Humanoid Robots via Visual-Language Models
Yu Bai, MingMing Yu, Chaojie Li et al. (+3 more)
Deploying
humanoid robots in real-world settings is fundamentally challenging, as it demands tight integration of perception, locomotion, and manipulation under partial-information observations and dynamically changing environments. As well as transitioning robustly between sub-tasks of different types. Towards addressing these challenges, we propose a novel task - EgoActing, which requires directly grounding high-level instructions into various, precise, spatially aware
humanoid actions. We further instantiate this task by introducing EgoActor, a unified and scalable
vision-
language model (VLM) that can predict locomotion primitives (e.g., walk, turn, move sideways, change height), head movements, manipulation commands, and human-
robot interactions to coordinate perception and execution in real-time. We leverage broad supervision over
egocentric RGB-only data from real-world demonstrations, spatial
reasoning question-answering, and simulated environment demonstrations, enabling EgoActor to make robust, context-aware decisions and perform fluent action inference (under 1s) with both 8B and 4B parameter models. Extensive evaluations in both simulated and real-world environments demonstrate that EgoActor effectively bridges abstract task planning and concrete motor execution, while generalizing across diverse tasks and unseen environments.
Read the full abstract ▾
All authors Yu Bai, MingMing Yu, Chaojie Li, Ziyi Bai, Xinlong Wang, Börje F. Karlsson
Computer Vision
Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee et al. (+2 more)
Despite recent advancements in Multi-modal Large
Language Models (MLLMs) on diverse understanding tasks, these models struggle to solve problems which require extensive multi-step
reasoning. This is primarily due to the progressive dilution of visual information during long-context generation, which hinders their ability to fully exploit test-time scaling. To address this issue, we introduce
Vision-aligned Latent
Reasoning (VaLR), a simple, yet effective
reasoning framework that dynamically generates
vision-aligned latent
tokens before each Chain of Thought
reasoning step, guiding the model to reason based on perceptual cues in the latent space. Specifically, VaLR is trained to preserve visual knowledge during
reasoning by aligning intermediate
embeddings of MLLM with those from
vision encoders. Empirical results demonstrate that VaLR consistently outperforms existing approaches across a wide range of
benchmarks requiring long-context understanding or precise visual perception, while exhibiting test-time scaling behavior not observed in prior MLLMs. In particular, VaLR improves the performance significantly from 33.0% to 52.9% on VSI-Bench, achieving a 19.9%p gain over Qwen2.5-VL.
Read the full abstract ▾
All authors Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee, Minsu Cho, Jinwoo Shin
Computer Vision
CC-Pan: Channel-wise Compression based Diffusion for Efficient Pan-Sharpening
Junjie Li, Congyang Ou, Haokui Zhang et al. (+3 more)
Recently,
diffusion models have brought novel insights to pan-sharpening and notably boosted fusion precision. However, most existing models perform
diffusion in the pixel space and train distinct models for different multispectral (MS) sensors, suffering from high inference latency and sensor-specific limitations. In this paper, we present CC-Pan, a cross-sensor latent
diffusion framework for efficient pan-sharpening. Specifically, CC-Pan trains a band-wise single-channel variational autoencoder (VAE) to encode high-resolution multispectral (HRMS) images into compact latent representations, naturally supporting MS images with varying band counts across different sensors and establishing a basis for inference acceleration. Spectral physical properties, along with PAN and MS images, are then injected into the
diffusion backbone through carefully designed unidirectional and bidirectional interactive control structures, achieving high-precision spatial–spectral fusion in the latent
diffusion process. Furthermore, a lightweight region-based cross-band
attention (RCBA) module is incorporated at the central layer of the
diffusion model, reinforcing inter-band spectral connections to boost spectral consistency and further elevate fusion precision. Extensive experimental results on GaoFen-2, QuickBird, and WorldView-3 demonstrate that CC-Pan outperforms state-of-the-art
diffusion-based methods across all three
benchmarks, attains a $2$–$3\times$ inference speedup, and exhibits robust cross-sensor generalization capability on the held-out WorldView-2 sensor without any sensor-specific retraining.
Read the full abstract ▾
All authors Junjie Li, Congyang Ou, Haokui Zhang, Guoting Wei, Shengqin Jiang, Ying Li
Computer Vision
Temporal Slowness in Central Vision Drives Semantic Object Learning
Timothy Schaumlöffel, Arthur Aubret, Gemma Roig et al. (+1 more)
Humans acquire semantic object representations from
egocentric visual streams with minimal supervision, but the underlying mechanisms remain unclear. Importantly, the visual system only processes the center of its field of view with high resolution and it learns similar representations for visual inputs occurring close in time. This emphasizes slowly changing information around gaze locations. This study investigates the role of central
vision and slowness learning in the formation of semantic object representations from human-like visual experience. We simulate five months of human-like visual experience using the Ego4D
dataset and a state-of-the-art gaze prediction model. We extract image crops around predicted gaze locations to train a time-contrastive Self-Supervised Learning model. Our results show that exploiting temporal slowness when learning from central visual field experience improves the encoding of different facets of object semantics. Specifically, focusing on central
vision strengthens the extraction of foreground object features, while considering temporal slowness, especially in conjunction with eye movements, allows the model to encode broader semantic information about objects. These findings provide new insights into the mechanisms by which humans may develop semantic object representations from natural visual experience. Our code will be made public upon acceptance. Code is available at https://github.com/t9s9/central-
vision-ssl.
Read the full abstract ▾
All authors Timothy Schaumlöffel, Arthur Aubret, Gemma Roig, Jochen Triesch
Computer Vision
Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search
Tianming Liang, Qirui Du, Jian-Fang Hu et al. (+3 more)
Segmentation based on
language has been a popular topic in computer
vision. While recent advances in
multimodal large
language models (MLLMs) have endowed
segmentation systems with
reasoning capabilities, these efforts remain confined by the frozen internal knowledge of MLLMs, which limits their potential for real-world scenarios that involve up-to-date information or domain-specific concepts. In this work, we propose Seg-ReSearch, a novel
segmentation paradigm that overcomes the knowledge bottleneck of existing approaches. By enabling interleaved
reasoning and external search, Seg-ReSearch empowers
segmentation systems to handle dynamic, open-world queries that extend beyond the frozen knowledge of MLLMs. To effectively train this capability, we introduce a hierarchical reward design that harmonizes initial guidance with progressive incentives, mitigating the dilemma between sparse outcome signals and rigid step-wise supervision. For evaluation, we construct OK-VOS, a challenging
benchmark that explicitly requires outside knowledge for
video object
segmentation. Experiments on OK-VOS and two existing
reasoning segmentation benchmarks demonstrate that our Seg-ReSearch improves state-of-the-art approaches by a substantial margin. Code and data will be released at https://github.com/iSEE-Laboratory/Seg-ReSearch.
Read the full abstract ▾
All authors Tianming Liang, Qirui Du, Jian-Fang Hu, Haichao Jiang, Zicheng Lin, Wei-Shi Zheng
Computer Vision
SynthVerse: A Large-Scale Diverse Synthetic Dataset for Point Tracking
Weiguang Zhao, Haoran Xu, Xingyu Miao et al. (+11 more)
Point tracking aims to follow visual points through complex motion, occlusion, and viewpoint changes, and has advanced rapidly with modern foundation models. Yet progress toward general point tracking remains constrained by limited high-quality data, as existing
datasets often provide insufficient diversity and imperfect trajectory annotations. To this end, we introduce SynthVerse, a large-scale, diverse synthetic
dataset specifically designed for point tracking. SynthVerse includes several new domains and object types missing from existing synthetic
datasets, such as animated-film-style content,
embodied manipulation, scene
navigation, and articulated objects. SynthVerse substantially expands
dataset diversity by covering a broader range of object categories and providing high-quality dynamic motions and interactions, enabling more robust training and evaluation for general point tracking. In addition, we establish a highly diverse point tracking
benchmark to systematically evaluate state-of-the-art methods under broader domain shifts. Extensive experiments and analyses demonstrate that training with SynthVerse yields consistent improvements in generalization and reveal limitations of existing trackers under diverse settings.
Read the full abstract ▾
All authors Weiguang Zhao, Haoran Xu, Xingyu Miao, Qin Zhao, Rui Zhang, Kaizhu Huang, Ning Gao, Peizhou Cao, Mingze Sun, Mulin Yu, Tao Lu, Linning Xu, Junting Dong, Jiangmiao Pang
Computer Vision
TrajVG: 3D Trajectory-Coupled Visual Geometry Learning
Xingyu Miao, Weiguang Zhao, Tao Lu et al. (+5 more)
Feed-forward multi-frame 3D reconstruction models often degrade on
videos with object motion. Global-reference becomes ambiguous under multiple motions, while the local pointmap relies heavily on estimated relative poses and can drift, causing cross-frame misalignment and duplicated structures. We propose TrajVG, a reconstruction framework that makes cross-frame 3D correspondence an explicit prediction by estimating camera-coordinate 3D trajectories. We couple sparse trajectories, per-frame local point maps, and relative camera poses with
geometric consistency objectives: (i) bidirectional trajectory-pointmap consistency with controlled gradient flow, and (ii) a pose consistency objective driven by static track anchors that suppresses gradients from dynamic regions. To scale training to in-the-wild
videos where 3D trajectory labels are scarce, we reformulate the same coupling constraints into self-supervised objectives using only pseudo 2D tracks, enabling unified training with mixed supervision. Extensive experiments across 3D tracking, pose estimation, pointmap reconstruction, and
video depth show that TrajVG surpasses the current feedforward performance baseline.
Read the full abstract ▾
All authors Xingyu Miao, Weiguang Zhao, Tao Lu, Linning Xu, Mulin Yu, Yang Long, Jiangmiao Pang, Junting Dong
Computer Vision
Aavash Chhetri, Bibek Niroula, Pratik Shrestha et al. (+5 more)
Federated learning (FL) enables collaborative model training across decentralized medical institutions while preserving data
privacy. However, medical FL
benchmarks remain scarce, with existing efforts focusing mainly on unimodal or bimodal modalities and a limited range of medical tasks. This gap underscores the need for standardized evaluation to advance systematic understanding in medical
MultiModal FL (MMFL). To this end, we introduce Med-MMFL, the first comprehensive MMFL
benchmark for the medical domain, encompassing diverse modalities, tasks, and federation scenarios. Our
benchmark evaluates six representative state-of-the-art FL
algorithms, covering different aggregation strategies, loss formulations, and regularization techniques. It spans
datasets with 2 to 4 modalities, comprising a total of 10 unique medical modalities, including text, pathology images, ECG, X-ray, radiology reports, and multiple MRI sequences. Experiments are conducted across naturally
federated, synthetic IID, and synthetic non-IID settings to simulate real-world heterogeneity. We assess
segmentation, classification, modality
alignment (
retrieval), and VQA tasks. To support reproducibility and fair comparison of future
multimodal federated learning (MMFL) methods under realistic medical settings, we release the complete
benchmark implementation, including data processing and partitioning pipelines, at https://github.com/bhattarailab/Med-MMFL-
Benchmark .
Read the full abstract ▾
All authors Aavash Chhetri, Bibek Niroula, Pratik Shrestha, Yash Raj Shrestha, Lesley A Anderson, Prashnna K Gyawali, Loris Bazzani, Binod Bhattarai
Computer Vision · Cross-listed from cs.ET
Self-evolving Embodied AI
Tongtong Feng, Xin Wang and Wenwu Zhu
Embodied Artificial Intelligence (AI) is an intelligent system formed by
agents and their environment through active perception,
embodied cognition, and action interaction. Existing
embodied AI remains confined to human-crafted setting, in which
agents are trained on given
memory and construct models for given tasks, enabling fixed embodiments to interact with relatively static environments. Such methods fail in in-the-wild setting characterized by variable embodiments and dynamic open environments. This paper introduces self-evolving
embodied AI, a new paradigm in which
agents operate based on their changing state and environment with
memory self-updating, task self-switching, environment self-prediction, embodiment self-adaptation, and model self-
evolution, aiming to achieve continually adaptive intelligence with autonomous
evolution. Specifically, we present the definition, framework, components, and mechanisms of self-evolving
embodied AI, systematically review state-of-the-art works for realized components, discuss practical applications, and point out future research directions. We believe that self-evolving
embodied AI enables
agents to autonomously learn and interact with environments in a human-like manner and provide a new perspective toward general artificial intelligence.
Read the full abstract ▾
Computer Vision
LCUDiff: Latent Capacity Upgrade Diffusion for Faithful Human Body Restoration
Jue Gong, Zihan Zhou, Jingkai Wang et al. (+4 more)
Existing methods for restoring degraded human-centric images often struggle with insufficient fidelity, particularly in human body restoration (HBR). Recent
diffusion-based restoration methods commonly adapt pre-trained text-to-image
diffusion models, where the variational autoencoder (VAE) can significantly bottleneck restoration fidelity. We propose LCUDiff, a stable one-step framework that upgrades a pre-trained latent
diffusion model from the 4-channel latent space to the 16-channel latent space. For VAE fine-tuning, channel splitting
distillation (CSD) is used to keep the first four channels aligned with pre-trained priors while allocating the additional channels to effectively encode high-frequency details. We further design prior-preserving adaptation (PPA) to smoothly bridge the mismatch between 4-channel
diffusion backbones and the higher-dimensional 16-channel latent. In addition, we propose a decoder router (DeR) for per-sample decoder routing using restoration-quality score annotations, which improves visual quality across diverse conditions. Experiments on synthetic and real-world
datasets show competitive results with higher fidelity and fewer artifacts under mild degradations, while preserving one-step efficiency. The code and model will be at https://github.com/gobunu/LCUDiff.
Read the full abstract ▾
All authors Jue Gong, Zihan Zhou, Jingkai Wang, Shu Li, Libo Liu, Jianliang Lan, Yulun Zhang
Computer Vision
Interactive Spatial-Frequency Fusion Mamba for Multi-Modal Image Fusion
Yixin Zhu, Long Lv, Pingping Zhang et al. (+5 more)
Multi-Modal Image Fusion (MMIF) aims to combine images from different modalities to produce fused images, retaining texture details and preserving significant information. Recently, some MMIF methods incorporate frequency domain information to enhance spatial features. However, these methods typically rely on simple serial or parallel spatial-frequency fusion without interaction. In this paper, we propose a novel Interactive Spatial-Frequency Fusion Mamba (ISFM) framework for MMIF. Specifically, we begin with a Modality-Specific Extractor (MSE) to extract features from different modalities. It models long-range dependencies across the image with linear computational complexity. To effectively leverage frequency information, we then propose a Multi-scale Frequency Fusion (MFF). It adaptively integrates low-frequency and high-frequency components across multiple scales, enabling robust representations of frequency features. More importantly, we further propose an Interactive Spatial-Frequency Fusion (ISF). It incorporates frequency features to guide spatial features across modalities, enhancing complementary representations. Extensive experiments are conducted on six MMIF
datasets. The experimental results demonstrate that our ISFM can achieve better performances than other state-of-the-art methods. The source code is available at https://github.com/Namn23/ISFM.
Read the full abstract ▾
All authors Yixin Zhu, Long Lv, Pingping Zhang, Xuehu Liu, Tongdan Tang, Feng Tian, Weibing Sun, Huchuan Lu
Computer Vision · Cross-listed from cs.RO
Quantile Transfer for Reliable Operating Point Selection in Visual Place Recognition
Dhyey Manish Rajani, Michael Milford and Tobias Fischer
Visual Place Recognition (VPR) is a key component for localization in Global
Navigation Satellite System (GNSS)-denied environments, but its performance critically depends on selecting an image matching threshold (operating point) that balances precision and recall. Thresholds are typically hand-tuned offline for a specific environment and fixed during deployment, leading to degraded performance under environmental change. We propose a method that automatically estimates the operating point of a VPR system to maximize recall whilst aiming to achieve 100% precision. The method uses a small calibration traversal with known correspondences and transfers thresholds to deployment via
quantile normalization of similarity score distributions. This
quantile transfer ensures that thresholds remain stable across calibration sizes and query subsets. Experiments with seven state-of-the-art VPR techniques across five
benchmark datasets demonstrate that our proposed approach consistently outperforms existing baselines, enabling the underlying VPR technique to operate at 100% precision in approximately twice as many deployment scenarios (median improvement), while retrieving up to 29% more correct matches at that precision. The method eliminates manual tuning by adapting to new environments and generalizing across operating conditions. Our code is available at https://github.com/DhyeyR-007/
Quantile-Transfer-for-Reliable-VPR.
Read the full abstract ▾
Computer Vision
Enabling Real-Time Colonoscopic Polyp Segmentation on Commodity CPUs via Ultra-Lightweight Architecture
Weihao Gao, Zhuo Deng, Zheng Gong et al. (+1 more)
Real-time polyp
segmentation is essential for early colorectal cancer detection, yet clinical deployment remains blocked by GPU dependency. We introduce the UltraSeg family, a set of CPU-native
segmentation models operating below 0.3M parameters. UltraSeg-108K (0.108M) establishes the extreme-compression frontier, while UltraSeg-130K (0.130M) integrates cross-layer lightweight fusion for enhanced multi-center generalization. The architecture replaces parameter-heavy components with grouped multi-rate dilated convolutions and
attention-gated cross-layer fusion, achieving real-time throughput on a single CPU core (exceeding 50 FPS at 256*256 and 30 FPS at 352*352) without sacrificing clinical-grade accuracy. Evaluated on seven public
datasets, UltraSeg-130K attains Dice scores exceeding 0.8 at both resolutions, substantially outperforming all existing sub-0.3M competitors. Notably, it approaches or exceeds UNet-Medium (7.76M parameters) on zero-shot external validations while using only 1.7% of its parameters, establishing the first strong baseline for CPU-native real-time polyp
segmentation. When scaled to 4.38M parameters, UltraSeg achieves accuracy competitive with heavyweight state-of-the-art models while maintaining an order-of-magnitude parameter advantage, demonstrating that the proposed design principles yield intrinsic representational gains across the entire efficiency spectrum. By delivering the first clinically deployable, CPU-native real-time solution, this work provides an immediately usable tool for resource-limited settings and a reproducible blueprint for real-time medical AI beyond endoscopy. Source code is publicly available.
Read the full abstract ▾
All authors Weihao Gao, Zhuo Deng, Zheng Gong, Lan Ma