Computer Vision
PrismGPT: Proxy-Guided Learning for Region-Aware Photo Editing with Self-Synthesized Reasoning
Ke Zhao, Hue Nguyen, Abhijith Punnappurath et al. (+3 more)
Professional photo finishing relies on both global adjustments and region-specific local edits guided by semantic masks, yet current automated methods handle this workflow only partially. We present PrismGPT, a
Vision-
Language Model (VLM) framework that produces structured, region-aware editing plans from a single input image without relying on commercial black-box tools. Training a VLM to simultaneously diagnose aesthetic deficiencies at both global and local levels while predicting precise editing parameters is challenging due to the vast combinatorial decision space. We address this through proxy-guided learning: two simpler proxy tasks – operation decomposition and region-aware aesthetic ranking – teach the foundational skills the model needs, while a competence-based dynamic scheduler automatically rebalances the multi-task training ratio, progressively shifting emphasis from the proxy tasks to the primary editing task as each skill is mastered. Crucially, all
reasoning traces used for supervised fine-tuning are self-synthesized by the same base model, eliminating the need for a stronger external teacher. Experiments on MIT-Adobe FiveK and SPIRE, a new professionally retouched
benchmark we introduce, show that PrismGPT achieves state-of-the-art results while using only ~6% of the training data compared to the previous best method.
Read the full abstract ▾
All authors Ke Zhao, Hue Nguyen, Abhijith Punnappurath, Zhongling Wang, Iqbal Mohomed, Michael S. Brown
Computer Vision
Ananke: Contractive Torus Attractor Networks
Zhongping Ji
We introduce Ananke, a representation-learning framework that scaffolds latent representations onto a structured product-
torus prior, and its flagship visual backbone realization, Contractive
Torus Attractor Networks (CTAN). By factorizing high-dimensional latent spaces into an orthogonal direct sum of two-dimensional phase planes ($\bigoplus_{k=1}^K \R^2$), Ananke coordinates feature updates via a decoupled dual-phase continuous flow: skew-symmetric
Hamiltonian transport moves features tangentially along energy level sets to preserve semantic phase invariants, while signed gradient dissipation contracts transverse perturbations normally toward target invariant
manifolds. For circular potential families with frozen parameters, logarithmic radial feedback yields the Exact Log-
Symplectic Flow (ELSF), an analytical closed-form mapping with exact exponential decay of log-radius error that evaluates in a single forward pass without numerical integration. We establish local input-to-state bounds for level-set deviations and log-radius errors, and characterize the normal hyperbolicity and persistence of the ideal product
torus under bounded perturbations. We further formulate the architecture through Lie–Trotter operator splitting, unifying spatial depthwise
diffusion with local
manifold contraction, and analyze both exact trigonometric flows and hardware-friendly
symplectic dual-shear variants. Across natural image
benchmarks (CIFAR-100) and clinically challenging endoscopy
datasets (Kvasir-v2), CTAN demonstrates exceptional parameter efficiency: an ultra-compact hierarchical model with merely 0.27M parameters achieves 90.52% accuracy on Kvasir-v2, outperforming 25M+ baselines (ResNet-50, DenseNet-161) by nearly two orders of magnitude in capacity, while scaled variants attain 80.32% top-1 accuracy on CIFAR-100.
Read the full abstract ▾
Computer Vision
MiTHras: Task-specific Hierarchical Semi-supervised Contrastive Masked Autoencoder for Mitotic Figure Analysis
Trinh T. L. Vuong, Simon Graham, Quoc Dang Vu et al. (+5 more)
Mitotic figure (MF) analysis supports tumor grading and prognostic assessment, but automated models remain sensitive to differences in tissue type and image acquisition. We present MiTHras, a task-specific pretraining framework that combines pseudo-label-guided image- and
token-level contrastive learning with masked reconstruction. We construct TCGA-MF-Pseudo, a corpus of 1.8 million
cell-centered images from 14 TCGA cohorts spanning 11 organ sites. Comprehensive evaluation on MF classification, detection, count-based survival prediction, and subtype classification demonstrates the efficacy of MiTHras. It achieves the highest mean F1 on all three MF classification
benchmarks and both subtype
benchmarks. MiTHras also outperforms general-purpose and pathology foundation encoders by a larger margin under frozen-encoder linear probing than under full fine-tuning. Although detection gains are modest due to a shared candidate-detection stage, ablations confirm that
token-level supervision improves typical-versus-atypical classification and linear probing. These findings establish that MiTHras yields robust, transferable representations for automated mitotic activity assessment.
Read the full abstract ▾
All authors Trinh T. L. Vuong, Simon Graham, Quoc Dang Vu, Phat T. H. Ho, Jeewoo Lim, Mostafa Jahanifar, Nasir Rajpoot, Jin T. Kwak
Computer Vision
GraphSVR: q-Space–Aware Graph-Based Slice-to-Volume Registration for Diffusion MRI
Noga Kertes, Daphna Link Sourani, Alex M. Bronstein et al. (+1 more)
Diffusion-weighted imaging (DWI) remains highly vulnerable to subject motion, particularly in time-efficient protocols and in motion-prone populations. While slice-to-volume registration (SVR) can mitigate inter-slice and inter-stack misalignment,
diffusion MRI introduces additional complexity due to
diffusion-direction-dependent contrast and the requirement to align dozens of measurements within a common reference frame, effectively yielding a 4D registration problem. Existing approaches rely primarily on sequential modeling or pairwise similarity and often degrade under sparse gradient sampling or severe motion. We introduce GraphSVR, a q-space-aware graph-based framework for 4D SVR registration in DWI. GraphSVR represents slice groups as nodes in an acquisition-structured graph, with edges encoding temporal proximity, spatial slice geometry and
diffusion encoding relationships. A graph neural network predicts globally consistent stack-wise rigid motion, optimized in a self-supervised, zero-shot manner using only an anatomical reference image, without requiring paired ground-truth motion. We evaluate GraphSVR using both fully synthetic
diffusion simulations and realistic recombination-based simulations from real acquisitions with controllable motion severity and gradient sparsity. Performance is quantified using grid error (mm) and rotation error relative to known ground-truth transforms. Under severe motion, GraphSVR reduces grid error and rotation error by 73% compared to FSL eddy, the standard DWI motion-correction method, with the largest gains observed in sparse-direction regimes. These results demonstrate that explicitly modeling acquisition structure through graph-based
reasoning improves robustness and global consistency in 4D DWI motion estimation. Code is available at https://github.com/nogakertes/GraphSVR.git.
Read the full abstract ▾
All authors Noga Kertes, Daphna Link Sourani, Alex M. Bronstein, Moti Freiman
Computer Vision
ReSTI: A Source-Grounded Audit and Repair of STI-Bench
Pengzhan Sun, Ramanathan Rajaraman, Shiu-Hong Kao et al. (+2 more)
Spatial–temporal
benchmarks are valid only when their questions, source annotations, and answer options identify the same physical quantity. We audit STI-Bench against the official ScanNet, Waymo, and Omni6DPose sources and find systematic coordinate-system and timestamp errors, under-specified targets and times, and disagreements between keyed options and answer details. We introduce ReSTI, a source-backed revision that reconstructs every recoverable answer under an explicit target, time, coordinate system, physical quantity, and unit. Source reconstruction reveals task-level
geometric failures: ScanNet Grounding omits the required
alignment between annotation and raw camera coordinate systems, while Orientation measures camera rotation on the wrong plane. ReSTI replaces these labels with explicit, source-consistent
geometric definitions and corrects other source-verifiable defects, including Waymo poses evaluated at the wrong timestamp. Across 2,064 legacy questions, ReSTI retains 1,782 questions and records 282 evidence-backed exclusions. ReSTI therefore provides a conservative and source-traceable basis for evaluating precise
video spatial–temporal
reasoning. Project page: https://github.com/pengzhansun/ReSTI.
Read the full abstract ▾
All authors Pengzhan Sun, Ramanathan Rajaraman, Shiu-Hong Kao, Junbin Xiao, Angela Yao
Computer Vision
What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization
Niklas Bubeck, Yundi Zhang, Vasiliki Sideri-Lampretsa et al. (+4 more)
Latent
diffusion models now dominate medical image generation, and every such pipeline rests on a tokenizer that compresses images into the latent codes for image generation to operate on. Thereby, the tokenizer choice bounds every downstream task from reconstruction fidelity and generation quality to the representations available for downstream analysis. Yet, medical imaging pipelines routinely utilize tokenizers from natural imaging on the hypothesis that their behavior carries over. However, this is an assumption never tested in the medical imaging regime, where
datasets are orders of magnitude smaller and images exhibit far lower inter-sample variance. We present a systematic evaluation of medical image tokenizers evaluating thirty configurations across ten model families on twelve
datasets at three compression factors, spanning reconstruction, generation, latent geometry, downstream classification, and memorization. We find that (1) performance on image reconstruction and generation strongly correlate, unlike prior reports on natural images; (2) modern tokenizers use nearly all of their codebook entries, but still leave most of the latent space unused; (3) training-set memorization is mild and is further suppressed by stronger latent space compression; and (4) discrete quantization can largely preserve downstream classification, with lookup-free schemes being the main exception.
Read the full abstract ▾
All authors Niklas Bubeck, Yundi Zhang, Vasiliki Sideri-Lampretsa, Julian McGinnis, Jiancheng Yang, Daniel Rueckert, Jiazhen Pan
Computer Vision · Cross-listed from cs.RO
Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies
Trung Dao, Sankalp Yamsani, Jaden Park et al. (+2 more)
Vision-
Language-Action (VLA) models map observations to actions with no objective that accounts for how the world responds, so their robustness is bounded primarily by data coverage. World models carry precisely that missing objective and are better grounded for it, yet rolling the future forward costs seconds per decision and rules them out of the control loop. We show the two can be separated. What a world model knows about physical scenes lives in its internal features; generating the future is merely the objective that produced them, so the grounding can be inherited while the generative machinery is left behind. We add one feature-
alignment term to ordinary VLA training: a frozen world model is run over the training frames once and cached, and the student learns to agree with that cache. No teacher is loaded during training, the projector is discarded after it, and the deployed policy is identical to the undistilled baseline, running in 32ms and 1.86GB on a consumer RTX5090, so every gain is attributable to the representation rather than to added capacity or test-time compute. A 0.8B student reaches 97.9% on LIBERO, improves from 48.2% to 50.5% on RoboCasa-GR1
humanoid manipulation, and the same objective carries over to real hardware, on both a single-arm and a bimanual platform. The gain survives changes of student scale, backbone,
alignment layer, and teacher, indicating a broad representational prior rather than a fragile
alignment between two particular networks. Project page: https://thaw-vla.trung-dt.com/.
Read the full abstract ▾
All authors Trung Dao, Sankalp Yamsani, Jaden Park, Joohyung Kim, Yong Jae Lee
Computer Vision
Ev-YOLO: Uncertainty-Aware Object Detection via a Unified Evidential Formulation
Simon Barbarit-Gaboriau, Hind Laghmara, Rémi Boutteau et al. (+1 more)
Reliable uncertainty estimation is essential for deploying object detectors in autonomous systems operating in uncertain environments. Evidential Deep Learning (EDL) provides a principled framework for uncertainty-aware classification by representing network outputs as evidence and interpreting predictions through subjective logic. However, existing evidential object detectors typically combine evidential classification with regression uncertainty models that do not share the same theoretical foundation. In this work, we propose an evidential version of YOLOv8 in which both classification and bounding-box regression are formulated within a common evidential framework. Our approach exploits YOLOv8's distribution-based bounding-box representation, allowing the evidential formulation to be applied not only to classification but also to localisation. As a result, both tasks produce belief, uncertainty, and probability estimates that can be interpreted within the Dempster–Shafer framework. Experiments on KITTI, MUSES, and nuScenes show that the resulting detector remains broadly competitive with standard YOLOv8 in terms of detection accuracy while providing a localisation uncertainty that effectively discriminates between correct and erroneous detections. Moreover, this uncertainty becomes increasingly discriminative under domain shift.
Read the full abstract ▾
All authors Simon Barbarit-Gaboriau, Hind Laghmara, Rémi Boutteau, Samia Ainouz
Computer Vision
High-resolution Nitrogen Dioxide Maps Reveal Exposure Limit Breaches across Europe
Linus Scheibenreif and Konrad Schindler
Nitrogen dioxide (NO2) is a common air pollutant, released into the atmosphere through the incomplete burning of fossil fuels, and associated with respiratory and cardiovascular diseases in humans. Ambient NO2 concentrations are regulated through air-quality limits assessed with a sparse network of fixed monitors. The revised EU Ambient Air Quality Directive (2024/2881) introduces a daily NO2 limit to be met from 2030. At present, neither the regulatory monitoring network nor existing coarse, annual-mean models can resolve NO2 concentrations at the spatio-temporal resolutions necessary to assess compliance. Here we map NO2 across Europe at hourly and 10m resolution with a machine-learning model that combines ground monitors with satellite, reanalysis, land-use, traffic and emission data and returns a calibrated predictive distribution at every location. Validated against held-out regulatory monitors and independent citizen-science campaigns, the maps resolve high-resolution spatiotemporal NO2 gradients for 110 metropolitan areas in Europe. We reconstruct the daily compliance statistic across those regions and find limit breaches in 91 EU air quality zones deemed compliant by the regulatory monitoring network, covering a population of approximately 135M. Beyond air quality zones and monitor locations, an estimated 9-9.4% (20M) of the population in mapped regions lives in areas where the daily NO2 limit is breached. The high-resolution maps offer a route to population-scale assessment of compliance with the 2030 limits.
Read the full abstract ▾
Computer Vision
FedMust: Semi-supervised Multi-task Student-Teacher Federated Learning for Multi-organ CT Segmentation
Ashkan Moradi, Bendik Skarre Abrahamsen and Mattijs Elschot
Multi-organ
segmentation using deep learning requires large amounts of annotated patient data; however, institutions often lack sufficiently large and diverse annotated
datasets.
Privacy constraints further prevent institutions from sharing patient data to overcome this limitation. Moreover, due to the labor-intensive nature of annotation and the scarcity of diverse expertise, institutions typically have labels for only a small portion of their local data, leaving the larger unlabeled portion unused. In this work, we propose a flexible semi-supervised
federated multi-task student-teacher framework that leverages
federated learning (FL) to improve multi-organ
segmentation using both labeled and unlabeled data across participating sites. At each communication round, the proposed framework initiates local training, where clients with labels for the same task form a federation to produce an aggregated teacher model. The resulting teachers generate task-specific features for all data at each client. Subsequently, all clients form a second federation to train a multi-task student model with a shared encoder and task-specific decoders that replicate the teacher-generated features across all
segmentation tasks. The aggregated student model is then used to update the local teachers and initiate the next training round. Extensive experiments demonstrated the effectiveness of the proposed method compared with local and
federated single-organ models, yielding an average performance gain of 13 percent across clients. The experiments also demonstrated the impact of multi-task learning and unlabeled data and the applicability of the framework in relaxing labeled-data requirements for client participation. The code is available at https://github.com/AshknMrd/FedMust.
Read the full abstract ▾
Computer Vision
Relationally Grounded Latent World Models for Autonomous Driving
Fabian Schmidt, Markus Enzweiler and Abhinav Valada
Latent world models learn predictive representations for autonomous driving, but the relational semantics these states preserve often remain implicit. We investigate whether traffic scene graphs can serve as privileged semantic supervision for latent world representations. Building on LAW, we construct actor-centric scene graphs from nuScenes 3D annotations, encode their serialized relational structure using a frozen text
embedding model, and align the visual latent representations with this semantic target during training. We remove the supervision branch at inference, so it requires neither scene graphs nor 3D annotations and adds no test-time computation. On nuScenes, our method reduces average trajectory L2 error from 0.661 to 0.622 (5.9%) and collision rate from 0.456 to 0.217 (52.4%) relative to our retrained LAW baseline. It also outperforms an unstructured caption-style semantic target, supporting the benefit of explicit relational structure for latent world-model representation learning.
Read the full abstract ▾
Computer Vision
Video-based Surgical Skill Assessment Using Dynamics-and-Uncertainty-Aware Tree-based Gaussian Process Classifier
Arefeh Rezaei, Mohammad Javad Ahmadi, Amir Molaei et al. (+1 more)
The proposed pipeline integrates a representation-flow convolutional neural network with a dynamics- and uncertainty-aware tree-based Gaussian Process classifier. In this framework, latent motion dynamics are exploited both as discriminative representations and as a source of input uncertainty, enhancing robustness against temporal variations and abnormal motion transitions. Compared with conventional deep learning approaches, the proposed strategy requires less training data and offers improved computational efficiency. To further improve classification performance, we introduce novel semantic-aware compound kernels that effectively capture semantic, flow, and dynamic information embedded in
surgical video features. In addition, uncertainty-aware kernels are developed to strengthen the robustness and practical applicability of the compound kernel framework. The proposed method is evaluated on two
benchmark datasets, namely the JIGSAWS and the Cataract-LMM (Capsulorhexis)
datasets. Experimental results demonstrate strong performance across both
datasets, including the LOSO and LOUO evaluation protocols on JIGSAWS, including the subject-independent LOUO protocol on JIGSAWS, on which the framework attains a mean accuracy of \ph{96.9}%; results under the within-subject LOSO protocol are reported for comparability with prior work, achieving competitive accuracy while substantially reducing computational cost. Overall, the proposed pipeline provides an efficient and accurate framework for
video-based
surgical skill assessment.
Read the full abstract ▾
All authors Arefeh Rezaei, Mohammad Javad Ahmadi, Amir Molaei, Hamid D. Taghirad
Computer Vision
Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Merging
Ruxi Gu, Zilei Wang and Wei Wang
Model merging aims to consolidate multiple task-specific models without access to extra training process. However, existing subspace-based methods largely rely on a uniform treatment of task updates, overlooking their intrinsic spectral and depth-wise heterogeneity. We identify two key deviations from this assumption: different tasks require different subspace capacity and exhibit different tolerance to spectral transformation, while subspace projection introduces depth-dependent distortion. Based on these observations, we propose SADA-Merging, a spectrum-aware and depth-adaptive framework for data-free model merging. SADA-Merging allocates task-specific subspace capacity according to spectral complexity, adapts spectral preservation according to task-wise plasticity, and applies depth-dependent anchoring to compensate for projection-induced distortion. This enables the fusion process to adapt to both the intrinsic geometry of each task and its sensitivity across network depth. SADA-Merging operates directly on task updates and is applicable to both full fine-tuning and LoRA settings. Extensive experiments demonstrate consistent improvements over existing data-free merging methods across different task scales and adaptation settings.
Read the full abstract ▾
Computer Vision
Applications of Neural Cellular Automata: State of the Art, Challenges and Opportunities
Nick Lemke, Niklas Ihm, John Kalkhof et al. (+11 more)
Neural Cellular Automata (NCAs) are a new type of neural network architecture which enable accurate and robust inference at extremely small model sizes. Recently, NCAs have advanced to become interesting low-resource alternatives to convolution- and
attention-based architectures for various tasks such as image analysis, synthetic image generation, and simulation. The rapid development and increased research interest necessitate a comprehensive review of the emerging technology. This review provides an overview of the fundamentals of NCAs, applications to medical imaging, as well as insights into the state of the art. We analyze recent modifications to the originally proposed NCA architecture with respect to their efficiency and accuracy. Furthermore, we review practical applications in real-world scenarios with a focus on medical image analysis,
segmentation, classification, registration, depth estimation, and image synthesis. Finally, we identify several advantages of NCAs, research gaps, and conclude with an analysis of future opportunities for NCAs in medical applications in confined settings or areas that have particular demands for robustness or efficient data processing.
Read the full abstract ▾
All authors Nick Lemke, Niklas Ihm, John Kalkhof, Mirko Konstantin, Henry J. Krumb, Daniel M. Lang, Ehsan Pajouheshagar, Ario Sadafi, Arjan Kuijper, Karim Lekadir, Marco Lorenzi, Carsten Marr, Julia A. Schnabel, Anirban Mukhopadhyay
Computer Vision · Cross-listed from cs.RO
What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior
Débora Oliveira Makowski, Samiran Gode, Abhijeet Nayak et al. (+4 more)
Modern
Vision-
Language Navigation (VLN) models rely mostly on pre-trained large
Vision-
Language Models (VLMs) to predict
navigation actions. While this fusion of
language instructions and visual observations allows
multimodal reasoning, it obscures how information is routed across modalities or what mechanisms drive
navigation decisions. Thus, it remains unclear whether VLN models ground their predictions in relevant semantic cues or can track task progress. In this work, we study the interpretability and steerability of VLN models. We use intervention-based metrics that measure how visual observations, instructions, and visual
memory causally influence
navigation decisions. Our results show that these
navigation policies are sensitive to all input modalities and do not depend on a single one. We further show that these
agents encode
navigation progress and retain semantic structure from their VLM backbones, enabling concept-level steering through internal activations. Finally, we extract activation vectors for abstract behaviors to transfer them zero-shot to out-of-distribution real-world scenarios, improving performance without additional fine-tuning.
Read the full abstract ▾
All authors Débora Oliveira Makowski, Samiran Gode, Abhijeet Nayak, Marco Hutter, Cordelia Schmid, Lukas Rosenberger Schmid, Wolfram Burgard
Computer Vision
What Survives on Real Drawings: Active Sampling, Connectome Wiring, and Matched Baselines in Architectural Document Vision
Dmitry Kuklev
A connectome-constrained model of the fly visual system, optimized for motion and then frozen, can be driven over architectural drawings by prescribed motion and used as a texture representation. We compare it with information-matched baselines that see the same 721 photoreceptor samples. On clean synthetic data the frozen model transfers but loses to task training: 0.857 area-weighted accuracy in one-shot hatch matching versus 0.959 for a 5,888-parameter CNN, and 0.619 IoU in wall
segmentation versus 0.905 for a matched network. Under scan noise and thickened strokes, the trained networks lose up to 0.188 accuracy while the frozen pipeline loses 0.030. On fourteen production sheets, opened once, a 1,876-parameter fly model reaches 0.505 average precision versus 0.415 for a network two hundred times larger. A preregistered held-out split confirms the clean-data ordering: 0.835 for the
circuit, 0.894 for receptors only, and 0.971-0.980 for trained CNNs. Rewiring the connectome while preserving degrees or type pairs and transmitter signs costs 0.271-0.356 accuracy across three seeds, so the exact wiring is load-bearing. Yet the intact
circuit does not beat its moving retina, and T4/T5 silencing leaves both tasks intact. Longer observations reverse the
circuit-receptor ordering once the stimulus spans a period, but not through T4/T5. Thus active sampling and exact structure matter, while clean-data practical performance remains dominated by task-trained networks and the useful transfer margin is largely retinal.
Read the full abstract ▾
Computer Vision
HyperCLIP++: Fine-tuning CLIP forOpen-vocabulary Semantic Segmentation in Hyperbolic Space
Zelin Peng, Zhengqin Xu, Changsong Wen et al. (+4 more)
CLIP, a foundational
vision-
language model, has emerged as a powerful tool for open-vocabulary semantic
segmentation. While freezing CLIP's text encoder is known to preserve its generalization capability, recent studies show that fine-tuning both CLIP's text and image encoders jointly significantly enhances
segmentation performance, especially for classes from open sets. In this work, we explain this phenomenon from the perspective of hierarchy
alignment, since during fine-tuning, the hierarchical level of image
embeddings shifts from image-level to pixel-level. We achieve this by leveraging
hyperbolic space, which naturally encodes hierarchical structures. Our key observation is that, during fine-tuning, the
hyperbolic radius of CLIP's text
embeddings decreases, facilitating better
alignment with the pixel-level granularity of visual data. Building on this, we propose HyperCLIP++, a novel and parameter-efficient adaptation strategy. HyperCLIP++ directly adjusts the
hyperbolic radius of CLIP's
embeddings via scaling transformations to achieve a hierarchy
alignment to the target task, i.e.,
segmentation. To ensure this hierarchy
alignment is effected consistently across both modalities and preserves their cross-modal
alignment during training, HyperCLIP++ integrates a Dual Cross-Relation Communication (DCRC) module that synchronizes these adjustments between the
vision and text pathways. Our experiments show that HyperCLIP++ achieves state-of-the-art performance across three
benchmarks while fine-tuning only approximately 5% of CLIP's total parameters. More importantly, we observe that after adjustment, CLIP's text
embeddings exhibit a relatively fixed
hyperbolic radius across
datasets, suggesting that the hierarchical level required for this
segmentation task might be quantified using the
hyperbolic radius.
Read the full abstract ▾
All authors Zelin Peng, Zhengqin Xu, Changsong Wen, Yu Huang, Yaoming Wang, Xiaokang Yang, Wei Shen
Computer Vision
Evaluating Transformation Models for pCLE Mosaic Registration
Ahmed Aboelela, Johannes Barcsay, Jana Friedhof et al. (+3 more)
Confocal
Laser Endomicroscopy (CLE) provides real-time, cellular-resolution optical biopsy but has a narrow field of view, which image mosaicing can extend to provide anatomical context. Because of line-by-line acquisition, probe motion, and probe-tissue interaction, frame
alignment generally requires a non-linear transformation whose accuracy is difficult to quantify: flexible transformation models can fit intensity features and noise, so appearance-based metrics such as Normalized Cross-Correlation (NCC) can improve without a genuine gain in
geometric accuracy. We therefore establish a
dataset of 132 frame pairs across fourteen pCLE sequences from 4 patients with manually annotated landmark correspondences, so that Target Registration Error (TRE) can serve as a geometrically grounded complement to NCC. We assess the effect of progressively increasing the transformation model's degrees of freedom, from translation to Thin Plate Spline (TPS), and of six feature-matching backends spanning classical (Shi-Tomasi, Lucas-Kanade) and learned (SuperPoint, SuperGlue, LightGlue, LoFTR, RoMa) approaches. Translation and rigid models prove insufficient under tissue deformation, while TPS with random sampling achieves the strongest landmark-derived
alignment of the evaluated configurations; among the learned matchers, used without fine-tuning, only RoMa offers a robust, if modest, advantage over other methods. At the sequence level, pairwise registration quality proved an unreliable predictor of final mosaic quality, so mosaic quality must be evaluated directly rather than inferred from pairwise metrics.
Read the full abstract ▾
All authors Ahmed Aboelela, Johannes Barcsay, Jana Friedhof, Miguel Gonçalves, Alexander Hann, Katharina Breininger
Computer Vision
Incentive Noise and Structural Prior Infusion for Multi-modal Object Re-Identification
Weixiang Zhou, Yuhao Wang, Xingguo Xu et al. (+4 more)
Multi-modal object Re-Identification (ReID) benefits from complementary information across heterogeneous imaging modalities. To further enrich semantic representation, text descriptions have recently been incorporated as an additional modality. However, recent
vision-
language approaches often treat text descriptions as clean, deterministic signals and overlook their inherent noise, including modality-mismatched phrases and semantically ambiguous expressions. Moreover, prevailing methods lack explicit mechanisms to reconcile fine-grained structural discrepancies between modalities, even after high-level semantic
alignment. To address these challenges, we propose a novel framework centered on Positive-Incentive Noise ({\pi}-noise) and structured
prompt modulation. First, the Semantic Cross-Modal Modulator harnesses task-aware {\pi}-noise, sampled from a distribution conditioned on both visual and text inputs, to perturb global
tokens and enable semantics-guided cross-modal compensation. Second, the Structure-Aware
Prompt Adapter injects learnable
geometric priors via
prompts to enhance spatial consistency. Third, the Context-Aware Sparse Fusion module distills structural context to guide adaptive fusion while shielding identity features from noisy local details. Experiments on three multi-modal ReID
benchmarks demonstrate the effectiveness and robustness of our approach. The code is available at https://github.com/zw-absin/INSPI.
Read the full abstract ▾
All authors Weixiang Zhou, Yuhao Wang, Xingguo Xu, Weizhen Zhou, Zhixun Su, Jinshan Pan, Cong Wang
Computer Vision
MIRAGE: Full-Body Bystander Privacy for Smart Glasses with Consent-Based Restoration
Muhammad Umair, Muhammad Danial Maqbool, Fatima Arshad Cheema et al. (+4 more)
Video recording on smart glasses exposes more than faces. Continuous capture reveals full-body biometric signatures, including gait, posture, and silhouette, that enable person re-identification (ReID) even after conventional face sanitization. We present MIRAGE, a three-tier architecture for
privacy-preserving smart glasses that enforces full-body
privacy, supports synthetic full-body replacement, and retains encrypted recovery material for consent-based restoration. We implement MIRAGE on a Raspberry Pi~5 (a CPU-only proxy for smart-glasses compute), companion phones, and a cloud generative backend. Compared to prior systems, MIRAGE achieves 0.948 AP and 0.976 AR while accurately detecting the complete visible body. Its bounding box masking reduces learned silhouette-based ReID to essentially random guessing, with 10.86% Rank-1 accuracy compared with an 11.12% measured chance level. Even against an adaptive adversary retrained on MIRAGE's sanitized pose signals, Rank-1 gait identification drops from 90.25% to 26.20%, removing 72.5% of the adversary's identification advantage.
Read the full abstract ▾
All authors Muhammad Umair, Muhammad Danial Maqbool, Fatima Arshad Cheema, Kapal Dev, Muhammad Hamad Alizai, Muhammad Ali Siddiqi, Naveed Anwar Bhatti
Computer Vision
Dynamic Thermal Gaussians: Multimodal 4D Gaussian Splatting
Rongfeng Lu, Lifeng Lin, Xiaobao Wei et al. (+7 more)
Thermography plays a vital role in military and broader thermal analysis applications. Recent progress in 3D thermal reconstruction has extended temperature analysis from 2D to 3D space, yet most existing works assume static temperature distributions, neglecting the temporal dynamics of heat transfer in real-world environments. To address this limitation, we propose the first dynamic RGB-Thermal reconstruction framework for complex scenes. Our method jointly models RGB appearance, thermal observations, and scene geometry as they change over time. Specifically, we introduce a
multimodal dynamic scene representation that anchors both the color and thermal modalities to a shared
geometric substrate, ensuring their consistency under spatiotemporal deformations. We further design
multimodal embeddings to enhance the motion expressiveness for each modality, and propose a
multimodal routing mechanism that retains a unified set of shared
multimodal Gaussians as the
geometric backbone while adaptively spawning modality-specific Gaussians to strengthen the representational capacity in detail-rich regions of each individual modality. In addition, we contribute a novel
benchmark dataset featuring high-frequency temperature variations to facilitate the evaluation of 4D reconstruction. Extensive experiments demonstrate that our method achieves high-fidelity spatiotemporal reconstruction of both appearance and temperature. Our code and
dataset are available at: https://github.com/LinLif1869/DTG.
Read the full abstract ▾
All authors Rongfeng Lu, Lifeng Lin, Xiaobao Wei, Quan Chen, Ming Lu, Yitian Xue, Yaoqi Sun, Yuhan Gao, Anke Xue, Chenggang Yan
Computer Vision
ME-VLM: A Unified VLM for Embodied Cognition and Agent Coordination
Foundation Model and Li Auto Inc
Physical AI requires models to ground visual and linguistic understanding in real-world environments while accounting for environmental constraints and execution feedback. We introduce MachEmbodied-VLM (ME-VLM), a unified
vision-
language model with two variants, 4B and 35B-A3B, that brings together
embodied cognition and
multimodal agent capabilities. Our work emphasizes physical perception and spatiotemporal
reasoning, together with planning, interaction, and outcome assessment in both digital and physical environments. We construct training data spanning
embodied and
multimodal agent tasks, including execution observations and feedback to support outcome assessment and decision refinement. The training pipeline comprises
embodied capability injection, separate reinforcement learning of
embodied and
multimodal-
agent experts, and multi-teacher on-policy
distillation that consolidates their complementary capabilities into a single model. Experiments show competitive performance on both
embodied and
agent benchmarks, as well as on autonomous-driving and
embodied-
navigation tasks. For edge deployment, visual
token compression, W4A8 quantization, and hardware-software co-optimization enable on-device inference of the 4B variant on the M100, reducing prefill latency from 400 ms to 188 ms. Project Page: https://machembodied.com/ME-
Brain/ME-VLM.html Code Repository: https://github.com/MachEmbodied/ME-VLM
Read the full abstract ▾
Computer Vision
Preoperative Prediction of Microvascular Invasion in Hepatocellular Carcinoma by Integrating Multimodal Ultrasound and Clinical Data: A Multicenter Study
Jun Cheng, Yuanyuan Kong, Qing Huang et al. (+9 more)
Background: Microvascular invasion (MVI) predicts recurrence and survival in hepatocellular carcinoma (HCC) but requires postoperative histopathology for diagnosis. We developed and validated a model integrating
multimodal ultrasound and clinical data for preoperative MVI prediction. Methods: This multicenter study included 489 patients with HCC from eight centers. All patients had B-mode ultrasound (BUS), color Doppler flow imaging (CDFI), dynamic contrast-enhanced ultrasound (DCE-US), and clinical information. Data from seven centers (n = 421) were used for model development with five-fold cross-validation; data from the remaining center (n = 68) formed an independent external validation cohort. The proposed
multimodal information fusion network used modality-specific encoders, a hemodynamic temporal change module for bidirectional DCE-US perfusion changes, and a representation consistency learning module to align heterogeneous ultrasound representations before
Transformer-based fusion. Results: In external validation, DCE-US achieved the highest single-modality area under the receiver operating characteristic curve (AUC; 0.8545+/-0.0198), versus clinical information (0.6715+/-0.0156), CDFI (0.6435+/-0.0344), and BUS (0.6087+/-0.0417). Pixel-difference sampling and the proposed temporal module outperformed alternative sampling and
video representation methods. The full model achieved the best performance, with an AUC of 0.8953+/-0.0180, accuracy of 81.18%+/-2.83%, sensitivity of 86.40%+/-6.69%, and specificity of 78.14%+/-6.28. Conclusions: Integrating
multimodal ultrasound and clinical information enabled promising preoperative MVI prediction in HCC. DCE-US was the main source of predictive information, while BUS, CDFI, and clinical information provided complementary value. The proposed framework may support preoperative risk stratification and individualized clinical decision-making.
Read the full abstract ▾
All authors Jun Cheng, Yuanyuan Kong, Qing Huang, Xiaotong Tan, Licong Dong, Yulong Han, Wufeng Xue, Ruobing Huang, Dong Ni, Qi Yang, Jie Yu, Ping Liang
Computer Vision · Cross-listed from cs.AI
Not All Task Vectors Need Equal Rank: Energy-Proportional Allocation for Model Merging
Hyunjoong Cho and Jinhyeok Jang
Model merging aims to combine multiple fine-tuned models derived from a common pretrained model into a single multi-task model without additional joint training. Recent spectral merging methods improve over simple weight averaging by exploiting low-rank structures of task-specific updates, but they commonly assign the same rank capacity to every task. This uniform allocation ignores that task vectors can have heterogeneous spectral complexity, causing the shared merging space to be used suboptimally. In this paper, we propose Spectral Energy-proportional Rank Allocation (SERA), a simple task-adaptive strategy that allocates ranks according to the singular-value energy structure of each task vector. By assigning richer spectral capacity to complex or isolated tasks and fewer directions to compact tasks, SERA extends SVD-based model merging from uniform-capacity merging to task-dependent capacity allocation. Experiments under standard
vision model merging protocols show that SERA improves multi-task merging performance while preserving the same total rank budget as existing spectral merging methods. Further analysis demonstrates that task-level spectral concentration is closely related to the per-task effect of adaptive rank allocation, providing insight into when and why SERA is effective.
Read the full abstract ▾
Computer Vision
0.5%>100%: Bidirectional Reciprocal Learning for Referring Image Segmentation
Xiaoqiang Lu, Licheng Jiao, Lingling Li et al. (+5 more)
Recent advances in
vision foundation models (VFMs) have shown remarkable capabilities across diverse unimodal visual tasks. However, adapting VFMs to referring image
segmentation (RIS) typically necessitates precise
vision-
language alignment via full fine-tuning, incurring substantial computational overhead and risking catastrophic forgetting. While existing parameter-efficient fine-tuning (PEFT) methods enable safe knowledge transfer with minimal training costs, they predominantly operate independently within individual modalities or focus exclusively on unidirectional guidance from
language to
vision, overlooking progressive cross-modal interaction and visual feedback for textual refinement. To address these limitations, we propose $\textbf{B}$idirectional $\textbf{R}$eciprocal $\textbf{L}$earning ($\textbf{BRL}$), a novel adapter-based PEFT framework that facilitates hierarchical, bidirectional information flow within both
token-mixing and channel-mixing layers of frozen foundation models. Specifically, BRL introduces two complementary lightweight modules. The Reciprocal
Attention Adapter (RAA) performs cross-modal query-key exchanges at the
token level, enabling visual and linguistic
tokens to mutually attend to each other for fine-grained spatial grounding. The Reciprocal Gate Adapter (RGA) generates cross-modal gating signals at the channel level, allowing global semantic context from one modality to adaptively recalibrate channel activations of the other. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg
benchmarks demonstrate the superiority of BRL over prior RIS methods, achieving state-of-the-art performance while requiring less than 0.5% backbone parameter updates. Code and models will be released at https://github.com/xiaoqiang-lu/BRL.
Read the full abstract ▾
All authors Xiaoqiang Lu, Licheng Jiao, Lingling Li, Yuting Yang, Long Sun, Wenping Ma, Xu Liu, Fang Liu
Computer Vision
CMAMBADEPTH: Self-supervised Monocular Depth Estimation with Channel Mamba and Hybrid Attention
Xuezhi Xiang, Jiayao Liu, Heqi Xiang et al. (+3 more)
Accurate
monocular depth estimation serves as a core enabler for single camera scene understanding. However, existing self-supervised
monocular depth estimation methods generally suffer from the bottleneck of inefficient cross-scale information interaction and difficulty in balancing local and global spatial modeling. In this paper, we propose CMambaDepth, a self-supervised framework that achieves efficient multi-scale feature fusion and fine-grained contextual modeling via channel-wise selective state propagation. Specifically, Bidirectional Channel Mamba (Bi-CMamba) aligns encoder features across scales and enables bidirectional information exchange among ordered scale groups. Unidirectional Channel Mamba (Uni-CMamba) progressively aggregates decoder features and retains fine-grained scale groups through a group selection mechanism for subsequent fusion. Furthermore, a Hybrid
Attention Module (HAM) is introduced to combine large-kernel local context and Manhattan self-
attention for complementary spatial modeling. Experimental results demonstrate that our method achieves highly competitive performance. Specifically, our model achieves an AbsRel of 0.094 and an RMSE of 4.156 on KITTI, and an AbsRel of 0.140 on DDAD. In the zero-shot cross-
dataset generalization test on NYUv2, it attains an AbsRel of 0.232, outperforming the baseline RA-Depth by 7.2%.
Read the full abstract ▾
All authors Xuezhi Xiang, Jiayao Liu, Heqi Xiang, Yuqi Hu, Yiming Chen, Shanjun Zhang
Computer Vision
Identity-Consistent Analysis of Long-Shot Windsurfing Video: A Domain-Specific Offline Tracking System
Bertil Braun
Long-shot windsurfing
video combines small targets, large camera pans, prolonged overlaps, and rapidly changing backgrounds. The desired output is not a generic MOT trace but a separate, stable rider-relative
video for each surfer; one false identity merge can invalidate an otherwise useful result. We present an offline analysis system that detects surfers, forms conservative local tracklets, links them globally with camera-compensated motion and a foreground-masked sail-color descriptor, and uses two pose keypoints on the rig to drive a rider-relative virtual camera. The tracking stage is evaluated on 21 manually reconstructed development
videos containing 41,004 retained observations. On this fixed-observation protocol, the production system achieves 0.957 pairwise precision, 0.918 recall, and 0.937 F1, compared with 0.792 F1 for OC-SORT and 0.828 for BoT-SORT. Compared with OC-SORT, it reduces fragmentation excess from 845 to 42, but nine of its 95 output tracks mix rider identities and these errors affect seven of the 21
videos.
Read the full abstract ▾
Computer Vision
AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos
Kirill Mazur, Nikita Karaev, Matthew Chang et al. (+2 more)
In this work, we present a method for shape reconstruction and tracking from
video via agentic analysis-by-synthesis. Unlike prior methods which first estimate dense pixel correspondences and then recover object motion from them, our method infers a structured 3D object model, including its geometry and kinematic structure, and uses this model to optimise object track estimates over time. In our optimisation loop, a
Vision-
Language Model (VLM)
agent iteratively refines shape or generalised pose through a render-and-compare loop, combining coarse visual
reasoning with numerical pose optimisation for precise state estimation. This structured formulation enables our method to track through large motion, articulation, and severe occlusion without relying on pixel-matching objectives. Quantitatively, on ARCTIC, our method substantially outperforms state-of-the-art 3D point-tracking baselines for articulated objects, and on HOT3D it outperforms all evaluated rigid-object tracking baselines.
Read the full abstract ▾
All authors Kirill Mazur, Nikita Karaev, Matthew Chang, Jitendra Malik, Nur Muhammad "Mahi” Shafiullah
Computer Vision
VPRune: Efficient Training-free Pre-LLM Visual Token Pruning
Guangchuan Lv, Dianxing Shi and Dingjie Fu
Visual
token pruning is a promising approach to reducing the inference cost of large
vision-
language models (LVLMs), yet aggressive
token reduction often causes substantial performance degradation. We identify three key factors behind this degradation: text-guided selection bias, information loss from discarded
tokens, and positional distortion caused by sequence compaction. Based on these observations, we propose VPRune, a training-free pre-LLM
pruning framework consisting of visual-only diversity selection, similarity-guided
token recycling, and position-preserving restoration. Experiments on FastVLM-1.5B across multiple
vision-
language benchmarks demonstrate that VPRune achieves a favorable accuracy–compression trade-off, with particularly pronounced advantages under aggressive compression. Furthermore, evaluations on edge-device show that VPRune effectively reduces end-to-end inference latency while maintaining superior task performance, demonstrating its practicality for resource-constrained LVLM deployment.
Read the full abstract ▾
Computer Vision
STA-TFM: Spatio-Temporal Aggregation Across Views TransForMer for Pose Estimation
Mena Kamel, Natalie Won, Amrut Sarangi et al. (+2 more)
Monocular 3D human pose estimation (HPE) remains challenging due to depth ambiguity, occlu- sions, and the need for temporal consistency. While multi-view methods provide superior accuracy over
monocular approaches, they often require complex setups. We introduce STA-TFM, a
transformer-based architecture that combines spatial and temporal information for multi-view pose estimation. The approach leverages DSTformer, a
monocular feature extractor, to capture long-range pose dependencies within each view. A fusion
transformer then aggregates information across views to produce coherent 3D estimates. To address training data scarcity, we use a data generation pipeline that transforms any existing 3D pose
dataset into multi-view setups with controllable parameters. Experiments on various
datasets demonstrate that STA-TFM outperforms existing camera-parameter-free multi-view methods. STA-TFM achieves 50.9% and 49.5% reductions in mean per joint position error (MPJPE) and mean per joint velocity error (MPJVE) on the DHP19
dataset. Furthermore, it achieves 6.7% and 7.7% respective reductions on HAA4D, and a 15.2% MPJPE reduction on TotalCapture. STA-TFM handles noisy and missing 2D inputs, supporting potential deployment in healthcare monitoring, athletic assessment, and immersive technologies. Code, training checkpoints, and data are available at https://zenodo.org/records/22832620.
Read the full abstract ▾
All authors Mena Kamel, Natalie Won, Amrut Sarangi, Sven Jager, Albert Pla Planas