Computer Vision
Efficient Point Cloud Processing with High-Dimensional Positional Encoding and Non-Local MLPs
Yanmei Zou, Hongshan Yu, Yaonan Wang et al. (+4 more)
Multi-Layer Perceptron (MLP) models are the foundation of contemporary point cloud processing. However, their complex network architectures obscure the source of their strength and limit the application of these models. In this article, we develop a two-stage abstraction and refinement (ABS-REF) view for modular feature extraction in point cloud processing. This view elucidates that whereas the early models focused on ABS stages, the more recent techniques devise sophisticated REF stages to attain performance advantages. Then, we propose a High-dimensional Positional Encoding (HPE) module to explicitly utilize intrinsic positional information, extending the “positional encoding” concept from
Transformer literature. HPE can be readily deployed in MLP-based architectures and is compatible with
transformer-based methods. Within our ABS-REF view, we rethink local aggregation in MLP-based methods and propose replacing time-consuming local MLP operations, which are used to capture local relationships among neighbors. Instead, we use non-local MLPs for efficient non-local information updates, combined with the proposed HPE for effective local information representation. We leverage our modules to develop HPENets, a suite of MLP networks that follow the ABS-REF paradigm, incorporating a scalable HPE-based REF stage. Extensive experiments on seven public
datasets across four different tasks show that HPENets deliver a strong balance between efficiency and effectiveness. Notably, HPENet surpasses PointNeXt, a strong MLP-based counterpart, by 1.1% mAcc, 4.0% mIoU, 1.8% mIoU, and 0.2% Cls. mIoU, with only 50.0%, 21.5%, 23.1%, 44.4% of FLOPs on ScanObjectNN, S3DIS, ScanNet, and ShapeNetPart, respectively. Source code is available at https://github.com/zouyanmei/HPENet_v2.git.
Read the full abstract ▾
All authors Yanmei Zou, Hongshan Yu, Yaonan Wang, Zhengeng Yang, Xieyuanli Chen, Kailun Yang, Naveed Akhtar
Computer Vision
Real Eyes Realize Faster: Gaze Stability and Pupil Novelty for Efficient Egocentric Learning
Ajan Subramanian, Sumukh Bettadapura and Rohan Sathish
Always-on
egocentric cameras are increasingly used as demonstrations for
embodied robotics, imitation learning, and assistive AR, but the resulting
video streams are dominated by redundant and low-quality frames. Under the storage and battery constraints of wearable devices, choosing which frames to keep is as important as how to learn from them. We observe that modern eye-tracking headsets provide a continuous, training-free side channel that decomposes into two complementary axes: gaze fixation captures visual stability (quality), while pupil response captures arousal-linked moments (novelty). We operationalize this insight as a Dual-Criterion Frame Curator that first gates frames by gaze quality and then ranks the survivors by pupil-derived novelty. On the Visual Experience
Dataset (VEDB), curated frames at 10% budget match the classification performance of the full stream, and naive signal fusion consistently destroys both contributions. The benefit is task-dependent: pupil ranking improves activity recognition, while gaze-only selection already dominates for scene recognition, confirming that the two signals serve genuinely different roles. Our method requires no model inference and operates at capture time, offering a path toward efficient, always-on
egocentric data curation.
Read the full abstract ▾
Computer Vision
CLIP-Guided Multi-Task Regression for Multi-View Plant Phenotyping
Simon Warmers, Muhammad Zawish, Fayaz Ali Dharejo et al. (+2 more)
Modeling plant growth dynamics plays a central role in modern agricultural research. However, learning robust predictors from multi-view plant imagery remains challenging due to strong viewpoint redundancy and viewpoint-dependent appearance changes. We propose a level-aware
vision language framework that jointly predicts plant age and leaf count using a single multi-task model built on CLIP
embeddings. Our method aggregates rotational views into angle-invariant representations and conditions visual features on lightweight text priors encoding viewpoint level for stable prediction under incomplete or unordered inputs. On the GroMo25
benchmark, our approach reduces mean age MAE from 7.74 to 3.91 and mean leaf-count MAE from 5.52 to 3.08 compared to the GroMo baseline, corresponding to improvements of 49.5% and 44.2%, respectively. The unified formulation simplifies the pipeline by replacing the conventional dual-model setup while improving robustness to missing views. The models and code is available at: https://github.com/SimonWarmers/CLIP-MVP
Read the full abstract ▾
All authors Simon Warmers, Muhammad Zawish, Fayaz Ali Dharejo, Steven Davy, Radu Timofte
Computer Vision
EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR
Zhenyu Li, Sai Kumar Dwivedi, Filip Maric et al. (+11 more)
Egocentric human motion estimation is essential for AR/VR experiences, yet remains challenging due to limited body coverage from the
egocentric viewpoint, frequent occlusions, and scarce labeled data. We present EgoPoseFormer v2, a method that addresses these challenges through two key contributions: (1) a
transformer-based model for temporally consistent and spatially grounded body pose estimation, and (2) an auto-labeling system that enables the use of large unlabeled
datasets for training. Our model is fully differentiable, introduces identity-conditioned queries, multi-view spatial refinement, causal temporal
attention, and supports both keypoints and parametric body representations under a constant compute budget. The auto-labeling system scales learning to tens of millions of unlabeled frames via uncertainty-aware semi-supervised training. The system follows a teacher-student schema to generate pseudo-labels and guide training with uncertainty
distillation, enabling the model to generalize to different environments. On the EgoBody3M
benchmark, with a 0.8 ms latency on GPU, our model outperforms two state-of-the-art methods by 12.2% and 19.4% in accuracy, and reduces temporal jitter by 22.2% and 51.7%. Furthermore, our auto-labeling system further improves the wrist MPJPE by 13.1%.
Read the full abstract ▾
All authors Zhenyu Li, Sai Kumar Dwivedi, Filip Maric, Carlos Chacon, Nadine Bertsch, Filippo Arcadu, Tomas Hodan, Michael Ramamonjisoa, Peter Wonka, Amy Zhao, Robin Kips, Cem Keskin, Anastasia Tkach, Chenhongyi Yang
Computer Vision
Revisiting the Role of Foundation Models in Cell-Level Histopathological Image Analysis under Small-Patch Constraints – Effects of Training Data Scale and Blur Perturbations on CNNs and Vision Transformers
Hiroki Kagiyama, Toru Nagasaka, Yukari Adachi et al. (+5 more)
Background and objective:
Cell-level pathological image analysis requires working with extremely small image patches (40x40 pixels), far below standard ImageNet resolutions. It remains unclear whether modern deep learning architectures and foundation models can learn robust and scalable representations under this constraint. We systematically evaluated architectural suitability and data-scale effects for small-patch
cell classification. Methods: We analyzed 303 colorectal cancer specimens with CD103/CD8 immunostaining, generating 185,432 annotated
cell images. Eight task-specific architectures were trained from scratch at multiple data scales (FlagLimit: 256–16,384 samples per class), and three foundation models were evaluated via linear probing and fine-tuning after resizing inputs to 224x224 pixels. Robustness to blur was assessed using pre- and post-resize Gaussian perturbations. Results: Task-specific models improved consistently with increasing data scale, whereas foundation models saturated at moderate sample sizes. A
Vision Transformer optimized for small patches (CustomViT) achieved the highest accuracy, outperforming all foundation models with substantially lower inference cost. Blur robustness was comparable across architectures, with no qualitative advantage observed for foundation models. Conclusion: For
cell-level classification under extreme spatial constraints, task-specific architectures are more effective and efficient than foundation models once sufficient training data are available. Higher clean accuracy does not imply superior robustness, and large pre-trained models offer limited benefit in the small-patch regime.
Read the full abstract ▾
All authors Hiroki Kagiyama, Toru Nagasaka, Yukari Adachi, Takaaki Tachibana, Ryota Ito, Mitsugu Fujita, Kimihiro Yamashita, Yoshihiro Kakeji
Computer Vision · Cross-listed from cs.LG
Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Models
Ziyuan Chen, Yujin Jeong, Tobias Braun et al. (+1 more)
As text-to-image
diffusion models become increasingly deployed in real-world applications, concerns about
backdoor attacks have gained significant
attention. Prior work on text-based
backdoor attacks has largely focused on
diffusion models conditioned on a single lightweight text encoder. However, more recent
diffusion models that incorporate multiple large-scale text encoders remain underexplored in this context. Given the substantially increased number of trainable parameters introduced by multiple text encoders, an important question is whether
backdoor attacks can remain both efficient and effective in such settings. In this work, we study Stable
Diffusion 3, which uses three distinct text encoders and has not yet been systematically analyzed for text-encoder-based
backdoor vulnerabilities. To understand the role of text encoders in
backdoor attacks, we define four categories of attack targets and identify the minimal sets of encoders required to achieve effective performance for each attack objective. Based on this, we further propose Multi-Encoder Lightweight aTtacks (MELT), which trains only low-rank adapters while keeping the pretrained text encoder weight frozen. We demonstrate that tuning fewer than 0.2% of the total encoder parameters is sufficient for successful
backdoor attacks on Stable
Diffusion 3, revealing previously underexplored vulnerabilities in practical attack scenarios in multi-encoder settings.
Read the full abstract ▾
All authors Ziyuan Chen, Yujin Jeong, Tobias Braun, Anna Rohrbach
Computer Vision
TumorFlow: Physics-Guided Longitudinal MRI Synthesis of Glioblastoma Growth
Valentin Biller, Niklas Bubeck, Lucas Zimmer et al. (+6 more)
Glioblastoma exhibits diverse, infiltrative, and patient-specific growth patterns that are only partially visible on routine MRI, making it difficult to reliably assess true tumor extent and personalize treatment planning and follow-up. We present a biophysically-conditioned generative framework that synthesizes biologically realistic 3D
brain MRI volumes from estimated, spatially continuous tumor-concentration fields. Our approach combines a generative model with tumor-infiltration maps that can be propagated through time using a biophysical growth model, enabling fine-grained control over tumor shape and growth while preserving patient anatomy. This enables us to synthesize consistent tumor growth trajectories directly in the space of real patients, providing interpretable, controllable estimation of tumor infiltration and progression beyond what is explicitly observed in imaging. We evaluate the framework on longitudinal glioblastoma cases and demonstrate that it can generate temporally coherent sequences with realistic changes in tumor appearance and surrounding tissue response. These results suggest that integrating mechanistic tumor growth priors with modern generative modeling can provide a practical tool for patient-specific progression visualization and for generating controlled synthetic data to support downstream neuro-oncology workflows. In longitudinal extrapolation, we achieve a consistent 75% Dice overlap with the biophysical model while maintaining a constant PSNR of 25 in the surrounding tissue. Our code is available at: https://github.com/valentin-biller/lgm.git
Read the full abstract ▾
All authors Valentin Biller, Niklas Bubeck, Lucas Zimmer, Ayhan Can Erdur, Sandeep Nagar, Anke Meyer-Baese, Daniel Rückert, Benedikt Wiestler, Jonas Weidner
Computer Vision
Long-Term Visual Localization in Dynamic Benthic Environments: A Dataset, Footprint-Based Ground Truth, and Visual Place Recognition Benchmark
Martin Kvisvik Larsen and Oscar Pizarro
Long-term visual localization has the potential to reduce cost and improve mapping quality in optical benthic monitoring with autonomous
underwater vehicles (AUVs). Despite this potential, long-term visual localization in benthic environments remains understudied, primarily due to the lack of curated
datasets for benchmarking. Moreover, limited georeferencing accuracy and image footprints necessitate precise
geometric information for accurate ground-truthing. In this work, we address these gaps by presenting a curated
dataset for long-term visual localization in benthic environments and a novel method to ground-truth visual localization results for near-nadir
underwater imagery. Our
dataset comprises georeferenced AUV imagery from five benthic reference sites, revisited over periods up to six years, and includes raw and color-corrected stereo imagery, camera calibrations, and sub-decimeter registered camera poses. To our knowledge, this is the first curated
underwater dataset for long-term visual localization spanning multiple sites and photic-zone habitats. Our ground-truthing method estimates 3D seafloor image footprints and links camera views with overlapping footprints, ensuring that ground-truth links reflect shared visual content. Building on this
dataset and ground truth, we
benchmark eight state-of-the-art visual place recognition (VPR) methods and find that Recall@K is significantly lower on our
dataset than on established terrestrial and
underwater benchmarks. Finally, we compare our footprint-based ground truth to a traditional location-based ground truth and show that distance-threshold ground-truthing can overestimate VPR Recall@K at sites with rugged terrain and altitude variations. Together, the curated
dataset, ground-truthing method, and VPR
benchmark provide a stepping stone for advancing long-term visual localization in dynamic benthic environments.
Read the full abstract ▾
Computer Vision
DQE-CIR: Distinctive Query Embeddings through Learnable Attribute Weights and Target Relative Negative Sampling in Composed Image Retrieval
Geon Park, Ji-Hoon Park and Seong-Whan Lee
Composed image
retrieval (CIR) addresses the task of retrieving a target image by jointly interpreting a reference image and a modification text that specifies the intended change. Most existing methods are still built upon contrastive learning frameworks that treat the ground truth image as the only positive instance and all remaining images as negatives. This strategy inevitably introduces relevance suppression, where semantically related yet valid images are incorrectly pushed away, and semantic confusion, where different modification intents collapse into overlapping regions of the
embedding space. As a result, the learned query representations often lack discriminativeness, particularly at fine-grained attribute modifications. To overcome these limitations, we propose distinctive query
embeddings through learnable attribute weights and target relative negative sampling (DQE-CIR), a method designed to learn distinctive query
embeddings by explicitly modeling target relative relevance during training. DQE-CIR incorporates learnable attribute weighting to emphasize distinctive visual features conditioned on the modification text, enabling more precise feature
alignment between
language and
vision. Furthermore, we introduce target relative negative sampling, which constructs a target relative similarity distribution and selects informative negatives from a mid-zone region that excludes both easy negatives and ambiguous false negatives. This strategy enables more reliable
retrieval for fine-grained attribute changes by improving query discriminativeness and reducing confusion caused by semantically similar but irrelevant candidates.
Read the full abstract ▾
Computer Vision
Volumetric Directional Diffusion: Anchoring Uncertainty Quantification in Anatomical Consensus for Ambiguous Medical Image Segmentation
Chao Wu, Mahesh Bhosale, Kangxian Xie et al. (+3 more)
Ambiguous 3D medical image
segmentation often involves boundaries where different expert delineations are non-identical yet clinically plausible. Modeling such inter-observer variability requires a careful balance between diversity and anatomical fidelity: deterministic models preserve coherent volumetric structures but collapse expert disagreement into a single mask, while
stochastic generative models can produce diverse samples but may introduce disconnected components or slice-to-slice inconsistency when generating full 3D masks from unstructured noise. We propose Volumetric Directional
Diffusion (VDD), a prior-anchored
diffusion framework that shifts
stochastic generation from full-mask synthesis to residual boundary exploration. VDD uses a coarse consensus prediction as an anatomical anchor and learns a directional
diffusion process to generate plausible boundary variations around ambiguous regions while preserving stable volumetric
topology. Experiments on three multi-rater
datasets, including LIDC-IDRI, KiTS21, and ISBI 2015, show that VDD improves uncertainty distribution
alignment while maintaining competitive
segmentation accuracy and 3D structural consistency. These results suggest that prior-anchored residual
diffusion can model clinically plausible expert disagreement without sacrificing anatomical fidelity.
Read the full abstract ▾
All authors Chao Wu, Mahesh Bhosale, Kangxian Xie, Pouya Karimian, David Doermann, Mingchen Gao
Computer Vision
Rethinking the Efficiency and Effectiveness of Reinforcement Learning for Radiology Report Generation
Zilin Lu, Ruifeng Yuan, Weiwei Cao et al. (+6 more)
Radiologists highly desire fully automated AI for radiology report generation (R2G), yet existing approaches fall short in clinical utility. Reinforcement learning (RL) holds potential to address these shortcomings, but its adoption in this task remains underexplored. In this paper, we revisit RL in terms of data efficiency and optimization effectiveness for R2G tasks. First, we explore the impact of data quantity and quality on the performance of RL in medical contexts, revealing that data quality plays a more critical role than quantity. To this end, we propose a diagnostic diversity-based data sampling strategy that enables comparable performance with fewer samples. Second, we observe that the majority of
tokens in radiology reports are template-like and diagnostically uninformative, whereas the low frequency of clinically critical
tokens heightens the risk of being overlooked during optimization. To tackle this, we introduce Diagnostic
Token-weighted Policy Optimization (DiTPO), which directly optimizes for clinical accuracy by using a diagnostic F1 score as the reward signal. Unlike standard RL approaches that treat all
tokens equally, DiTPO explicitly models the varying importance of different
tokens through rule- or gradient-based mechanisms to prioritize clinically relevant content. Extensive experiments on the MIMIC-CXR, IU-Xray, and CheXpert Plus
datasets demonstrate that our framework achieves state-of-the-art (SOTA) performance while requiring substantially fewer training samples in RL. Notably, on MIMIC-CXR, our framework attains an F1 score of 0.516 using only 20% of the RL training samples.
Read the full abstract ▾
All authors Zilin Lu, Ruifeng Yuan, Weiwei Cao, Wanxing Chang, Zhongyu Wei, Sinuo Wang, Yong Xia, Ling Zhang, Jianpeng Zhang
Computer Vision
Discriminative Perception via Anchored Description for Reasoning Segmentation
Tao Yang, Qing Zhou, Yanliang Li et al. (+1 more)
Reasoning segmentation increasingly employs reinforcement learning to generate explanatory
reasoning chains that guide
Multimodal Large
Language Models. While these
geometric rewards are primarily confined to guiding the final localization, they are incapable of discriminating whether the
reasoning process remains anchored on the referred region or strays into irrelevant context. Lacking this discriminative guidance, the model's
reasoning often devolves into unfocused and verbose chains that ultimately fail to disambiguate and perceive the target in complex scenes. This suggests a need to complement the RL objective with Discriminative Perception, an ability to actively distinguish a target from its context. To realize this, we propose DPAD to compel the model to generate a descriptive caption of the referred object, which is then used to explicitly discriminate by contrasting the caption's semantic relevance to the referred object against the wider context. By optimizing for this discriminative capability, the model is forced to focus on the unique attributes of the target, leading to a more converged and efficient
reasoning chain. The descriptive caption also serves as an interpretability rationale that aligns with the
segmentation. Experiments on the
benchmarks confirm the validity of our approach, delivering substantial performance gains, with the cIoU on ReasonSeg increasing by 3.09% and the
reasoning chain length decreasing by approximately 42%. Code is available at https://github.com/mrazhou/DPAD
Read the full abstract ▾
All authors Tao Yang, Qing Zhou, Yanliang Li, Qi Wang
Computer Vision
Weakly Supervised Patch Annotation for Improved Screening of Diabetic Retinopathy
Shramana Dey, Abhirup Banerjee, B. Uma Shankar et al. (+2 more)
Diabetic Retinopathy (DR) requires timely screening to prevent irreversible
vision loss. However, its early detection remains a significant challenge since often the subtle pathological manifestations (lesions) get overlooked due to insufficient annotation. Existing literature primarily focuses on image-level supervision, weakly-supervised localization, and clustering-based representation learning, which fail to systematically annotate unlabeled lesion region(s) for refining the
dataset. Expert-driven lesion annotation is labor-intensive and often incomplete, limiting the performance of deep learning models. We introduce Similarity-based Annotation via Feature-space Ensemble (SAFE), a two-stage framework that unifies weak supervision, contrastive learning, and patch-wise
embedding inference, to systematically expand sparse annotations in the pathology. SAFE preserves fine-grained details of the lesion(s) under partial clinical supervision. In the first stage, a dual-arm Patch
Embedding Network learns semantically structured, class-discriminative
embeddings from expert annotated patches. Next, an ensemble of independent
embedding spaces extrapolates labels to the unannotated regions based on spatial and semantic proximity. An abstention mechanism ensures trade-off between highly reliable annotation and noisy coverage. Experimental results demonstrate reliable separation of healthy and diseased patches, achieving upto 0.9886 accuracy. The annotation generated from SAFE substantially improves downstream tasks such as DR classification, demonstrating a substantial increase in F1-score of the diseased class and a performance gain as high as 0.545 in Area Under the Precision-Recall Curve (AUPRC). Qualitative analysis, with explainability, confirms that SAFE focuses on clinically relevant lesion patterns; and is further validated by ophthalmologists.
Read the full abstract ▾
All authors Shramana Dey, Abhirup Banerjee, B. Uma Shankar, Ramachandran Rajalakshmi, Sushmita Mitra
Computer Vision
Faces in Things, Faces in Nothing: Probing Face Selectivity in Vision Models with Pareidolia
Qianpu Chen, Derya Soydaner and Rob Saunders
An electrical outlet can look like a face without depicting a person. Face pareidolia is thus a revealing probe of how
vision models turn ambiguous visual evidence into category judgments. When a model calls such an object Human, does that answer depend on its face-like arrangement? Using the FacesInThings
dataset, we test six
vision models, from a
vision-only ViT to CLIP and generative
vision-
language models, with face detectors as a reference. Each face-like region is paired with three controls from the same photograph that lack the arrangement: a background crop, the region with its tiles shuffled, and its average color. A Human answer driven by the arrangement should appear less often on these controls. We also vary how the categories are described. Three findings emerge. First, models respond to illusory faces, but not as people do: face detectors rarely return a face for these regions, yet CLIP ViT-L/14 and two generative models label roughly half of them Human when the categories are described as kinds of face. Annotators, by contrast, read these faces as non-human, such as animal or cartoon faces. For these models, a face is a human face by default. Second, similar Human rates hide opposite behavior. LLaVA-1.5 answers Human less often once the arrangement is removed; Qwen2-VL calls every average-color patch Human and describes a person in each; CLIP ViT-B/32 calls shuffled regions Human more often than real faces; SmolVLM rarely answers Human, but mostly for face-like regions. Third, rewording the categories changes whether Human answers favor the face at all, removing this preference in LLaVA-1.5 and creating it in CLIP. A Human answer is not evidence of a face:
vision models can keep giving it after the face is gone.
Read the full abstract ▾
Computer Vision
RIVER: A Real-Time Interaction Benchmark for Video LLMs
Yansong Shi, Qingsong Zhao, Tianxiang Jiang et al. (+3 more)
The rapid advancement of
multimodal large
language models has demonstrated impressive capabilities, yet nearly all operate in an offline paradigm, hindering real-time interactivity. Addressing this gap, we introduce the Real-tIme
Video intERaction Bench (RIVER Bench), designed for evaluating online
video comprehension. RIVER Bench introduces a novel framework comprising Retrospective
Memory, Live-Perception, and Proactive Anticipation tasks, closely mimicking interactive dialogues rather than responding to entire
videos at once. We conducted detailed annotations using
videos from diverse sources and varying lengths, and precisely defined the real-time interactive format. Evaluations across various model categories reveal that while offline models perform well in single question-answering tasks, they struggle with real-time processing. Addressing the limitations of existing models in online
video interaction, especially their deficiencies in long-term
memory and future perception, we proposed a general improvement method that enables models to interact with users more flexibly in real time. We believe this work will significantly advance the development of real-time interactive
video understanding models and inspire future research in this emerging field.
Datasets and code are publicly available at https://github.com/OpenGVLab/RIVER.
Read the full abstract ▾
All authors Yansong Shi, Qingsong Zhao, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, Limin Wang
Computer Vision
GeoSeg: Training-Free Reasoning-Driven Segmentation in Remote Sensing Imagery
Lifan Jiang, Yuhang Pei, oxi Wu et al. (+5 more)
Recent advances in MLLMs are reframing
segmentation from fixed-category prediction to instruction-grounded localization. While
reasoning based
segmentation has progressed rapidly in natural scenes, remote sensing lacks a generalizable solution due to the prohibitive cost of
reasoning-oriented data and domain-specific challenges like overhead viewpoints. We present GeoSeg, a zero-shot, training-free framework that bypasses the supervision bottleneck for
reasoning-driven remote sensing
segmentation. GeoSeg couples MLLM
reasoning with precise localization via: (i) bias-aware coordinate refinement to correct systematic grounding shifts and (ii) a dual-route prompting mechanism to fuse semantic intent with fine-grained spatial cues. We also introduce GeoSeg-Bench, a diagnostic
benchmark of 810 image–query pairs with hierarchical difficulty levels. Experiments show that GeoSeg consistently outperforms all baselines, with extensive ablations confirming the effectiveness and necessity of each component.
Read the full abstract ▾
All authors Lifan Jiang, Yuhang Pei, oxi Wu, Yan Zhao, Tianrun Wu, Shulong Yu, Lihui Zhang, Deng Cai
Computer Vision · Cross-listed from cs.AI
Jyoti Aneja, Michael Harrison, Neel Joshi et al. (+3 more)
We present Phi-4-
reasoning-
vision-15B, a compact open-weight
multimodal reasoning model, and share the motivations, design choices, experiments, and learnings that informed its development. Our goal is to contribute practical insight to the research community on building smaller, efficient
multimodal reasoning models and to share the result of these learnings as an open-weight model that is good at common
vision and
language tasks and excels at scientific and mathematical
reasoning and understanding user interfaces. Our contributions include demonstrating that careful architecture choices and rigorous data curation enable smaller, open-weight
multimodal models to achieve competitive performance with significantly less training and inference-time compute and
tokens. The most substantial improvements come from systematic filtering, error correction, and synthetic augmentation – reinforcing that data quality remains the primary lever for model performance. Systematic ablations show that high-resolution, dynamic-resolution encoders yield consistent improvements, as accurate perception is a prerequisite for high-quality
reasoning. Finally, a hybrid mix of
reasoning and non-
reasoning data with explicit mode
tokens allows a single model to deliver fast direct answers for simpler tasks and chain-of-thought
reasoning for complex problems.
Read the full abstract ▾
All authors Jyoti Aneja, Michael Harrison, Neel Joshi, Tyler LaBonte, John Langford, Eduardo Salinas
Computer Vision · Cross-listed from cs.LG
Dual-Solver: A Generalized ODE Solver for Diffusion Models with Dual Prediction
Soochul Park and Yeon Ju Lee
Diffusion models achieve state-of-the-art image quality. However, sampling is costly at inference time because it requires a large number of function evaluations (NFEs). To reduce NFEs, classical ODE numerical methods have been adopted. Yet, the choice of prediction type and integration domain leads to different sampling behaviors. To address these issues, we introduce Dual-Solver, which generalizes multistep samplers through learnable parameters that continuously (i) interpolate among prediction types, (ii) select the integration domain, and (iii) adjust the residual terms. It retains the standard predictor-corrector structure while preserving second-order local accuracy. These parameters are learned via a classification-based objective using a frozen pretrained classifier (e.g., MobileNet or CLIP). For ImageNet class-conditional generation (DiT, GM-DiT) and text-to-image generation (SANA, PixArt-$\alpha$), Dual-Solver improves FID and CLIP scores in the low-NFE regime ($3 \le$ NFE $\le 9$) across backbones.
Read the full abstract ▾
Computer Vision
Scaling Dense Event-Stream Pretraining from Visual Foundation Models
Zhiwen Chen, Junhui Hou, Zhiyu Zhu et al. (+2 more)
Learning versatile, fine-grained representations from irregular event streams is pivotal yet nontrivial, primarily due to the heavy annotation that hinders scalability in
dataset size, semantic richness, and application scope. To mitigate this dilemma, we launch a novel self-supervised pretraining method that distills visual foundation models (VFMs) to push the boundaries of event representation at scale. Specifically, we curate an extensive synchronized image-event collection to amplify cross-modal
alignment. Nevertheless, due to inherent mismatches in sparsity and granularity between image-event domains, existing
distillation paradigms are prone to semantic collapse in event representations, particularly at high resolutions. To bridge this gap, we propose to extend the
alignment objective to semantic structures provided off-the-shelf by VFMs, indicating a broader receptive field and stronger supervision. The key ingredient of our method is a structure-aware
distillation loss that grounds higher-quality image-event correspondences for
alignment, optimizing dense event representations. Extensive experiments demonstrate that our approach takes a great leap in downstream
benchmarks, significantly surpassing traditional methods and existing pretraining techniques. This breakthrough manifests in enhanced generalization, superior data efficiency and elevated transferability.
Read the full abstract ▾
All authors Zhiwen Chen, Junhui Hou, Zhiyu Zhu, Jinjian Wu, Guangming Shi
Computer Vision
UniRain: Unified Image Deraining with RAG-based Dataset Distillation and Multi-objective Reweighted Optimization
Qianfeng Yang, Qiyuan Guan, Xiang Chen et al. (+3 more)
Despite significant progress has been made in image deraining, we note that most existing methods are often developed for only specific types of rain degradation and fail to generalize across diverse real-world rainy scenes. How to effectively model different rain degradations within a universal framework is important for real-world image deraining. In this paper, we propose UniRain, an effective unified image deraining framework capable of restoring images degraded by rain streak and raindrop under both daytime and nighttime conditions. To better enhance unified model generalization, we construct an intelligent
retrieval augmented generation (RAG)-based
dataset distillation pipeline that selects high-quality training samples from all public deraining
datasets for better mixed training. Furthermore, we incorporate a simple yet effective multi-objective reweighted optimization strategy into the asymmetric mixture-of-experts (MoE) architecture to facilitate consistent performance and improve robustness across diverse scenes. Extensive experiments show that our framework performs favorably against the state-of-the-art models on our proposed
benchmarks and multiple public
datasets.
Read the full abstract ▾
All authors Qianfeng Yang, Qiyuan Guan, Xiang Chen, Jiyu Jin, Guiyue Jin, Jiangxin Dong
Computer Vision
BLOCK: An Open-Source Bi-Stage MLLM Character-to-Skin Pipeline for Minecraft
Hengquan Guo
We present BLOCK, an open-source bi-stage character-to-skin pipeline that generates pixel-perfect Minecraft skins from arbitrary character concepts. BLOCK decomposes the problem into (i) a 3D preview synthesis stage driven by a large
multimodal model (MLLM) with a carefully designed
prompt-and-reference template, producing a consistent dual-panel (front/back) oblique-view Minecraft-style preview; and (ii) a skin decoding stage based on a fine-tuned FLUX.2 model that translates the preview into a skin atlas image. We further propose EvolveLoRA, a progressive LoRA curriculum (text-to-image $\rightarrow$ image-to-image $\rightarrow$ preview-to-skin) that initializes each phase from the previous adapter to improve stability and efficiency. BLOCK is released with all
prompt templates and fine-tuned weights to support reproducible character-to-skin generation.
Read the full abstract ▾
Computer Vision
ProFound: A moderate-sized vision foundation model for multi-task prostate imaging
Yipei Wang, Yinsong Xu, Weixi Yi et al. (+11 more)
Many diagnostic and therapeutic clinical tasks for prostate cancer increasingly rely on multi-parametric MRI. Automating these tasks is challenging because they necessitate expert interpretations, which are difficult to scale to capitalise on modern deep learning. Although modern automated systems achieve expert-level performance in isolated tasks, their general clinical utility remains limited by the requirement of large task-specific labelled
datasets. In this paper, we present ProFound, a domain-specialised
vision foundation model for volumetric prostate mpMRI. ProFound is pre-trained using several variants of self-supervised approaches on a diverse, multi-institutional collection of 5,000 patients, with a total of over 22,000 unique 3D MRI volumes (over 1,800,000 2D image slices). We conducted a systematic evaluation of ProFound across a broad spectrum of $11$ downstream clinical tasks on over 3,000 independent patients, including prostate cancer detection, Gleason grading, lesion localisation, gland volume estimation, zonal and surrounding structure
segmentation. Experimental results demonstrate that finetuned ProFound consistently outperforms or remains competitive with state-of-the-art specialised models and existing medical
vision foundation models trained/finetuned on the same data.
Read the full abstract ▾
All authors Yipei Wang, Yinsong Xu, Weixi Yi, Shaheer Ullah Saeed, Natasha Thorley, Alexander Ng, Yukun Zhou, Wen Yan, Dean Barratt, Shonit Punwani, Veeru Kasivisvanathan, Mark Emberton, Daniel C. Alexander, Yipeng Hu
Computer Vision · Cross-listed from cs.RO
Structural Action Transformer for 3D Dexterous Manipulation
Xiaohan Lei, Min Wang, Bohong Weng et al. (+2 more)
Achieving human-level dexterity in
robots via imitation learning from heterogeneous
datasets is hindered by the challenge of cross-embodiment skill transfer, particularly for high-DoF robotic hands. Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations and fail to handle embodiment heterogeneity. This paper proposes the Structural Action
Transformer (SAT), a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective. We reframe each action chunk not as a temporal sequence, but as a variable-length, unordered sequence of joint-wise trajectories. This structural formulation allows a
Transformer to natively handle heterogeneous embodiments, treating the joint count as a variable sequence length. To encode structural priors and resolve ambiguity, we introduce an
Embodied Joint Codebook that embeds each joint's functional role and kinematic properties. Our model learns to generate these trajectories from 3D point clouds via a continuous-time flow matching objective. We validate our approach by pre-training on large-scale heterogeneous
datasets and fine-tuning on simulation and real-world dexterous manipulation tasks. Our method consistently outperforms all baselines, demonstrating superior sample efficiency and effective cross-embodiment skill transfer. This structural-centric representation offers a new path toward scaling policies for high-DoF, heterogeneous manipulators.
Read the full abstract ▾
All authors Xiaohan Lei, Min Wang, Bohong Weng, Wengang Zhou, Houqiang Li
Computer Vision
Towards Generalized Multimodal Homography Estimation
Jinkun You, Jiaxin Cheng, Jie Zhang et al. (+1 more)
Supervised and unsupervised homography estimation methods depend on image pairs tailored to specific modalities to achieve high accuracy. However, their performance deteriorates substantially when applied to unseen modalities. To address this issue, we propose a training data synthesis method that generates unaligned image pairs with ground-truth offsets from a single input image. Our approach renders the image pairs with diverse textures and colors while preserving their structural information. These synthetic data empower the trained model to achieve greater robustness and improved generalization across various domains. Additionally, we design a network to fully leverage cross-scale information and decouple color information from feature representations, thus improving estimation accuracy. Extensive experiments show that our training data synthesis method improves generalization performance. The results also confirm the effectiveness of the proposed network.
Read the full abstract ▾
All authors Jinkun You, Jiaxin Cheng, Jie Zhang, Yicong Zhou
Computer Vision · Cross-listed from cs.RO
Jaewon Lee, Jaeseok Heo, Gunmin Lee et al. (+3 more)
Safe visual
navigation is critical for indoor mobile
robots operating in cluttered environments. Existing
benchmarks, however, often neglect collisions or are designed for outdoor scenarios, making them unsuitable for indoor visual
navigation. To address this limitation, we introduce the reactive visual
navigation benchmark (RVN-Bench), a collision-aware
benchmark for indoor mobile
robots. In RVN-Bench, an
agent must reach sequential goal positions in previously unseen environments using only visual observations and no prior map, while avoiding collisions. Built on the Habitat 2.0 simulator and leveraging high-fidelity HM3D scenes, RVN-Bench provides large-scale, diverse indoor environments, defines a collision-aware
navigation task and evaluation metrics, and offers tools for standardized training and benchmarking. RVN-Bench supports both online and offline learning by offering an environment for online reinforcement learning, a trajectory image
dataset generator, and tools for producing negative trajectory image
datasets that capture collision events. Evaluations demonstrate that policies trained on RVN-Bench generalize effectively across unseen simulated environments. Furthermore, initial physical experiments using a Jackal UGV indicate promising sim-to-real transfer. Code and additional materials are available at: https://sequor-robotics-research.github.io/projects/RVN-Bench/.
Read the full abstract ▾
All authors Jaewon Lee, Jaeseok Heo, Gunmin Lee, Howoong Jun, Jeongwoo Oh, Songhwai Oh
Computer Vision
SCP: Spatial Causal Prediction in Video
Yanguang Zhao, Jie Yang, Shengqiong Wu et al. (+9 more)
Spatial
reasoning, the ability to understand spatial relations, causality, and dynamic
evolution, is central to human intelligence and essential for real-world applications such as autonomous driving and robotics. Existing studies, however, primarily assess models on visible spatio-temporal understanding, overlooking their ability to infer unseen past or future spatial states. In this work, we introduce Spatial Causal Prediction (SCP), a new task paradigm that challenges models to reason beyond observation and predict spatial causal outcomes. We further construct SCP-Bench, a
benchmark comprising 2,500 QA pairs across 1,181
videos spanning diverse viewpoints, scenes, and causal directions, to support systematic evaluation. Through comprehensive experiments on {23} state-of-the-art models, we reveal substantial gaps between human and model performance, limited temporal extrapolation, and weak causal grounding. We further analyze key factors influencing performance and propose perception-enhancement and
reasoning-guided strategies toward advancing spatial causal intelligence. The project page is https://guangstrip.github.io/SCP-Bench.
Read the full abstract ▾
All authors Yanguang Zhao, Jie Yang, Shengqiong Wu, Shutong Hu, Hongbo Qiu, Yu Wang, Guijia Zhang, Tan Kai Ze, Hao Fei, Chia-Wen Lin, Mong-Li Lee, Wynne Hsu
Computer Vision
Slice-wise quality assessment of high b-value breast DWI via deep learning-based artifact detection
Ameya Markale, Luise Brock, Ihor Horishnyi et al. (+10 more)
Diffusion-weighted imaging (DWI) can support lesion detection and characterization in breast
magnetic resonance imaging (MRI), however especially high b-value
diffusion-weighted acquisitions can be prone to intensity artifacts that can affect diagnostic image assessment. This study aims to detect both hyper- and hypointense artifacts on high b-value
diffusion-weighted images (b=1500 s/mm2) using deep learning, employing either a binary classification (artifact presence) or a multiclass classification (artifact intensity) approach on a slice-wise
dataset.This IRB-approved retrospective study used the single-center
dataset comprising n=11806 slices from routine 3T breast MRI examinations performed between 2022 and mid-2023. Three convolutional neural network (CNN) architectures (DenseNet121, ResNet18, and SEResNet50) were trained for binary classification of hyper- and hypointense artifacts. The best performing model (DenseNet121) was applied to an independent holdout test set and was further trained separately for multiclass classification. Evaluation included area under receiver operating characteristic curve (AUROC), area under precision recall curve (AUPRC), precision, and recall, as well as analysis of predicted bounding box positions, derived from the network Grad-CAM heatmaps. DenseNet121 achieved AUROCs of 0.92 and 0.94 for hyper- and hypointense artifact detection, respectively, and weighted AUROCs of 0.85 and 0.88 for multiclass classification on single-slice high b-value
diffusion-weighted images. A radiologist evaluated bounding box precision on a 1-5 Likert-like scale across 200 slices, achieving mean scores of 3.33+-1.04 for hyperintense artifacts and 2.62+-0.81 for hypointense artifacts. Hyper- and hypointense artifact detection in slice-wise breast DWI MRI
dataset (b=1500 s/mm2) using CNNs particularly DenseNet121, seems promising and requires further validation.
Read the full abstract ▾
All authors Ameya Markale, Luise Brock, Ihor Horishnyi, Dominika Skwierawska, Tri-Thien Nguyen, Hannes Schreiter, Shirin Heidarikahkesh, Lorenz A. Kapsner, Michael Uder, Sabine Ohlmeyer, Frederik B Laun, Andrzej Liebert, Sebastian Bickelhaupt
Computer Vision
Cross-Modal Mapping and Dual-Branch Reconstruction for 2D-3D Multimodal Industrial Anomaly Detection
Radia Daci, Vito Renò, Cosimo Patruno et al. (+4 more)
Multimodal industrial anomaly detection benefits from integrating RGB appearance with 3D surface geometry, yet existing unsupervised approaches commonly rely on
memory banks, teacher-student architectures, or fragile fusion schemes, limiting robustness under noisy depth, weak texture, or missing modalities. This paper introduces CMDR-IAD, a lightweight and modality-flexible unsupervised framework for reliable anomaly detection in 2D+3D
multimodal as well as single-modality (2D-only or 3D-only) settings. CMDR-IAD combines bidirectional 2D$\leftrightarrow$3D cross-modal mapping to model appearance-geometry consistency with dual-branch reconstruction that independently captures normal texture and
geometric structure. A two-part fusion strategy integrates these cues: a reliability-gated mapping anomaly highlights spatially consistent texture-geometry discrepancies, while a confidence-weighted reconstruction anomaly adaptively balances appearance and
geometric deviations, yielding stable and precise anomaly localization even in depth-sparse or low-texture regions. On the MVTec 3D-AD
benchmark, CMDR-IAD achieves state-of-the-art performance while operating without
memory banks, reaching 97.3% image-level AUROC (I-AUROC), 99.6% pixel-level AUROC (P-AUROC), and 97.6% AUPRO. On a real-world polyurethane cutting
dataset, the 3D-only variant attains 92.6% I-AUROC and 92.5% P-AUROC, demonstrating strong effectiveness under practical industrial conditions. These results highlight the framework's robustness, modality flexibility, and the effectiveness of the proposed fusion strategies for industrial visual inspection. Our source code is available at https://github.com/ECGAI-Research/CMDR-IAD/
Read the full abstract ▾
All authors Radia Daci, Vito Renò, Cosimo Patruno, Angelo Cardellicchio, Abdelmalik Taleb-Ahmed, Marco Leo, Cosimo Distante
Computer Vision
DISC: Dense Integrated Semantic Context for Large-Scale Open-Set Semantic Mapping
Felix Igelbrink, Lennart Niecksch, Martin Atzmueller et al. (+1 more)
Open-set semantic mapping enables
language-driven robotic perception, but current instance-centric approaches are bottlenecked by context-depriving and computationally expensive crop-based feature extraction. To overcome this fundamental limitation, we introduce DISC (Dense Integrated Semantic Context), featuring a novel single-pass, distance-weighted extraction mechanism. By deriving high-fidelity CLIP
embeddings directly from the
vision transformer's intermediate layers, our approach eliminates the latency and domain-shift artifacts of traditional image cropping, yielding pure, mask-aligned semantic representations. To fully leverage these features in large-scale continuous mapping, DISC is built upon a fully GPU-accelerated architecture that replaces periodic offline processing with precise, on-the-fly voxel-level instance refinement. We evaluate our approach on standard
benchmarks (Replica, ScanNet) and a newly generated large-scale-mapping
dataset based on Habitat-Matterport 3D (HM3DSEM) to assess scalability across complex scenes in multi-story buildings. Extensive evaluations demonstrate that DISC significantly surpasses current state-of-the-art zero-shot methods in both semantic accuracy and query
retrieval, providing a robust, real-time capable framework for robotic deployment. The full source code, data generation and evaluation pipelines will be made available at https://github.com/DFKI-NI/DISC.
Read the full abstract ▾
All authors Felix Igelbrink, Lennart Niecksch, Martin Atzmueller, Joachim Hertzberg
Computer Vision
N-gram Injection into Transformers for Dynamic Language Model Adaptation in Handwritten Text Recognition
Florent Meyer, Laurent Guichard, Yann Soullard et al. (+3 more)
Transformer-based encoder-decoder networks have recently achieved impressive results in handwritten text recognition, partly thanks to their auto-regressive decoder which implicitly learns a
language model. However, such networks suffer from a large performance drop when evaluated on a target corpus whose
language distribution is shifted from the source text seen during training. To retain recognition accuracy despite this
language shift, we propose an external n-gram injection (NGI) for dynamic adaptation of the network's
language modeling at inference time. Our method allows switching to an n-gram
language model estimated on a corpus close to the target distribution, therefore mitigating bias without any extra training on target image-text pairs. We opt for an early injection of the n-gram into the
transformer decoder so that the network learns to fully leverage text-only data at the low additional cost of n-gram inference. Experiments on three handwritten
datasets demonstrate that the proposed NGI significantly reduces the performance gap between source and target corpora.
Read the full abstract ▾
All authors Florent Meyer, Laurent Guichard, Yann Soullard, Denis Coquenet, Guillaume Gravier, Bertrand Coüasnon