Computer Vision
Task-Aware 3D Affordance Segmentation via 2D Guidance and Geometric Refinement
Lian He, Meng Liu, Qilang Ye et al. (+3 more)
Understanding 3D scene-level affordances from natural
language instructions is essential for enabling
embodied agents to interact meaningfully in complex environments. However, this task remains challenging due to the need for semantic
reasoning and spatial grounding. Existing methods mainly focus on object-level affordances or merely lift 2D predictions to 3D, neglecting rich
geometric structure information in point clouds and incurring high computational costs. To address these limitations, we introduce Task-Aware 3D Scene-level Affordance
segmentation (TASA), a novel geometry-optimized framework that jointly leverages 2D semantic cues and 3D
geometric reasoning in a coarse-to-fine manner. To improve the affordance detection efficiency, TASA features a task-aware 2D affordance detection module to identify manipulable points from
language and visual inputs, guiding the selection of task-relevant views. To fully exploit 3D
geometric information, a 3D affordance refinement module is proposed to integrate 2D semantic priors with local 3D geometry, resulting in accurate and spatially coherent 3D affordance masks. Experiments on SceneFun3D demonstrate that TASA significantly outperforms the baselines in both accuracy and efficiency in scene-level affordance
segmentation.
Read the full abstract ▾
All authors Lian He, Meng Liu, Qilang Ye, Yu Zhou, Xiang Deng, Gangyi Ding
Computer Vision
EPSegFZ: Efficient Point Cloud Semantic Segmentation for Few- and Zero-Shot Scenarios with Language Guidance
Jiahui Wang, Haiyue Zhu, Haoren Guo et al. (+3 more)
Recent approaches for few-shot 3D point cloud semantic
segmentation typically require a two-stage learning process, i.e., a pre-training stage followed by a few-shot training stage. While effective, these methods face overreliance on pre-training, which hinders model flexibility and adaptability. Some models tried to avoid pre-training yet failed to capture ample information. In addition, current approaches focus on visual information in the support set and neglect or do not fully exploit other useful data, such as textual annotations. This inadequate utilization of support information impairs the performance of the model and restricts its zero-shot ability. To address these limitations, we present a novel pre-training-free network, named Efficient Point Cloud Semantic
Segmentation for Few- and Zero-shot scenarios. Our EPSegFZ incorporates three key components. A Prototype-Enhanced Registers
Attention (ProERA) module and a Dual Relative Positional Encoding (DRPE)-based cross-
attention mechanism for improved feature extraction and accurate query-prototype correspondence construction without pre-training. A
Language-Guided Prototype
Embedding (LGPE) module that effectively leverages textual information from the support set to improve few-shot performance and enable zero-shot inference. Extensive experiments show that our method outperforms the state-of-the-art method by 5.68% and 3.82% on the S3DIS and ScanNet
benchmarks, respectively.
Read the full abstract ▾
All authors Jiahui Wang, Haiyue Zhu, Haoren Guo, Abdullah Al Mamun, Cheng Xiang, Tong Heng Lee
Computer Vision · Cross-listed from cs.LG
Toward Dignity-Aware AI: Next-Generation Elderly Monitoring from Fall Detection to ADL
Xun Shao, Aoba Otani, Yuto Hirasuka et al. (+2 more)
This position paper envisions a next-generation elderly monitoring system that moves beyond fall detection toward the broader goal of Activities of Daily Living (ADL) recognition. Our ultimate aim is to design
privacy-preserving, edge-deployed, and
federated AI systems that can robustly detect and understand daily routines, supporting independence and dignity in aging societies. At present, ADL-specific
datasets are still under collection. As a preliminary step, we demonstrate feasibility through experiments using the SISFall
dataset and its GAN-augmented variants, treating fall detection as a proxy task. We report initial results on
federated learning with non-IID conditions, and embedded deployment on Jetson Orin Nano devices. We then outline open challenges such as domain shift, data scarcity, and
privacy risks, and propose directions toward full ADL monitoring in smart-room environments. This work highlights the transition from single-task detection to comprehensive daily activity recognition, providing both early evidence and a roadmap for sustainable and human-centered elderly care AI.
Read the full abstract ▾
All authors Xun Shao, Aoba Otani, Yuto Hirasuka, Runji Cai, Seng W. Loke
Computer Vision · Cross-listed from cs.AI
Value-Aligned Prompt Moderation via Zero-Shot Agentic Rewriting for Safe Image Generation
Xin Zhao, Xiaojun Chen, Bingshan Liu et al. (+3 more)
Generative
vision-
language models like Stable
Diffusion demonstrate remarkable capabilities in creative media synthesis, but they also pose substantial risks of producing unsafe, offensive, or culturally inappropriate content when prompted adversarially. Current defenses struggle to align outputs with human values without sacrificing generation quality or incurring high costs. To address these challenges, we introduce VALOR (Value-Aligned LLM-Overseen Rewriter), a modular, zero-shot agentic framework for safer and more helpful text-to-image generation. VALOR integrates layered
prompt analysis with human-aligned value
reasoning: a multi-level NSFW detector filters lexical and semantic risks; a cultural value
alignment module identifies violations of social norms, legality, and representational ethics; and an intention disambiguator detects subtle or indirect unsafe implications. When unsafe content is detected,
prompts are selectively rewritten by a large
language model under dynamic, role-specific instructions designed to preserve user intent while enforcing
alignment. If the generated image still fails a safety check, VALOR optionally performs a stylistic regeneration to steer the output toward a safer visual domain without altering core semantics. Experiments across
adversarial, ambiguous, and value-sensitive
prompts show that VALOR significantly reduces unsafe outputs by up to 100.00% while preserving
prompt usefulness and creativity. These results highlight VALOR as a scalable and effective approach for deploying safe, aligned, and helpful image generation systems in open-world settings.
Read the full abstract ▾
All authors Xin Zhao, Xiaojun Chen, Bingshan Liu, Zeyao Liu, Zhendong Zhao, Xiaoyan Gu
Computer Vision · Cross-listed from cs.LG
AnchorDS: Anchoring Dynamic Sources for Semantically Consistent Text-to-3D Generation
Jiayin Zhu, Linlin Yang, Yicong Li et al. (+1 more)
Optimization-based text-to-3D methods distill guidance from 2D generative models via Score
Distillation Sampling (SDS), but implicitly treat this guidance as static. This work shows that ignoring source dynamics yields inconsistent trajectories that suppress or merge semantic cues, leading to "semantic over-smoothing" artifacts. As such, we reformulate text-to-3D optimization as mapping a dynamically evolving source distribution to a fixed target distribution. We cast the problem into a dual-conditioned latent space, conditioned on both the text
prompt and the intermediately rendered image. Given this joint setup, we observe that the image condition naturally anchors the current source distribution. Building on this insight, we introduce AnchorDS, an improved score
distillation mechanism that provides state-anchored guidance with image conditions and stabilizes generation. We further penalize erroneous source estimates and design a lightweight filter strategy and fine-tuning strategy that refines the anchor with negligible overhead. AnchorDS produces finer-grained detail, more natural colours, and stronger semantic consistency, particularly for complex
prompts, while maintaining efficiency. Extensive experiments show that our method surpasses previous methods in both quality and efficiency.
Read the full abstract ▾
All authors Jiayin Zhu, Linlin Yang, Yicong Li, Angela Yao
Computer Vision · Cross-listed from cs.LG
Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
Fei Song, Yi Li, Rui Wang et al. (+3 more)
Test-time
prompt tuning for
vision-
language models has demonstrated impressive generalization capabilities under zero-shot settings. However, tuning the learnable
prompts solely based on unlabeled test data may induce
prompt optimization bias, ultimately leading to suboptimal performance on downstream tasks. In this work, we analyze the underlying causes of
prompt optimization bias from both the model and data perspectives. In terms of the model, the
entropy minimization objective typically focuses on reducing the
entropy of model predictions while overlooking their correctness. This can result in overconfident yet incorrect outputs, thereby compromising the quality of
prompt optimization. On the data side,
prompts affected by optimization bias can introduce misalignment between visual and textual modalities, which further aggravates the
prompt optimization bias. To this end, we propose a Doubly Debiased Test-Time
Prompt Tuning method. Specifically, we first introduce a dynamic
retrieval-augmented modulation module that retrieves high-confidence knowledge from a dynamic knowledge base using the test image feature as a query, and uses the retrieved knowledge to modulate the predictions. Guided by the refined predictions, we further develop a reliability-aware
prompt optimization module that incorporates a confidence-based weighted ensemble and cross-modal consistency
distillation to impose regularization constraints during
prompt tuning. Extensive experiments across 15
benchmark datasets involving both natural distribution shifts and cross-
datasets generalization demonstrate that our method outperforms baselines, validating its effectiveness in mitigating
prompt optimization bias.
Read the full abstract ▾
All authors Fei Song, Yi Li, Rui Wang, Jiahuan Zhou, Changwen Zheng, Jiangmeng Li
Computer Vision · Cross-listed from cs.LG
Hierarchical Schedule Optimization for Fast and Robust Diffusion Model Sampling
Aihua Zhu, Rui Su, Qinglin Zhao et al. (+3 more)
Diffusion probabilistic models have set a new standard for generative fidelity but are hindered by a slow iterative sampling process. A powerful training-free strategy to accelerate this process is Schedule Optimization, which aims to find an optimal distribution of timesteps for a fixed and small Number of Function Evaluations (NFE) to maximize sample quality. To this end, a successful schedule optimization method must adhere to four core principles: effectiveness, adaptivity, practical robustness, and computational efficiency. However, existing paradigms struggle to satisfy these principles simultaneously, motivating the need for a more advanced solution. To overcome these limitations, we propose the Hierarchical-Schedule-Optimizer (HSO), a novel and efficient bi-level optimization framework. HSO reframes the search for a globally optimal schedule into a more tractable problem by iteratively alternating between two synergistic levels: an upper-level global search for an optimal initialization strategy and a lower-level local optimization for schedule refinement. This process is guided by two key innovations: the Midpoint Error Proxy (MEP), a solver-agnostic and numerically stable objective for effective local optimization, and the Spacing-Penalized Fitness (SPF) function, which ensures practical robustness by penalizing pathologically close timesteps. Extensive experiments show that HSO sets a new state-of-the-art for training-free sampling in the extremely low-NFE regime. For instance, with an NFE of just 5, HSO achieves a remarkable FID of 11.94 on LAION-Aesthetics with Stable
Diffusion v2.1. Crucially, this level of performance is attained not through costly retraining, but with a one-time optimization cost of less than 8 seconds, presenting a highly practical and efficient paradigm for
diffusion model acceleration.
Read the full abstract ▾
All authors Aihua Zhu, Rui Su, Qinglin Zhao, Li Feng, Meng Shen, Shibo He
Computer Vision · Cross-listed from cs.LG
Stratified Knowledge-Density Super-Network for Scalable Vision Transformers
Longhua Li, Lei Qi and Xin Geng
Training and deploying multiple
vision transformer (ViT) models for different resource constraints is costly and inefficient. To address this, we propose transforming a pre-trained ViT into a stratified knowledge-density super-network, where knowledge is hierarchically organized across weights. This enables flexible extraction of sub-networks that retain maximal knowledge for varying model sizes. We introduce Weighted PCA for Attention Contraction (WPAC), which concentrates knowledge into a compact set of critical weights. WPAC applies
token-wise weighted principal component analysis to intermediate features and injects the resulting transformation and inverse
matrices into adjacent layers, preserving the original network function while enhancing knowledge compactness. To further promote stratified knowledge organization, we propose Progressive Importance-Aware Dropout (PIAD). PIAD progressively evaluates the importance of weight groups, updates an importance-aware dropout list, and trains the super-network under this dropout regime to promote knowledge stratification. Experiments demonstrate that WPAC outperforms existing
pruning criteria in knowledge concentration, and the combination with PIAD offers a strong alternative to state-of-the-art model compression and model expansion methods.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
MPCM-Net: Multi-scale network integrates partial attention convolution with Mamba for ground-based cloud image segmentation
Penghui Niu, Jiashuai She, Taotao Cai et al. (+4 more)
Ground-based cloud image
segmentation is a critical research domain for photovoltaic power
forecasting. Current deep learning approaches primarily focus on encoder-decoder architectural refinements. However, existing methodologies exhibit several limitations:(1)they rely on dilated convolutions for multi-scale context extraction, lacking the partial feature effectiveness and interoperability of inter-channel;(2)
attention-based feature enhancement implementations neglect accuracy-throughput balance; and (3)the decoder modifications fail to establish global interdependencies among hierarchical local features, limiting inference efficiency. To address these challenges, we propose MPCM-Net, a Multi-scale network that integrates Partial
attention Convolutions with Mamba architectures to enhance
segmentation accuracy and computational efficiency. Specifically, the encoder incorporates MPAC, which comprises:(1)a MPC block with ParCM and ParSM that enables global spatial interaction across multi-scale cloud formations, and (2)a MPA block combining ParAM and ParSM to extract discriminative features with reduced computational complexity. On the decoder side, a M2B is employed to mitigate contextual loss through a SSHD that maintains linear complexity while enabling deep feature aggregation across spatial and scale dimensions. As a key contribution to the community, we also introduce and release a
dataset CSRC, which is a clear-label, fine-grained
segmentation benchmark designed to overcome the critical limitations of existing public
datasets. Extensive experiments on CSRC demonstrate the superior performance of MPCM-Net over state-of-the-art methods, achieving an optimal balance between
segmentation accuracy and inference speed. The
dataset and source code will be available at https://github.com/she1110/CSRC.
Read the full abstract ▾
All authors Penghui Niu, Jiashuai She, Taotao Cai, Yajuan Zhang, Ping Zhang, Junhua Gu, Jianxin Li
Computer Vision · Cross-listed from cs.LG
Probabilistic Wildfire Susceptibility from Remote Sensing Using Random Forests and SHAP
Udaya Bhasker Cheerala, Varun Teja Chirukuri, Venkata Akhil Kumar Gummadi et al. (+2 more)
Wildfires pose a significant global threat to ecosystems worldwide, with California experiencing recurring fires due to various factors, including
climate, topographical features, vegetation patterns, and human activities. This study aims to develop a comprehensive wildfire risk map for California by applying the random forest (RF)
algorithm, augmented with Explainable Artificial Intelligence (XAI) through Shapley Additive exPlanations (SHAP), to interpret model predictions. Model performance was assessed using both spatial and temporal validation strategies. The RF model demonstrated strong predictive performance, achieving near-perfect discrimination for grasslands (AUC = 0.996) and forests (AUC = 0.997). Spatial cross-validation revealed moderate transferability, yielding ROC-AUC values of 0.6155 for forests and 0.5416 for grasslands. In contrast, temporal split validation showed enhanced generalization, especially for forests (ROC-AUC = 0.6615, PR-AUC = 0.8423). SHAP-based XAI analysis identified key ecosystem-specific drivers: soil organic carbon, tree cover, and Normalized Difference Vegetation Index (NDVI) emerged as the most influential in forests, whereas Land Surface Temperature (LST), elevation, and vegetation health indices were dominant in grasslands. District-level classification revealed that Central Valley and Northern Buttes districts had the highest concentration of high-risk grasslands, while Northern Buttes and North Coast Redwoods dominated forested high-risk areas. This RF-SHAP framework offers a robust, comprehensible, and adaptable method for assessing wildfire risks, enabling informed decisions and creating targeted strategies to mitigate dangers.
Read the full abstract ▾
All authors Udaya Bhasker Cheerala, Varun Teja Chirukuri, Venkata Akhil Kumar Gummadi, Jintu Moni Bhuyan, Praveen Damacharla
Computer Vision · Cross-listed from cs.LG
Free-Boundary Quasiconformal Maps via a Least-squares Operator in Diffeomorphism Optimization
Zhehao Xu and Lok Ming Lui
Free-boundary diffeomorphism optimization, an important and widely occurring task in
geometric modeling, computer graphics, and biological imaging, requires simultaneously determining a planar target domain and a locally bijective map with well-controlled distortion. We formulate this task through the least-squares quasiconformal (LSQC) operator and establish key structural properties of the LSQC minimizer, including well-posedness under mild conditions, invariance under similarity transformations, and resolution-independent behavior with stability under mesh refinement. We further analyze the sensitivity of the LSQC solution with respect to the Beltrami coefficient, establishing stability and differentiability properties that enable gradient-based optimization over the space of Beltrami coefficients. To make this differentiable formulation practical at scale and to facilitate the optimization process, we introduce the Spectral Beltrami Network (SBN), a multiscale mesh-spectral surrogate that approximates the LSQC solution operator in a single differentiable forward pass. This yields SBN-Opt, an optimization framework that searches over admissible Beltrami coefficients and pinning conditions to solve free-boundary diffeomorphism objectives with explicit distortion control. Extensive experiments on equiareal parameterization and inconsistent surface registration demonstrate consistent improvements over traditional numerical
algorithms.
Read the full abstract ▾
Computer Vision
CARScenes: Semantic VLM Dataset for Safe Autonomous Driving
Yuankai He and Weisong Shi
CAR-Scenes is a frame-level
dataset for autonomous driving that enables training and evaluation of
vision-
language models (VLMs) for interpretable, scene-level understanding. We annotate 5,192 images drawn from Argoverse 1, Cityscapes, KITTI, and nuScenes using a 28-key category/sub-category knowledge base covering environment, road geometry, background-vehicle behavior, ego-vehicle behavior, vulnerable road users, sensor states, and a discrete severity scale (1-10), totaling 350+ leaf attributes. Labels are produced by a GPT-4o-assisted
vision-
language pipeline with human-in-the-loop verification; we release the exact
prompts, post-processing rules, and per-field baseline model performance. CAR-Scenes also provides attribute co-occurrence graphs and JSONL records that support semantic
retrieval,
dataset triage, and risk-aware scenario mining across sources. To calibrate task difficulty, we include reproducible, non-
benchmark baselines, notably a LoRA-tuned Qwen2-VL-2B with deterministic decoding, evaluated via scalar accuracy, micro-averaged F1 for list attributes, and severity MAE/RMSE on a fixed validation split. We publicly release the annotation and analysis scripts, including graph construction and evaluation scripts, to enable explainable, data-centric workflows for future intelligent vehicles.
Dataset: https://github.com/Croquembouche/CAR-Scenes
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
DualVision ArthroNav: Investigating Opportunities to Enhance Localization and Reconstruction in Image-based Arthroscopy Navigation via External Cameras
Hongchao Shu, Lalithkumar Seenivasan, Mingxu Liu et al. (+6 more)
Arthroscopic procedures can greatly benefit from
navigation systems that enhance spatial awareness, depth perception, and field of view. However, existing optical tracking solutions impose strict workspace constraints and disrupt
surgical workflow.
Vision-based alternatives, though less invasive, often rely solely on the
monocular arthroscope camera, making them prone to drift, scale ambiguity, and sensitivity to rapid motion or occlusion. We propose DualVision ArthroNav, a multi-camera arthroscopy
navigation system that integrates an external camera rigidly mounted on the arthroscope. The external camera provides stable visual odometry and absolute localization, while the
monocular arthroscope
video enables dense scene reconstruction. By combining these complementary views, our system resolves the scale ambiguity and long-term drift inherent in
monocular SLAM and ensures robust relocalization. Experiments demonstrate that our system effectively compensates for calibration errors, achieving an average absolute trajectory error of 1.09 mm. The reconstructed scenes reach an average target registration error of 2.16 mm, with high visual fidelity (SSIM = 0.69, PSNR = 22.19). These results indicate that our system provides a practical and cost-efficient solution for arthroscopic
navigation, bridging the gap between optical tracking and purely
vision-based systems, and paving the way toward clinically deployable, fully
vision-based arthroscopic guidance.
Read the full abstract ▾
All authors Hongchao Shu, Lalithkumar Seenivasan, Mingxu Liu, Yunseo Hwang, Yu-Chun Ku, Jonathan Knopf, Alejandro Martin-Gomez, Mehran Armand, Mathias Unberath
Computer Vision
From Street to Orbit: Training-Free Cross-View Retrieval via Location Semantics and LLM Guidance
Jeongho Min, Dongyoung Kim and Jaehyup Lee
Cross-view image
retrieval, particularly street-to-satellite matching, is a critical task for applications such as autonomous
navigation, urban planning, and localization in GPS-denied environments. However, existing approaches often require supervised training on curated
datasets and rely on panoramic or
UAV-based images, which limits real-world deployment. In this paper, we present a simple yet effective cross-view image
retrieval framework that leverages a pretrained
vision encoder and a large
language model (LLM), requiring no additional training. Given a
monocular street-view image, our method extracts geographic cues through web-based image search and LLM-based location inference, generates a satellite query via geocoding API, and retrieves matching tiles using a pretrained
vision encoder (e.g., DINOv2) with PCA-based whitening feature refinement. Despite using no ground-truth supervision or finetuning, our proposed method outperforms prior learning-based approaches on the
benchmark dataset under zero-shot settings. Moreover, our pipeline enables automatic construction of semantically aligned street-to-satellite
datasets, which is offering a scalable and cost-efficient alternative to manual annotation. All source codes will be made publicly available at https://jeonghomin.github.io/street2orbit.github.io/.
Read the full abstract ▾
Computer Vision
Lumos3D: A Single-Forward Framework for Low-Light 3D Scene Restoration
Hanzhou Liu, Peng Jiang, Jia Huang et al. (+1 more)
Restoring 3D scenes with low-light conditions is challenging, and most existing methods depend on precomputed camera poses and scene-specific optimization, which greatly restricts their application to real-world scenarios. To overcome these limitations, we propose Lumos3D, a pose-free single-forward framework for 3D low-light scene restoration. First, we develop a cross-illumination
distillation scheme, where a frozen teacher network takes normal-light ground truth images as input to distill accurate
geometric information to the student model. Second, we define a Lumos loss to improve the restoration quality of the reconstructed 3D Gaussian space. Trained on a single
dataset, Lumos3D performs inference in a purely feed-forward manner, directly restoring illumination and structure from unposed, low-light multi-view images without any per-scene training or optimization. Experiments on real-world
datasets demonstrate that Lumos3D achieves competitive restoration results compared to scene-specific methods. Our codes are available at https://github.com/HanzhouLiu/Lumos3D.
Read the full abstract ▾
All authors Hanzhou Liu, Peng Jiang, Jia Huang, Mi Lu
Computer Vision
Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models
Konstantinos M. Dafnis and Dimitris N. Metaxas
Vision-
Language Models (VLMs) excel at zero-shot inference but often degrade under test-time domain shifts. For this reason, episodic test-time adaptation strategies have recently emerged as powerful techniques for adapting VLMs to a single unlabeled image. However, existing adaptation strategies, such as test-time
prompt tuning, typically require backpropagating through large encoder weights or altering core model components. In this work, we introduce Spectrum-Aware Test-Time Steering (STS), a lightweight adaptation framework that extracts a spectral subspace from the textual
embeddings to define principal semantic directions and learns to steer latent representations in a spectrum-aware manner by adapting a small number of per-sample shift parameters to minimize
entropy across augmented views. STS operates entirely at inference in the latent space, without backpropagation through or modification of the frozen encoders. Building on standard evaluation protocols, our comprehensive experiments demonstrate that STS largely surpasses or compares favorably against state-of-the-art test-time adaptation methods, while introducing only a handful of additional parameters and achieving inference speeds up to 8x faster with a 12x smaller
memory footprint than conventional test-time
prompt tuning. The code is available at https://github.com/kdafnis/STS.
Read the full abstract ▾
Computer Vision
PANDA – Patch And Distribution-Aware Augmentation for Long-Tailed Exemplar-Free Continual Learning
Siddeshwar Raghavan, Jiangpeng He and Fengqing Zhu
Exemplar-Free Continual Learning (EFCL) restricts the storage of previous task data and is highly susceptible to catastrophic forgetting. While pre-trained models (PTMs) are increasingly leveraged for EFCL, existing methods often overlook the inherent imbalance of real-world data distributions. We discovered that real-world data streams commonly exhibit dual-level imbalances,
dataset-level distributions combined with extreme or reversed skews within individual tasks, creating both intra-task and inter-task disparities that hinder effective learning and generalization. To address these challenges, we propose PANDA, a Patch-and-Distribution-Aware Augmentation framework that integrates seamlessly with existing PTM-based EFCL methods. PANDA amplifies low-frequency classes by using a CLIP encoder to identify representative regions and transplanting those into frequent-class samples within each task. Furthermore, PANDA incorporates an adaptive balancing strategy that leverages prior task distributions to smooth inter-task imbalances, reducing the overall gap between average samples across tasks and enabling fairer learning with frozen PTMs. Extensive experiments and ablation studies demonstrate PANDA's capability to work with existing PTM-based CL methods, improving accuracy and reducing catastrophic forgetting.
Read the full abstract ▾
Computer Vision
STORM: Segment, Track, and Object Re-Localization from a Single Image
Yu Deng, Teng Cao, Hikaru Shindo et al. (+3 more)
Accurate 6D pose estimation and tracking are core capabilities for physical AI systems, yet real-world deployment remains brittle and labor-intensive. Many pipelines rely on CAD models, manual masking, or per-object adaptation, and still fail under occlusion or fast motion without a principled way to recognize failure. We propose STORM, a unified framework for reference-conditioned 6D tracking that can operate from a single reference image, with minimal manual input and improved robustness. STORM combines: (i) Hierarchical Spatial Fusion
Attention (HSFA), a task-driven reference-query fusion architecture that supports both single-reference and multi-reference conditioning and can optionally use
vision-
language semantic conditioning to resolve instance ambiguities; and (ii) a BCE-trained tracking verifier whose continuous compatibility logit is used as an energy-like score to detect drift and trigger automatic re-initialization. Experiments on LM-O and YCB-
Video show that STORM improves annotation-free pose tracking accuracy over strong baselines and recovers reliably from severe occlusions and rapid viewpoint changes with minimal overhead.
Read the full abstract ▾
All authors Yu Deng, Teng Cao, Hikaru Shindo, Quentin Delfosse, Jiahong Xue, Kristian Kersting
Computer Vision
Gradient-Guided Exploration of Generative Model's Latent Space for Controlled Iris Image Augmentations
Mahsa Mitcheff, Siamul Karim Khan and Adam Czajka
Developing reliable iris recognition and presentation attack detection methods requires diverse
datasets that capture realistic variations in iris features and a wide spectrum of anomalies. Because of the rich texture of iris images, which spans a wide range of spatial frequencies, synthesizing same-identity iris images while controlling specific attributes remains challenging. In this work, we introduce a new iris image augmentation strategy by traversing a generative model's latent space toward latent codes that represent same-identity samples but with some desired iris image properties manipulated. The latent space traversal is guided by a gradient of specific geometrical, textural, or quality-related iris image features (e.g., sharpness, pupil size, iris size, or pupil-to-iris ratio) and preserves the identity represented by the image being manipulated. The proposed approach can be easily extended to manipulate any attribute for which a differentiable loss term can be formulated. Additionally, our approach can use either randomly generated images using either a pre-train GAN model or real-world iris images. We can utilize GAN inversion to project any given iris image into the latent space and obtain its corresponding latent code.
Read the full abstract ▾
Computer Vision
Feature Quality and Adaptability of Medical Foundation Models: A Comparative Evaluation for Radiographic Classification and Segmentation
Frank Li, Theo Dapamede, Mohammadreza Chavoshi et al. (+12 more)
Foundation models (FMs) promise to generalize medical imaging, but their effectiveness varies. It remains unclear how pre-training domain (medical vs. general), paradigm (e.g., text-guided), and architecture influence
embedding quality, hindering the selection of optimal encoders for specific radiology tasks. To address this, we evaluate
vision encoders from eight medical and general-domain FMs for chest X-ray analysis. We
benchmark classification (pneumothorax, cardiomegaly) and
segmentation (pneumothorax, cardiac boundary) using linear probing and fine-tuning. Our results show that domain-specific pre-training provides a significant advantage; medical FMs consistently outperformed general-domain models in linear probing, establishing superior initial feature quality. However, feature utility is highly task-dependent. Pre-trained
embeddings were strong for global classification and segmenting salient anatomy (e.g., heart). In contrast, for segmenting complex, subtle pathologies (e.g., pneumothorax), all FMs performed poorly without significant fine-tuning, revealing a critical gap in localizing subtle disease. Subgroup analysis showed FMs use confounding shortcuts (e.g., chest tubes for pneumothorax) for classification, a strategy that fails for precise
segmentation. We also found that expensive text-image
alignment is not a prerequisite; image-only (RAD-DINO) and label-supervised (Ark+) FMs were among top performers. Notably, a supervised, end-to-end baseline remained highly competitive, matching or exceeding the best FMs on
segmentation tasks. These findings show that while medical pre-training is beneficial, architectural choices (e.g., multi-scale) are critical, and pre-trained features are not universally effective, especially for complex localization tasks where supervised models remain a strong alternative.
Read the full abstract ▾
All authors Frank Li, Theo Dapamede, Mohammadreza Chavoshi, Young Seok Jeon, Bardia Khosravi, Abdulhameed Dere, Beatrice Brown-Mulry, Rohan Satya Isaac, Aawez Mansuri, Chiratidzo Sanyika, Janice Newsome, Saptarshi Purkayastha, Imon Banerjee, Hari Trivedi, Judy Gichoya
Computer Vision
Soiling detection for Advanced Driver Assistance Systems
Filip Beránek, Václav Diviš and Ivan Gruber
Soiling detection for automotive cameras is a crucial part of advanced driver assistance systems to make them more robust to external conditions like weather,
dust, etc. In this paper, we regard the soiling detection as a semantic
segmentation problem. We provide a comprehensive comparison of popular
segmentation methods and show their superiority in performance while comparing them to tile-level classification approaches. Moreover, we present an extensive analysis of the Woodscape
dataset showing that the original
dataset contains a data-leakage and imprecise annotations. To address these problems, we create a new data subset, which, despite being much smaller, provides enough information for the
segmentation method to reach comparable results in a much shorter time. All our codes and
dataset splits are available at https://github.com/filipberanek/woodscape_revision.
Read the full abstract ▾
Computer Vision
Social LSTM with Dynamic Occupancy Modeling for Realistic Pedestrian Trajectory Prediction
Ahmed Alia, Mohcine Chraibi and Armin Seyfried
In dynamic and crowded environments, realistic pedestrian trajectory prediction remains a challenging task due to the complex nature of human motion and the mutual influences among individuals. Deep learning models have recently achieved promising results by implicitly learning such patterns from 2D trajectory data. However, most approaches treat pedestrians as point entities, ignoring the physical space that each person occupies. To address these limitations, this paper proposes a novel deep learning model that enhances the Social LSTM with a new Dynamic Occupied Space loss function. This loss function guides Social LSTM in learning to avoid realistic collisions without increasing displacement error across different crowd densities, ranging from low to high, in both homogeneous and heterogeneous density settings. Such a function achieves this by combining the average displacement error with a new collision penalty that is sensitive to scene density and individual spatial occupancy. For efficient training and evaluation, five
datasets were generated from real pedestrian trajectories recorded during the Festival of Lights in Lyon 2022. Four
datasets represent homogeneous crowd conditions – low, medium, high, and very high density – while the fifth corresponds to a heterogeneous density distribution. The experimental findings indicate that the proposed model not only lowers collision rates but also enhances displacement prediction accuracy in each
dataset. Specifically, the model achieves up to a 31% reduction in the collision rate and reduces the average displacement error and the final displacement error by 5% and 6%, respectively, on average across all
datasets compared to the baseline. Moreover, the proposed model consistently outperforms several state-of-the-art deep learning models across most test sets.
Read the full abstract ▾
Computer Vision
PALMS+: Modular Image-Based Floor Plan Localization Leveraging Depth Foundation Model
Yunqian Cheng, Benjamin Princen and Roberto Manduchi
Indoor localization in GPS-denied environments is crucial for applications like emergency response and assistive
navigation.
Vision-based methods such as PALMS enable infrastructure-free localization using only a floor plan and a stationary scan, but are limited by the short range of smartphone
LiDAR and ambiguity in indoor layouts. We propose PALMS$+$, a modular, image-based system that addresses these challenges by reconstructing scale-aligned 3D point clouds from posed RGB images using a foundation
monocular depth estimation model (Depth Pro), followed by
geometric layout matching via convolution with the floor plan. PALMS$+$ outputs a posterior over the location and orientation, usable for direct or sequential localization. Evaluated on the Structured3D and a custom campus
dataset consisting of 80 observations across four large campus buildings, PALMS$+$ outperforms PALMS and F3Loc in stationary localization accuracy – without requiring any training. Furthermore, when integrated with a particle filter for sequential localization on 33 real-world trajectories, PALMS$+$ achieved lower localization errors compared to other methods, demonstrating robustness for camera-free tracking and its potential for infrastructure-free applications. Code and data are available at https://github.com/Head-inthe-Cloud/PALMS-Plane-based-Accessible-Indoor-Localization-Using-Mobile-Smartphones
Read the full abstract ▾
Computer Vision
Density Estimation and Crowd Counting
Balachandra Devarangadi Sunil, Rakshith Venkatesh and Shantanu Todmal
This study enhances a crowd density estimation
algorithm originally designed for image-based analysis by adapting it for
video-based scenarios. The proposed method integrates a denoising probabilistic model that utilizes
diffusion processes to generate high-quality crowd density maps. To improve accuracy, narrow Gaussian kernels are employed, and multiple density map outputs are generated. A regression branch is incorporated into the model for precise feature extraction, while a consolidation mechanism combines these maps based on similarity scores to produce a robust final result. An event-driven sampling technique, utilizing the Farneback optical flow
algorithm, is introduced to selectively capture frames showing significant crowd movements, reducing computational load and storage by focusing on critical crowd dynamics. Through qualitative and quantitative evaluations, including overlay plots and Mean Absolute Error (MAE), the model demonstrates its ability to effectively capture crowd dynamics in both dense and sparse settings. The efficiency of the sampling method is further assessed, showcasing its capability to decrease frame counts while maintaining essential crowd events. By addressing the temporal challenges unique to
video analysis, this work offers a scalable and efficient framework for real-time crowd monitoring in applications such as public safety, disaster response, and event management.
Read the full abstract ▾
Computer Vision
SliderEdit: Continuous Image Editing with Fine-Grained Instruction Control
Arman Zarei, Samyadeep Basu, Mobina Pournemat et al. (+3 more)
Instruction-based image editing models have recently achieved impressive performance, enabling complex edits to an input image from a multi-instruction
prompt. However, these models apply each instruction in the
prompt with a fixed strength, limiting the user's ability to precisely and continuously control the intensity of individual edits. We introduce SliderEdit, a framework for continuous image editing with fine-grained, interpretable instruction control. Given a multi-part edit instruction, SliderEdit disentangles the individual instructions and exposes each as a globally trained slider, allowing smooth adjustment of its strength. Unlike prior works that introduced slider-based attribute controls in text-to-image generation, typically requiring separate training or fine-tuning for each attribute or concept, our method learns a single set of low-rank adaptation
matrices that generalize across diverse edits, attributes, and compositional instructions. This enables continuous interpolation along individual edit dimensions while preserving both spatial locality and global semantic consistency. We apply SliderEdit to state-of-the-art image editing models, including FLUX-Kontext and Qwen-Image-Edit, and observe substantial improvements in edit controllability, visual consistency, and user steerability. To the best of our knowledge, we are the first to explore and propose a framework for continuous, fine-grained instruction control in instruction-based image editing models. Our results pave the way for interactive, instruction-driven image manipulation with continuous and compositional control.
Read the full abstract ▾
All authors Arman Zarei, Samyadeep Basu, Mobina Pournemat, Sayan Nag, Ryan Rossi, Soheil Feizi
Computer Vision
Classifying Phonotrauma Severity from Vocal Fold Images with Soft Ordinal Regression
Katie Matton, Purvaja Balaji, Hamzeh Ghasemzadeh et al. (+7 more)
Phonotrauma refers to vocal fold tissue damage resulting from exposure to forces during voicing. It occurs on a continuum from mild to severe, and treatment options can vary based on severity. Assessment of severity involves a clinician's expert judgment, which is costly and can vary widely in reliability. In this work, we present the first method for automatically classifying phonotrauma severity from vocal fold images. To account for the ordinal nature of the labels, we adopt a widely used ordinal regression framework. To account for label uncertainty, we propose a novel modification to ordinal regression loss functions that enables them to operate on soft labels reflecting annotator rating distributions. Our proposed soft ordinal regression method achieves predictive performance approaching that of clinical experts, while producing well-calibrated uncertainty estimates. By providing an automated tool for phonotrauma severity assessment, our work can enable large-scale studies of phonotrauma, ultimately leading to improved clinical understanding and patient care.
Read the full abstract ▾
All authors Katie Matton, Purvaja Balaji, Hamzeh Ghasemzadeh, Jameson C. Cooper, Daryush D. Mehta, Jarrad H. Van Stan, Robert E. Hillman, Rosalind Picard, John Guttag, S. Mazdak Abulnaga
Computer Vision
PriVi: Towards A General-Purpose Video Model For Primate Behavior In The Wild
Felix B. Mueller, Jan F. Meier, Timo Lueddecke et al. (+20 more)
Non-human primates are our closest living relatives, and analyzing their behavior is central to research in cognition,
evolution, and conservation. Computer
vision could greatly aid this research, but existing methods often rely on human-centric pretrained models and focus on single
datasets, which limits generalization. We address this limitation by shifting from a model-centric to a data-centric approach and introduce PriVi, a large-scale primate-centric
video pretraining
dataset. PriVi contains 424 hours of curated
video, combining 174 hours from behavioral research across 11 settings with 250 hours of diverse web-sourced footage, assembled through a scalable data curation pipeline. We continue pretraining V-JEPA, a large-scale
video model, on PriVi to learn primate-specific representations and evaluate it using a lightweight frozen classifier. Across four
benchmark datasets, ChimpACT, PanAf500, BaboonLand, and ChimpBehave, our approach consistently outperforms prior work, including fully finetuned baselines, and scales favorably with fewer labels. These results demonstrate for the first time that domain-level pretraining, where pretraining is conducted on similar data but not the target
dataset itself, works for
video models. Our primate-centric pretraining substantially improves data efficiency and generalization, making it a promising approach for low-label applications.
Dataset, code, and models are available: https://privi.eckerlab.org
Read the full abstract ▾
All authors Felix B. Mueller, Jan F. Meier, Timo Lueddecke, Richard Vogg, Roger L. Freixanet, Valentin Hassler, Tiffany Bosshard, Elif Karakoc, William J. O'Hearn, Sofia M. Pereira, Sandro Sehner, Kaja Wierucka, Judith Burkart, Claudia Fichtel, Julia Fischer, Alexander Gail, Catherine Hobaiter, Julia Ostner, Liran Samuni, Oliver Schülke, Neda Shahidi, Erin G. Wessling, Alexander S. Ecker
Computer Vision
MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and Generation
Ye Tian, Ling Yang, Jiongfan Yang et al. (+10 more)
While thinking-aware generation aims to improve performance on complex tasks, we identify a critical failure mode where existing sequential, autoregressive approaches can paradoxically degrade performance due to error propagation. To systematically analyze this issue, we propose ParaBench, a new
benchmark designed to evaluate both text and image output modalities. Our analysis using ParaBench reveals that this performance degradation is strongly correlated with poor
alignment between the generated
reasoning and the final image. To resolve this, we propose a parallel
multimodal diffusion framework, MMaDA-Parallel, that enables continuous, bidirectional interaction between text and images throughout the entire denoising trajectory. MMaDA-Parallel is trained with supervised finetuning and then further optimized by Parallel Reinforcement Learning (ParaRL), a novel strategy that applies semantic rewards along the trajectory to enforce cross-modal consistency. Experiments validate that our model significantly improves cross-modal
alignment and semantic consistency, achieving a 6.9% improvement in Output
Alignment on ParaBench compared to the state-of-the-art model, Bagel, establishing a more robust paradigm for thinking-aware image synthesis. Our code is open-sourced at https://github.com/tyfeld/MMaDA-Parallel
Read the full abstract ▾
All authors Ye Tian, Ling Yang, Jiongfan Yang, Anran Wang, Yu Tian, Jiani Zheng, Haochen Wang, Zhiyang Teng, Zhuochen Wang, Yinjie Wang, Yunhai Tong, Mengdi Wang, Xiangtai Li
Computer Vision
FedeCouple: Fine-Grained Balancing of Global-Generalization and Local-Adaptability in Federated Learning
Ming Yang, Dongrun Li, Xin Wang et al. (+5 more)
In
privacy-preserving mobile network transmission scenarios with heterogeneous client data, personalized
federated learning methods that decouple feature extractors and classifiers have demonstrated notable advantages in enhancing learning capability. However, many existing approaches primarily focus on feature space consistency and classification personalization during local training, often neglecting the local adaptability of the extractor and the global generalization of the classifier. This oversight results in insufficient coordination and weak coupling between the components, ultimately degrading the overall model performance. To address this challenge, we propose FedeCouple, a
federated learning method that balances global generalization and local adaptability at a fine-grained level. Our approach jointly learns global and local feature representations while employing dynamic knowledge
distillation to enhance the generalization of personalized classifiers. We further introduce anchors to refine the feature space; their strict locality and non-transmission inherently preserve
privacy and reduce communication overhead. Furthermore, we provide a theoretical analysis proving that FedeCouple converges for nonconvex objectives, with iterates approaching a stationary point as the number of communication rounds increases. Extensive experiments conducted on five image-classification
datasets demonstrate that FedeCouple consistently outperforms nine baseline methods in effectiveness, stability, scalability, and security. Notably, in experiments evaluating effectiveness, FedeCouple surpasses the best baseline by a significant margin of 4.3%.
Read the full abstract ▾
All authors Ming Yang, Dongrun Li, Xin Wang, Feng Li, Lisheng Fan, Chunxiao Wang, Xiaoming Wu, Peng Cheng
Computer Vision · Cross-listed from cs.RO
IFG: Internet-Scale Guidance for Functional Grasping Generation
Ray Muxin Liu, Mingxuan Li, Kenneth Shaw et al. (+1 more)
Large
Vision Models trained on internet-scale data have demonstrated strong capabilities in segmenting and semantically understanding object parts, even in cluttered, crowded scenes. However, while these models can direct a
robot toward the general region of an object, they lack the
geometric understanding required to precisely control dexterous robotic hands for 3D grasping. To overcome this, our key insight is to leverage simulation with a force-closure grasping generation pipeline that understands local geometries of the hand and object in the scene. Because this pipeline is slow and requires ground-truth observations, the resulting data is distilled into a
diffusion model that operates in real-time on camera point clouds. By combining the global semantic understanding of internet-scale models with the
geometric precision of a simulation-based locally-aware force-closure, \our achieves high-performance semantic grasping without any manually collected training data. For visualizations of this please visit our website at https://ifgrasping.github.io/
Read the full abstract ▾
All authors Ray Muxin Liu, Mingxuan Li, Kenneth Shaw, Deepak Pathak