Computer Vision
Beyond the Foreground: FOV-Aware Polyp Image Synthesis via Lesion-Guided Adaptive Mucosal Context Propagation
Tong Wang, Yuting He, Bin Ren et al. (+2 more)
Synthetic image and mask pairs can alleviate scarce colonoscopy annotations, but realistic synthesis requires preserving the supplied lesion while generating compatible mucosa. Existing foreground-guided methods treat all non-foreground pixels as background and rely mainly on local integration. Directly applying them to colonoscopy causes two problems: non-mucosal black regions contaminate generated tissue, and local
reasoning produces inconsistent mucosal texture and illumination. We propose LAMP, the first foreground-guided framework for polyp image synthesis based on lesion-guided adaptive mucosal context propagation. LAMP explicitly separates the lesion, valid mucosa, and camera exterior using a field-of-view (FOV) mask. Lesion-to-Mucosa cross-
attention extracts lesion appearance conditions for valid-mucosa locations, while FOV-constrained multidirectional
Vision Receptance Weighted Key Value propagates them over legal tissue support. An adaptive gate then controls their residual fusion into the
diffusion U-Net. Extensive experiments on five polyp
datasets demonstrate that LAMP substantially outperforms existing methods in overall generation quality and consistently improves five downstream
segmentation models. Our code will be released at https://github.com/wangtong627/LAMP.
Read the full abstract ▾
All authors Tong Wang, Yuting He, Bin Ren, Yutong Xie, Guanyu Yang
Computer Vision
Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement
Yitong Xing, Yuhao Cheng, Yanping Li et al. (+1 more)
Detection
Transformers (DETRs) achieve strong performance in object detection but remain challenging to deploy on edge devices due to their high computational cost. Existing DETR
distillation methods mainly focus on aligning
distillation points, while largely overlooking the quality of the teacher's supervision itself. We observe that due to stage-wise non-monotonic prediction behavior in DETRs, well-localized or correctly classified predictions from earlier stages may degrade in later ones, and some negative predictions become increasingly overconfident. As a result, relying solely on the current stage's predictions yields inaccurate and inconsistent supervision. To address this issue, we propose Teacher Prediction Refinement
Distillation (TPRD), a plug-and-play module that refines teacher predictions before
distillation by exploiting stage-wise prediction information. TPRD improves supervision quality through Positive Prediction Correction (PPC), which corrects degraded positive predictions by restoring more accurate ones from earlier stages, ensuring reliable localization and classification signals, and Negative Prediction Suppression (NPS) suppresses the influence of overconfident negatives, preventing them from providing misleading supervision to the student. To preserve informative dark knowledge, we further introduce Maximum Dark Knowledge Preservation (MDKP), which selectively refines target-class logits while retaining non-target relations. Extensive experiments on MS COCO and PASCAL VOC demonstrate the effectiveness and robustness of the proposed method. Our code is available at https://github.com/xingyitong1/TPRD.
Read the full abstract ▾
All authors Yitong Xing, Yuhao Cheng, Yanping Li, Yichao Yan
Computer Vision · Cross-listed from cs.RO
LapaTrack-3D: 6 DoF pre-operative shape tracking for laparoscopic surgery
Jingwei Song, Javid Hussain Jakir, Ray Zhang et al. (+4 more)
This work proposes a real-time 6 Degree-of-Freedom (6 DoF) tracking
algorithm for
monocular laparoscopic surgery. It provides
alignment between intra-operative
video and pre-operative data (e.g., CT). The 6 DoF tracking offers a solution for accurately locating the internal anatomy of the target organ despite the lack of tactile feedback and transparency. The ORB-SLAM2 framework is adopted and modified for prior-based 3D tracking with four major modifications. First, the primitive 3D shape is used for fast initialization of the ORB-SLAM2
monocular mode. Second, a pseudo-
segmentation strategy is employed to separate the target organ from the background for tracking. Third, the 3D shape is incorporated as a
geometric prior in its pose graph optimization. Fourth, the Multi-Scale Retinex with Chromaticity Preservation (MSRCP)
algorithm is leveraged and modified for image enhancement in challenging illumination scenarios. In-vivo and ex-vivo experiments validate that LapaTrack-3D provides robust 3D tracking and effectively handles typical challenges such as poor illumination, fast motion, out-of-field-of-view scenarios, partial visibility, and “organ-background” relative motion. LapaTrack-3D achieves a processing rate of 13 Hz for 1280*720 pixel
video.
Read the full abstract ▾
All authors Jingwei Song, Javid Hussain Jakir, Ray Zhang, Wenwei Zhang, Hao Zhou, Xiaomeng Xian, Maani Ghaffari
Computer Vision
DirtyMoCap: Robust Motion Capture from Unconstrained Markers
Long Wang, Shuting Zhao, Shen Yan et al. (+5 more)
Optical motion capture delivers high-fidelity human motion, but its reliance on strict marker layouts and clean trajectories severely limits its real-world applicability. In practice, tracking systems frequently output unconstrained markers: sparse, noisy, and unordered point clouds with unknown or varying configurations. To bridge the gap between corrupted raw markers and parametric human models, we introduce DirtyMoCap, a robust, marker-layout-free framework. Our core insight is to map unordered marker observations to a fixed set of "proxy anchors" comprising skeletal joints and body surface points, which serve as a stable intermediate representation. We first initialize and track these anchors over long sequences using a recurrent sliding-window architecture. Then, a custom differentiable Gauss-Newton solver fits the SMPL-H model to the tracked anchors to recover full-body pose, translation, and shape. By explicitly deriving
geometric residuals, our solver learns adaptive observation confidence, smoothness, and prior weights end-to-end, adapting dynamically to the reliability of the input data. Extensive experiments on diverse, noisy marker configurations demonstrate that DirtyMoCap successfully generalizes across arbitrary layouts using only a single trained model. It consistently outperforms state-of-the-art configuration-specific baselines in both joint and vertex reconstruction accuracy, while our custom CUDA solver achieves up to a 100x speedup over standard PyTorch implementations. We further apply DirtyMoCap to heterogeneous raw optical MoCap recordings of traditional Chinese martial arts, yielding a Kung Fu motion
dataset of temporally coherent SMPL-H reconstructions. Code and data are available at https://wanglongzju.github.io/DirtyMoCap-Project-Page.
Read the full abstract ▾
All authors Long Wang, Shuting Zhao, Shen Yan, Siyuan Yu, Xiaoben Li, Zeyu Cai, Yumeng Hou, Yuliang Xiu
Computer Vision
CitySTAR: Structured and Topology-Aware Reasoning for Open-Vocabulary Urban 3D Grounding
Shuai Zhang, Hongye Hou, Qinghe Liu et al. (+5 more)
3D grounding aims to localize target entities in complex scenes from natural
language and plays a fundamental role in
embodied perception and spatial
reasoning. However, existing approaches mostly rely on feature similarity or direct matching, making it difficult to connect natural-
language intent with the implicit semantic and
geometric structures hidden in billion-scale urban point clouds. We reformulate city-scale 3D grounding as structured constraint
reasoning, where description semantics are organized into computable cross-modal constraints over open-vocabulary 3D entities, attributes, and spatial relations. We present CitySTAR, a training-free framework for
reasoning-driven urban 3D grounding. CitySTAR lifts raw billion-scale urban point clouds into a query-ready scene graph of open-vocabulary 3D instances, with CodeLLM-driven tools supplying
multimodal evidence for node attributes and 3D spatial relations. It then models target-context
topology with paired hypergraphs and performs bidirectional
topology verification for structural disambiguation. Finally, a Reflective Cross-modal Grounding module integrates
topology consistency and candidate-centered 2D visual evidence to make decisions over a metric-aware 3D context graph. To further support this setting, we introduce CitySTAR-3D, an enhanced
benchmark that improves semantic coverage, instance completeness, bounding-box fidelity, and spatial-relation complexity in city-scale 3D grounding. Extensive experiments show that CitySTAR consistently improves open-world urban 3D grounding while maintaining strong interpretability and generalization.
Read the full abstract ▾
All authors Shuai Zhang, Hongye Hou, Qinghe Liu, Zhuoxiao Li, Dongli Wu, Jing Ou, Yuan Liu, Wufan Zhao
Computer Vision
GS-PI: An Optimization-Decoupled Appearance Decomposition Approach for Generating PBR Gaussian Assets
Jieting Xu, Rengan Xie, Zijian Huang et al. (+3 more)
Gaussian Splatting (GS) excels at novel-view synthesis but encodes baked-in radiance, tightly entangling illumination with geometry and preventing seamless integration into physically based rendering (PBR) pipelines. Existing inverse-rendering methods attempt to disentangle materials via joint optimization, but often suffer from competing objectives that cause severe ambiguities and residual lighting artifacts. To overcome this, we present GS-PI, a novel optimization-decoupled framework that casts PBR material generation as a geometry-conditioned
diffusion process on 3D point clouds. By operating directly in the 3D domain, our method inherently guarantees multi-view consistency, sidestepping the severe pixel correspondence issues that challenge 2D
diffusion approaches. We introduce a multi-scale cross-view conditioning mechanism that integrates three complementary components: a global semantic prior, source-anchored photometric cues, and an absolute spatial learned view-direction conditioning signal. This design efficiently compresses complex multi-view evidence, mitigating cross-view projection misalignment and successfully preventing specular highlights from baking into intrinsic colors. By extracting a point cloud from a pre-trained Gaussian model, predicting PBR attributes via conditional
diffusion, and distilling them back through differentiable rasterisation, we yield a fully relightable PBR-GS asset. GS-PI outperforms recent inverse-rendering baselines while replacing per-scene joint illumination/BRDF optimization with a learned
diffusion pass followed by a short target-driven
distillation, without requiring proxy meshes.
Read the full abstract ▾
All authors Jieting Xu, Rengan Xie, Zijian Huang, Zehui Jin, Rui Wang, Yuchi Huo
Computer Vision
BinoGen: Scaling egocentric binocular data for embodied visual perception and learning
Chunpeng Li and Ya-tang Li
Embodied visual perception relies on temporally coherent visual experience accumulated through continuous engagement with the environment. However, collecting large-scale
egocentric binocular observations together with dense annotations remains costly and difficult. Moreover, visual experience is shaped not only by the environment but also by the embodiment of the observer, including viewing height, field of view, binocular geometry, and motion through the scene. To address these challenges, we present BinoGen, an automated framework for generating large-scale, embodiment-aware
egocentric binocular visual experiences in indoor environments. BinoGen jointly models environmental and observer variation through generative scene synthesis, probabilistic object instantiation, appearance randomization,
stochastic trajectory generation, and configurable binocular camera setups. The framework produces synchronized binocular
videos together with dense
multimodal supervision, including depth maps, optical flow, surface normals, semantic maps, object coordinates, and camera poses. Using BinoGen, we construct a
dataset comprising more than 20 million annotated images for supervised learning. We demonstrate two complementary utilities of BinoGen. First, incorporating BinoGen data consistently improves real-world visual perception, including depth estimation, object detection, and
video object tracking. Second, paired human-inspired and mouse-inspired observations from the same environments enable controlled investigation of how observer embodiment affects perceptual learning. Embodiment-specific adaptation substantially improves performance, while joint training enables a single model to perform competitively across both embodiments. Together, these results demonstrate that large-scale, controllable visual experience can improve
embodied perception...
Read the full abstract ▾
Computer Vision
SlugTrails: An Egocentric Benchmark for Floor Plan Localization in Large Buildings
Yunqian Cheng and Roberto Manduchi
Floor-plan-based indoor visual localization enables infrastructure-free positioning, but most methods are developed and evaluated in small residential environments unlike the large public buildings of real deployment. We introduce SlugTrails, a floor plan localization
benchmark for large indoor spaces under realistic
egocentric sensing: $30$ Hz Aria glasses recordings across three campus buildings and six floors ($22089$ m$^2$ of floor plan outline), CAD-derived floor plans with semantic classes and circulation space masks, and trajectories aligned into the floor plan frame using
laser-surveyed anchors. One protocol covers three practical ways of gathering geometry under a limited field of view – a single walking frame, a stationary multi-view sweep, and a walking stream with odometry – so methods designed for different regimes are compared on the same buildings and ground truth. Evaluating five representative
geometric and learned systems under their native sensing configurations, we find that stock checkpoints (official released weights) are near zero on SlugTrails (at most $0.004$ R@1m30$^{\circ}$ on walking single frames), while fine-tuning on SlugTrails improves every trainable family on all three tasks (e.g., F$^3$Loc $0.0 \rightarrow 0.141$ single-frame and $0.03 \rightarrow 0.66$ sequential), with gains compounding as observations accumulate. The same fine-tuned weights also improve cross-
dataset generalization on LaMAR with no LaMAR training (sequential R@1m $0.048 \rightarrow 0.143$ for F$^3$Loc and $0.063 \rightarrow 0.127$ for UnLoc), whereas train-from-scratch on SlugTrails alone stays far below fine-tuning from stock weights – evidence that floor plan localization is currently limited by indoor data rather than by architecture. We release the
dataset, protocols, and tools at https://github.com/Head-inthe-Cloud/SlugTrails.
Read the full abstract ▾
Computer Vision
BINDER: A Latent Variable Model for Probabilistic Medical Image Registration
Stefano Cerri, Amirhossein Hassankhani, Yaël Balbastre et al. (+1 more)
We propose a new probabilistic model for general-purpose medical image registration that builds upon the mutual information registration criterion. It centers around a spatial interpolation technique that assumes latent voxel-wise correspondences between the images being registered. By exploiting these latent variables, we derive dedicated optimization and MCMC sampling techniques that only involve closed-form iterative updates. When applied to nonlinear registration, an efficient demons-like optimization
algorithm is obtained that shows robust out-of-the-box performance across a variety of monomodal and
multimodal registration tasks. We also demonstrate a corresponding sampler that can quantify, for the first time, uncertainty in
multimodal registration scenarios with very high-dimensional 3D deformations. Our code, which we call BINDER (
Bayesian INference for DEformable Registration), is freely available at https://github.com/ste93ste/BINDER.
Read the full abstract ▾
All authors Stefano Cerri, Amirhossein Hassankhani, Yaël Balbastre, Koen Van Leemput
Computer Vision
Ruchao Bao, Wenzheng Wu, Chucheng Xiang et al. (+5 more)
3D assembly is fundamental to modern manufacturing and digital content creation. In this paper, we present PART, a unified
transformer-based framework for 3D part
retrieval and assembly: given a target shape and a part library, PART automatically selects the appropriate parts and predicts their 6-DoF poses to reconstruct the target. While prior work has achieved impressive progress on assembling a pre-defined set of parts, this more practical
retrieval-based setting remains largely unexplored. The task faces three key challenges: (i) a combinatorially explosive search space that grows exponentially with library size; (ii) variable-length outputs, as different targets require different numbers of parts; and (iii) continuous 6-DoF pose estimation for part assembly. To address these, we formulate
retrieval and assembly as a set prediction problem and design a novel
transformer-based framework that retrieves parts and regresses their poses with variable-length output. Additionally, we exploit the duality between part pose estimation and target
segmentation through joint training and a novel
segmentation-enhanced optimization module. Finally, We curate a large-scale
dataset of 80K+ shapes, and the results show that PART generalizes to scene layouts, image targets, and real-world scans. Project Page: https://iambrc.github.io/PART-project-page/.
Read the full abstract ▾
All authors Ruchao Bao, Wenzheng Wu, Chucheng Xiang, Zhongyuan Liu, Yuan Liu, Jinxin Dong, Ligang Liu, Ziqi Wang
Computer Vision
Socialized UAV Cross-Task Learning: Towards Cross-Granularity Collaboration through Hierarchical Interaction
Xinjie Yao, Ruipu Zhao, Yunqi Zhu et al. (+6 more)
Joint learning across heterogeneous tasks is often treated as task coupling through feature sharing,
distillation, or auxiliary supervision. However, in cross-task learning, mismatched representational and supervisory granularities make such coupling prone to interference, teacher bias, or unidirectional collapse. We argue that cross-granularity learning is fundamentally a problem of hierarchical interaction regulation rather than simple task coupling. This issue is particularly evident in
UAV perception, where visual shifts and detection–
segmentation objectives naturally form coarse- and fine-grained knowledge sources. To systematically study this problem, we introduce CrossUAV, a
UAV benchmark for joint object detection and instance
segmentation that provides a unified evaluation platform for cross-granularity task collaboration. To address these challenges, we propose Cross-Granularity Socialized Collaboration (CGSC), a progressive and adaptive framework that regulates when, where, and how tasks exchange information across network hierarchies. CGSC progressively activates cross-task interactions and adaptively adjusts the strength according to task contribution, suppressing harmful interference while exploiting complementary coarse- and fine-grained structures. Extensive experiments demonstrate consistent improvements on both tasks, validating hierarchical dynamic interaction as an effective mechanism for cross-granularity collaboration.
Read the full abstract ▾
All authors Xinjie Yao, Ruipu Zhao, Yunqi Zhu, Zhihe Fan, Zhoupeng Guo, Weihao Li, Zhen Wang, Qilong Wang, Pengfei Zhu
Computer Vision · Cross-listed from cs.RO
Feeling Terrain Before Crossing: World Models for Off-Road Navigation
E-In Son, Dong-Wook Kim, Ji-Hoon Hwang et al. (+4 more)
Navigation world models plan by foresight, predicting the future that each candidate action sequence produces and selecting the best, rather than mapping observations to actions directly. Unlike urban settings where a predicted scene is a sufficient proxy, off-road
navigation hinges on the
robot–terrain interaction, so the prediction must cover not only what the camera will see but what the
robot will feel. However, existing scene-focused models do not predict how much the
robot will slip, tilt or shake along a planned trajectory. Proprioception captures these dynamics directly and, when used as input, improves the prediction of the physical future. We present Feel-WM, the first off-road
navigation world model that conditions on proprioception and predicts what the
robot will feel alongside what the camera will see. The physical future takes the form of a future proprioceptive state and a failure risk, both learned from the
robot's own experience without human labels. The planner rolls out the physical future alongside the scene and weighs the predicted failure risk against goal similarity in a separable score. Experiments on real off-road data and in simulation demonstrate that Feel-WM outperforms visual-only
navigation world models in open-loop planning and closed-loop rough-terrain
navigation across wheeled and legged platforms. Deployed on a Husky on mountain trails, Feel-WM plans onboard, predicts rough ground ahead and steers around it, completing courses that an end-to-end policy fails.
Read the full abstract ▾
All authors E-In Son, Dong-Wook Kim, Ji-Hoon Hwang, Kangsun Lee, Jisung Bae, Jung-Taak Kim, Seung-Woo Seo
Computer Vision
PACE: Precise AI Cinematic Expression
Bing Duan, Qiang Guo, Linpu Li et al. (+6 more)
Between a screenplay and a film sits a planning problem that is spatial first: who stands where, and what a camera sees from where it stands. An image
diffusion model asked for a shot in free text settles that plan by its own defaults. We present PACE (Precise AI Cinematic Expression), a typed representation for the plan: the screenplay evidence, the characters, props and locations it needs, where each subject stands, and what the camera does. A value is written once at the level it belongs to (script, scene, shot or panel) and inherited below it. A compiler turns the result into both the
prompt sent to the
diffusion model and a 3D scene built in metres, and a camera solver places the camera so that the declared framing is the framing built. Where a declared value becomes geometry, PACE measures, field by field, how far the compiled camera and the staged render sit from the declaration, rather than asking a model to judge. On the 11-scene Automatic Drive screenplay, every staged single-subject panel places its subject within 1.2% of frame width of its declared position; with two or three subjects one camera pose cannot satisfy every position, and the residual is reported rather than absorbed. On 204 external director-storyboard shots, delivered head height is 1.906 times the staged target from the director's words, 1.733 from the compiled
prompt, and 0.955 with the greybox control; the condition that holds framing best draws the described action least. Declaring the pose on 30 shots raises the action drawn from 58.9% to 74.4% without moving the framing. Transitions, fitted motion and human review of the generated panels remain open. Code: https://github.com/StudioPiLabs/pace-core
Read the full abstract ▾
All authors Bing Duan, Qiang Guo, Linpu Li, Zhijian Mao, Min Zhu, Zhirui Ren, Yiwei Yan, Xi Chu, Xiaoding Li
Computer Vision
KoUniTalk: A Lightweight Articulation-Centered Korean-English 3D Talking Face Benchmark
Hyunjung Chung and Unsang Park
High-quality 3D talking face
datasets remain largely English-centric, and Korean 3D facial motion data are difficult to combine with standard English
benchmarks because of differences in mesh
topology, spatial scale, coordinate system, and temporal sampling. We present KoUniTalk, a lightweight articulation-centered Korean-English 3D talking face
benchmark that retargets VOCASET and the released Korean
speech-based 3D talking face data to a shared mesh
topology using deformation transfer. Rather than proposing a new deformation-transfer
algorithm or a full-head identity-preserving avatar
dataset, KoUniTalk provides an identity-neutral canonical output space for controlled
speech-driven facial articulation training and evaluation across English and Korean. The unified template contains 1,176 vertices and focuses on the mouth and adjacent lower- and mid-face regions, reducing the output dimensionality from 15,069 and 72,147 dimensions to 3,528 dimensions, corresponding to 4.27-fold and 20.45-fold reductions compared with VOCASET/FLAME and the original Korean mesh, respectively. To examine whether retargeting preserves
speech-relevant motion, we evaluate semantic mouth-landmark trajectories, including mouth opening, mouth width, aperture ratio, and mouth-opening dynamics. Since the official test set of the Korean
dataset is not publicly released, we additionally define a subject-disjoint Korean
benchmark split. The processed matched
benchmark contains 22 speakers, 4,978 sequences, and 642,781 frames, enabling Korean-English cross-
dataset evaluation of
speech-driven 3D facial animation models in a single compact articulation-template space. Source-reported inventory counts are listed separately from these processed counts.
Read the full abstract ▾
Computer Vision
SnapPhysics: A Physics-Aware Scene Graph from a Single View for Interactive Mixed Reality Scenes
Suji Kang, Seok-Young Kim, Young Bin Kim et al. (+4 more)
We propose SnapPhysics, a training-free framework that reconstructs 3D objects and estimates their physical properties such as mass, friction, and center of
gravity from a single image. For physically coherent interactions in mixed reality (MR), such properties are as important as geometry. Prior approaches infer them by analyzing object dynamics in
video, which is computationally costly, or by querying
vision-
language models (VLMs) on single images, which lacks
geometric grounding and inter-object relationships. We address these limitations by combining instance-level 3D reconstruction and spatial
alignment with a physics-aware scene graph that encodes these relationships and per-object metric geometry as structured context for VLM-based property
reasoning. Experiments on 3D-FRONT show that SnapPhysics improves scene-level F-Score by 18.6% over the best learning-based method, and on real captured scenes with ground-truth mass, it reduces the mean absolute log difference error (mALDE) by up to 20.5% and improves log-scale correlation ($r^2_{\mathrm{ls}}$) by up to 19.6% over VLM-only estimation. SnapPhysics enables physically interactive MR experiences without manual parameter tuning. Project page: https://snapphysics-ismar2026.github.io/.
Read the full abstract ▾
All authors Suji Kang, Seok-Young Kim, Young Bin Kim, Taewook Ha, Dieter Schmalstieg, Shohei Mori, Woontack Woo
Computer Vision
Absence is Presence: Understanding Visual Scene Negative Events Under Safety Cognitive Constraint
Zhiyun Jiang, Hanyong Wang, Binbin Liang et al. (+3 more)
Traditional scene understanding focuses on affirmative information objectively present in images. However, in safety-critical domains, comprehending key information that should exist but is actually absent is vital for risk mitigation. To bridge this gap, we focus on visual scene negative captioning with safety as the cognitive constraint. The core challenge is to convert physical absence into semantic negative events. Existing
vision-
language models (VLMs) struggle with this process because affirmation bias suppresses negative
reasoning, while limited mental filling capability and representation bias further hinder the inference of absent information. To address these challenges, we propose a negative captioning framework based on counterfactual reconstruction and contrastive decoding (CRCD). Inspired by human cognition, CRCD reformulates the task as counterfactual latent change captioning to bypass affirmation bias. It contrasts a synthesized safe expectation with reality to identify semantic omissions. To address limited mental filling, we design a dual-branch counterfactual reconstruction architecture. The amodal completion branch restores defective objects, while the functional association branch infers completely absent safety objects. Concurrently, a multi-condition representation learning mechanism is integrated to mitigate representation bias by projecting universal features onto predefined safety criteria subspaces, thereby capturing information across more dimensions. By decoding feature-level semantic residuals between the reconstructed scene prototype and raw input, CRCD bounds the non-existence search space and activates the decoder's negative logic. Extensive experiments validate the effectiveness of CRCD, establishing a high-performance baseline for this pioneering task.
Read the full abstract ▾
All authors Zhiyun Jiang, Hanyong Wang, Binbin Liang, Yu Xie, Menglong Yang, Wei Li
Computer Vision
AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalization
Xu Yuan, Yi Wang, Zhuohang Jiang et al. (+8 more)
Recent advances in artificial intelligence (AI) are reshaping smart glasses from
egocentric capture and display devices into platforms for wearable intelligence. Smart glasses increasingly serve as wearable AI systems that connect first-person observation with real-time assistance under strict form-factor constraints. We frame this transition through the lens of AI smart glasses and define them as a system-level concept in which
egocentric sensing, resource-aware computing, intelligent
reasoning,
multimodal interaction, and real-world application constraints are co-designed for personalized assistance in the physical world. To systematically study this perspective, we organize the survey around four connected dimensions. First, we examine the hardware foundation that bounds sensing, computation, feedback delivery, and sustained deployment. Second, we study wearable intelligence, where
egocentric signals are transformed into perceptual, contextual, and agentic capabilities. Third, we discuss interaction design, through which users request, receive, correct, and regulate assistance during ongoing activity. Fourth, we analyze application scenarios across healthcare, accessibility, situated learning, daily life assistance, cultural tourism, and industrial support, showing how domain requirements reshape system design and evaluation. We further identify five cross-cutting research challenges for future AI smart glasses: next-generation hardware, trustworthy
egocentric intelligence, lifelong personalized
memory, proactive intelligence, and
embodied foundation models. By centering smart glasses as wearable-intelligence platforms, this survey provides a unified framework for organizing technologies, applications, and open challenges in this emerging area.
Read the full abstract ▾
All authors Xu Yuan, Yi Wang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Shanru Lin, Guoliang Xing, Hongxia Yang, Jiannong Cao, Qing Li, Wenqi Fan
Computer Vision · Cross-listed from cs.GR
Printing the Underdetermined: Materializing Multi-solutionness in Figurative Paintings
Yutao Ming, Teng Xu, Youjia Wang et al. (+6 more)
Figurative paintings are often approached as if they depict a single recoverable 3D scene: viewers infer depth and occlusion, and reconstruction pipelines attempt to converge to one stable model. We instead foreground multi-solutionness, the non-uniqueness of 3D configurations compatible with a single painted image, and propose a workflow that keeps this non-uniqueness visible and material. Multi-solutionness arises from two sources: unobserved content, where backsides and occluded volumes admit multiple plausible completions, and observed cues, where perspective, shading, and occlusion still underconstrain geometry. When additional views are synthesized by a
video generative model without explicit 3D constraints, small frame-level drifts become inevitable rather than exceptional. Our pipeline samples multiple camera-orbit multi-view
video sequences from one painting, reconstructs each sequence with 3D Gaussian Splatting into a point-based Gaussian scene representation where density
halos and ghosting expose unresolved degrees of freedom, and fabricates these representations as physical artifacts using DreamPrinting. By treating multiple compatible interpretations as explicit outputs rather than residual error, we provide a computational framework for spatial readings of figurative painting that can be inspected, compared, and discussed in both digital and physical form.
Read the full abstract ▾
All authors Yutao Ming, Teng Xu, Youjia Wang, Yunyang Liu, Fengmin Yang, Fuqiang Zhao, Jingyi Yu, Hua Yang, Yanjun Zhou
Computer Vision
Benchmarking MLLMs via Cognitive Expected Scene Graph for Safety-Critical Visual Negation Understanding
Zhiyun Jiang, Hanyong Wang, Binbin Liang et al. (+3 more)
True machine intelligence requires transcending passive pixel registration to master top-down functional
reasoning over absent information via visual negation understanding. However, unconstrained visual negation paradigms remain overly open-ended, and pervasive affirmation bias causes both existing Multi-Modal Large
Language Models (MLLMs) and evaluation metrics to fail under negative semantics. To solve these intertwined challenges systematically, we first anchor the boundaries of negation
reasoning within specific cognitive goals. Specifically, by focusing on safety as a highly pragmatic and critical cognitive dimension, we define the task of Scene Negation Understanding under Safety Cognition (SNUS). Under this framework, we construct a high-fidelity negative caption
dataset mapping dense assertions of localized hazards. Concurrently, we propose the Cognitive Expected Scene Graph (CESG) Score, a structure-grounded, polarity-aware evaluation metric. Extensive experiments demonstrate that while current models struggle on the task, traditional metrics completely collapse under semantic reversals. Conversely, our framework delivers a solid
benchmark for SNUS, providing a rigorous foundation to advance risk-aware situational comprehension and counterfactual cognition.
Read the full abstract ▾
All authors Zhiyun Jiang, Hanyong Wang, Binbin Liang, Yu Xie, Menglong Yang, Wei Li
Computer Vision · Cross-listed from eess.IV
HyperAMS-Net: Adaptive Multi-Scale Spatial Hypergraph Network for Brain Disorder Classification
Proloy Kumar Mondal, Md Kamran Hussin Chowdhury and Hoi Leong Lee
Accurate classification of
brain disorders from neuroimaging data remains challenging because of substantial inter-subject heterogeneity and the complex multi-scale patterns present in functional connectivity and morphological representations. To address these challenges, we propose HyperAMS-Net, a deep learning framework for
brain disorder classification using neuroimaging representations derived from resting-state functional MRI or structural MRI. HyperAMS-Net integrates adaptive multi-scale convolution, hypergraph
attention, spatial-channel
attention, and adaptive feature fusion. Specifically, adaptive multi-scale convolution learns data-driven weights over multiple receptive fields to capture complementary patterns at different scales. Hypergraph
attention models higher-order dependencies among learned feature representations through node–hyperedge–node message passing, while spatial-channel
attention enhances discriminative feature learning. Adaptive feature fusion further aggregates complementary information across parallel network branches. HyperAMS-Net is evaluated on three
benchmark datasets spanning distinct
brain disorders: ABIDE for autism spectrum disorder, REST-meta-MDD for major depressive disorder, and ADNI for Alzheimer's disease, using 5-fold stratified cross-validation. HyperAMS-Net achieves state-of-the-art performance across all evaluated
datasets, attaining the highest accuracy and AUC among the compared methods. Ablation studies further demonstrate the contribution of each proposed component, with the largest performance degradation observed when hypergraph
attention is removed.
Read the full abstract ▾
Computer Vision
STAR: Structure-aware Test-time Adaptation for diffusion-based light field Reconstruction
Wontae Choi, Ki Ryum Moon, Jae Young Lee et al. (+2 more)
Light field (LF) reconstruction from limited and noisy focal stack (FS) measurements is a highly ill-posed inverse problem. Although the LF-to-FS imaging geometry is fixed for a given optical setup, LF spatial-angular structure—including within-view spatial details, cross-view angular dependencies, and disparity across views—varies across scenes. Consequently, a fixed pre-trained prior may not optimally capture the spatial-angular structure of each test LF. We propose Structure-aware Test-time Adaptation for
diffusion-based light field Reconstruction (
STAR), the first test-time adaptation framework for reconstructing an LF from FS. For each test LF,
STAR freezes a pre-trained
diffusion prior and fits three lightweight adapters to the observed FS to jointly adapt the three components of the LF's spatial-angular structure.
STAR outperforms existing state-of-the-art methods in both two- and three-focal-sheet settings, with shorter inference times than those with test-time parameter updates.
Read the full abstract ▾
All authors Wontae Choi, Ki Ryum Moon, Jae Young Lee, Hyung Sup Yun, Il Yong Chun
Computer Vision
Region-Level Policy Optimization for Fine-grained MLLM Perception
Yuheng Shi, Xiaohuan Pei, Minjing Dong et al. (+1 more)
Fine-grained visual perception in MLLMs is commonly improved by raising the resolution, but the added visual
tokens inflate
vision-encoding and
language-model prefilling costs. We show that the two operations underlying fine-grained perception, localizing the region of interest (RoI) and recognizing its content, have different resolution requirements. In a controlled diagnostic, localization tolerates roughly 3 to 4 times stronger
token compression than recognition, which motivates localizing from a coarse view and concentrating resolution on the selected evidence. Decoding coordinates with the MLLM can be trained end-to-end from answers, but costs a full model pass per query and depends on grounding ability. A lightweight proposal network distilled from the model's
attention is fast, but inherits the noise of its
attention targets. The RoI from the proposal network reaches the answer through a discrete region choice, so its faithfulness to the answer cannot supervise the network. We therefore optimize the proposal network with region-level reinforcement learning, which we call
Vision-RL2. It treats coherent regions as actions, and a frozen MLLM reader scores each one by how its removal changes the answer likelihood. Complementary subtractive and additive objectives suppress distracting proposals and recover missing evidence, updating only the predictor without region annotations, response sampling, or
reasoning trajectories. The refined proposal further enables a sparse encoding that magnifies evidence and excludes background
tokens. Across six fine-grained
benchmarks and four MLLM backbones,
Vision-RL2 improves accuracy over the base model at every
token budget and surpasses its largest-budget accuracy with about 4 times fewer visual
tokens. Code is available at https://github.com/YuHengsss/VisionRL2 .
Read the full abstract ▾
All authors Yuheng Shi, Xiaohuan Pei, Minjing Dong, Chang Xu
Computer Vision
Federated Learning Framework for Privacy-Preserving Kidney Stone Detection
Najiyya Younas, Omar Abdulkader, Yaser Ali Shah et al. (+3 more)
Recent innovations in deep learning have significantly enhanced the diagnosis of medical images, although they are based on the use of centralized data storage that pose severe threats to patient
privacy and medical data security. To address this issue, this research proposes a
Federated Learning (FL) model that is coupled with an optimized YOLOv8 network to detect the kidney stones on a computed
tomography (CT) image and at the same time, protect
privacy of the patients. The suggested system can help various medical organizations to jointly train a common model without exchanging the information about the patients. This is to ensure that data protection laws like GDPR and HIPAA are adhered to. The residual feature fusion and DropBlock regularization among other architectural improvements are also included in YOLOv8 to enhance detection robustness and minimize overfitting. Experimental analysis carried out on a distributed CT
dataset demonstrated that the
federated YOLOv8 model has a mAP at 50 of 0.733 and is able to keep the data confidential. Moreover, its lean design facilitates fast edge deployment and real-time inference across a clinical setting. Altogether, these findings indicate that
Federated Learning is a safe and efficient solution to AI-assisted diagnosis in contemporary healthcare when combined with the use of sophisticated object detection models.
Read the full abstract ▾
All authors Najiyya Younas, Omar Abdulkader, Yaser Ali Shah, Muhammad Jawad Ikram, Jebran Khan, Amaad Khalil
Computer Vision · Cross-listed from eess.IV
The segmentation ceiling: why explicit left-ventricular masks do not improve learned ejection-fraction regression
Farshid Farhadi Khouzani, Paul La Plante, Bryar Mustafa Shareef et al. (+1 more)
Accurate estimation of left ventricular ejection fraction (EF) from echocardiography is central to cardiovascular care, and deep learning enables automated EF prediction from echocardiographic
video. Because EF is clinically derived from left-ventricular (LV) volumes, a widely held intuition is that explicit LV
segmentation should improve prediction. We introduce a quantitative criterion, the
segmentation ceiling, that makes this testable: from EF as a normalized difference of end-diastolic and end-systolic volumes, we derive in closed form how per-frame
segmentation area error propagates into EF error, and thus the accuracy a mask must reach before it can improve on direct regression. Using EchoNet-Dynamic, a UniFormer-S backbone, and the empirically measured within-patient error correlation, the criterion places the break-even near 10% per-frame area error, whereas a representative segmenter operates at roughly 14%, above the ceiling. Consistent with this, four strategies for injecting
segmentation or area information (a predicted-mask channel, end-diastolic/end-systolic clip sampling, and per-bin and amplitude area-consistency objectives) fail to beat a raw-
video baseline; ground-truth masks help only through label leakage. Input representation thus not being the limit, we identify generalization as the practical lever: weight averaging with strong augmentation attains a test R^2 of 0.806 (MAE 4.08) under a matched dense-clip protocol, comparable to an R(2+1)D baseline (0.811) while tightening the validation-to-test gap. Finally, a heteroscedastic beta-NLL formulation yields informative, well-calibrated per-prediction uncertainty, larger for clinically harder low-EF cases, where Monte-Carlo dropout does not. The
segmentation ceiling gives a concrete design criterion for when mask-guided EF estimation is worthwhile, plus a simple, uncertainty-aware recipe for EF regression.
Read the full abstract ▾
All authors Farshid Farhadi Khouzani, Paul La Plante, Bryar Mustafa Shareef, Laxmi Gewali
Computer Vision
Recency Forcing: Bridging the Long-Horizon Gap in Autoregressive Video Generation
Tri Cao, Hung Nguyen, Phong Nguyen et al. (+1 more)
Autoregressive (AR)
video generation degrades over long horizons due to an overlooked train-inference discrepancy we term KV eviction mismatch: models train on short clips where all context frames reside in the KV cache, but at inference,
memory constraints force distant frames to be evicted from the KV cache - removing context the model was conditioned on. Rather than simulating eviction via context truncation - which discards temporal information the model still needs and degrades motion coherence - we keep the context but while progressively reducing the influence of distant frames, making their eventual eviction negligible. To guide this design, we introduce the positional response $R( \Delta, \, t_{\text{denoise}})$, a perturbation-based sensitivity measure revealing that context influence decays steeply with temporal distance and varies systematically across denoising steps. Motivated by this analysis, we propose Recency Forcing, which applies a non-positive, timestep-dependent bias, termed Temporal Response Bias (TRB), on pre-softmax
attention logits derived directly from $R$, closing the train-inference gap without modifying context length or training objectives. We further introduce Biased
Attention Reparameterization (BAR), an exact reformulation that moves the bias outside the softmax, making TRB a standard FlashAttention call at zero overhead. Recency Forcing operates in both training-free mode and training-based mode. Experiments on VBench and VBench-Long demonstrate state-of-the-art long-horizon generation quality at no additional inference cost.
Read the full abstract ▾
All authors Tri Cao, Hung Nguyen, Phong Nguyen, Khoi Nguyen
Computer Vision
SeetaPsych v1.0: An Open-source Computer Vision Toolkit for Behavior-based Psychological Measurement
Jiabei Zeng, Chiqin Li, Kaizhou Li et al. (+7 more)
Automated visual analysis opens new avenues for behavior–based psychological measurement. Nevertheless, existing technological modules are typically scattered across task specific systems with heterogeneous interfaces and disparate deployment requirements. In this work, we present SeetaPsych v1.0, an open source, unified and extensible computer
vision toolkit designed to extract psychologically relevant signals from facial images and/or face based
videos. The current release encompasses four major core modules aiming at behavior–based physiological perception: unified face based
emotion analysis (simultaneous facial expression recognition, facial action unit detection, and valence–arousal estimation), camera based heart rate estimation, screen point–of–gaze estimation, and scene gaze following. A suite of auxiliary preprocessing modules for human centric visual analysis is also included, comprising face detection, facial landmark detection, and head detection. These functionalities are encapsulated within a modular Pipeline/Runner architecture that automatically resolves attribute dependencies, constructs computation graphs, and support intermediate result sharing among modules. SeetaPsych provides standardized Python APIs to facilitate reproducible, large scale analyses, alongside an interactive WebUI for rapid, code–free method evaluation. Overall, SeetaPsych offers an integrated and accessible visual measurement platform for research in psychology, behavioral science, human computer interaction, and related fields.
Read the full abstract ▾
All authors Jiabei Zeng, Chiqin Li, Kaizhou Li, Fei Chang, Yong Li, Yuanhao Zhao, Dan Han, Wenqiang Yang, Xilin Chen, Shiguang Shan
Computer Vision
GAPrompt++: Multi-Granular Geometry-Aware Point Cloud Prompt for 3D Vision Model
Zixiang Ai, Zhenyu Cui, Yufei Guo et al. (+4 more)
Pre-trained 3D
vision models have substantially advanced point cloud analysis, yet adapting them to downstream tasks via full fine-tuning is computationally expensive and storage-intensive. Parameter-Efficient Fine-Tuning (PEFT) offers a promising alternative by reducing both adaptation cost and storage burden. However, existing prompting-based approaches ignore the intrinsic
geometric structures of point clouds, thereby limiting their adaptation capability. This limitation stems from their inability to encode both fine-grained
geometric cues and coarse-grained structural semantics, as well as failing to propagate such information effectively through the model hierarchy. To address these challenges, we propose GAPrompt++, a multi-granular geometry-aware prompting method that provides richer
geometric guidance for efficient 3D task adaptation. Specifically, we introduce a Point Shift Prompter that extracts multi-granular
geometric features across different scales, enabling instance-specific
geometric adjustments during adaptation. Next, a Keypoint Prompter adaptively generates point-level
prompts to highlight local
geometric saliency and fine-grained structural details. Furthermore, a
Prompt Propagation mechanism injects these multi-granular
geometric cues throughout the feature extraction hierarchy, strengthening the ability to capture essential
geometric characteristics. Extensive experiments show that GAPrompt++ achieves state-of-the-art performance among prompting-based PEFT methods and even surpasses full fine-tuning across diverse
benchmarks, while requiring less than 2% trainable parameters. In addition, to address the saturation of existing evaluation
datasets, we construct two more challenging
benchmarks derived from 3D Gaussian Splatting and Multi-View Stereo reconstruction, offering diverse and realistic point cloud scenarios to promote future research.
Read the full abstract ▾
All authors Zixiang Ai, Zhenyu Cui, Yufei Guo, Wenwen Qiang, Lei Chen, Jiwen Lu, Jiahuan Zhou
Computer Vision
Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generation
Daeun Kim, Junwha Hong, Changhun Oh et al. (+3 more)
Autoregressive image generation has emerged as a paradigm for
multimodal AI systems due to its compatibility with
transformer-based LLM serving infrastructures. However, generating thousands of visual
tokens per request makes decoding increasingly bottlenecked by KV cache accesses during
attention computation. Sparse
attention is particularly attractive for this workload because many visual generation applications tolerate moderate quality degradation in exchange for improved performance and efficiency. While sparse
attention has been extensively explored for text-based LLM inference, it remains unclear whether its sparsity assumptions generalize effectively to autoregressive image generation. We present the first systematic characterization of
attention sparsity in autoregressive image generation across diverse workloads and representative open-source models. Our analysis reveals several distinguishing properties, including a pronounced prefill-decode asymmetry, strong
attention concentration on
prompt and local
tokens, and a unique diagonal
attention sparsity pattern arising from the spatial locality of visual
tokens. Motivated by these observations, we propose a diagonal-aware sparse
attention mechanism that selectively skips KV entries along the diagonal
attention direction within a recent window. Implemented on top of a GPU-based serving system using FlexGen, FlashAttention-2, and custom kernels, our approach achieves up to 3.1x throughput and 1.19x latency improvements with less than 2% quality degradation compared to dense inference.
Read the full abstract ▾
All authors Daeun Kim, Junwha Hong, Changhun Oh, Yoonsung Kim, Yoonhyeong Lee, Jongse Park
Computer Vision
IMFD: End-to-end Multi-Face Forgery Detection through Instruction-based Large Vision-Language Models
Dasom Choi, Sangjun Moon, Hyeongchan Im et al. (+5 more)
The rapid increase of deepfakes has raised significant concerns due to their spread on social media. Traditional multi-face forgery detectors crop and verify each face independently, ignoring background context and inter-face relationships, which often yields suboptimal performance. To overcome these limitations, we leverage instruction-based Large
Vision-
Language Models (LVLMs), which can interpret entire images and follow complex textual instructions. We propose a simple yet effective single-stage multi-face forgery detector, called IMFD (Instruction-based Multi-face Forgery Detector), which is trained end-to-end to jointly localize faces and predict per-face forgery labels. Rather than treating face box prediction only as a joint objective, IMFD explicitly integrates predicted face bounding boxes into the instruction as visual cues that enhance instruction grounding and forgery detection. To support the training and evaluation of IMFD, we convert existing multi-face forgery
datasets into an instruction-based format. Experimental results and analyses show that IMFD improves multi-face forgery detection by integrating face bounding boxes into the instruction, and consistently outperforms various state-of-the-art methods.
Read the full abstract ▾
All authors Dasom Choi, Sangjun Moon, Hyeongchan Im, Jaeeon Park, Jingun Kwon, Hidetaka Kamigaito, Taro Watanabe, Manabu Okumura
Computer Vision · Cross-listed from cs.AR
MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Acceleration
Yuan Liao and Jae-sun Seo
The deployment of
Vision-
Language Models (VLMs) on edge devices is severely bottlenecked by
memory bandwidth, necessitating aggressive sub-8-bit quantization. Since edge accelerators are strictly constrained by area and power, they require end-to-end quantized models. However, the extreme dynamic range gap between multi-modal
tokens causes standard block formats to suffer "microscaling collapse," where a single massive outlier hijacks the shared exponent, underflowing surrounding elements and destroying
attention maps. To break this bottleneck, we propose Micro-Inverted-Scaling (MiX), a novel format that mathematically inverts the microscaling paradigm: rather than grouping multiple mantissas under one shared exponent, MiX groups private, per-element exponents under a single shared mantissa. To handle asymmetric VLM outlier
topologies, we introduce an adaptive dual-format (MiX-MX) inference framework. By algebraically factoring out the shared MiX mantissa, this framework maps to a custom accelerator, replacing multipliers with efficient shifters. Evaluated end-to-end on multiple VLMs, our 4.5-bit MiX formulation exhibits equivalent or superior accuracy on multi-modal
benchmarks compared to NVFP4. Simultaneously, the MiX accelerator delivers a 25% improvement in area efficiency over the NVFP4 baseline and a 2.3-4.5x speedup with 1.4-2.9x energy reduction across models compared to the state-of-the-art accelerator Focus, proving the inverted-scaling datapath is physically superior for efficient VLM deployment.
Read the full abstract ▾