Computer Vision
Contrastive Joint-Embedding Prediction for Representation Learning in Structural MRI
Fabian Mager and Lars Kai Hansen
Self-supervised learning offers a compelling approach for medical imaging, where labeled data are scarce and acquisition costs are high. We present COJEPA, a self-supervised framework for volumetric
brain MRI that combines a joint-
embedding predictive architecture (JEPA) with a contrastive loss (CO), targeting two complementary properties: local predictivity and global discriminability. The model is trained without labels on T1-weighted structural MRI from two cohorts (HCP-YA and AABC, $N{=}2286$, ages 22 to 90), extending I-JEPA to 3D with foreground-aware block masking, a hierarchical convolutional patch
embedding, and world-space sinusoidal positional encodings. We evaluate all three objectives across zero-shot twin
retrieval,
brain tumor
segmentation (BraTS 2024), and age regression (OpenBHB). COJEPA achieves the best monozygotic twin recall at rank@1 (0.84), the best finetuning age MAE (2.55 years on OpenBHB 3.0T), and matches CO on BraTS whole-tumor Dice, demonstrating that the combined objective yields representations that are simultaneously discriminative and locally structured.
Read the full abstract ▾
Computer Vision
Anomalous Frame Detection Using VLM-Based Description Comparison for Extracting Expert-Specific Actions and Contextual Decision-Making Scenes with Intra-Video Self-Similarity
Ryo Sakai and Kaname Yokoyama
Maintenance of critical infrastructures, such as railways and power plants, is essential for ensuring operational safety and reliability. However, the declining number of skilled maintenance workers highlights the need to transfer expert know-how to less experienced workers. Previous studies have attempted to extract candidates of expert knowledge by comparing
videos of manual-based work with those of expert workers, mainly focusing on differences in observable actions. However, expert know-how is often embedded not only in actions but also in contextual decision-making during task execution. This paper proposes a method that detects anomalous frames between two task
videos to automatically extract candidate scenes containing expert-specific actions and contextual decision-making scenes. The method generates frame-wise visual descriptions using a
vision-
language model (VLM). Expert-specific actions are extracted based on frame similarities computed from description comparisons between two
videos, while contextual decision-making scenes are extracted using segment similarities derived from intra-
video self-similarity of the descriptions. In simulated distribution board maintenance experiments involving 27 task scenarios, the proposed method achieved extraction rates of 65% for action candidates and 61% for decision-scene candidates, improving over conventional methods that achieved 59% and 33%, respectively. These results demonstrate the effectiveness of the proposed approach in discovering candidate scenes containing expert know-how.
Read the full abstract ▾
Computer Vision
Learning Anatomy-Grounded CT Vision-Language Representations with Organ-Hierarchical Report Knowledge
Guoliang You, Hongming Li, Yuanwang Zhang et al. (+1 more)
Medical
vision-
language pretraining (VLP) from paired CT images and radiology reports enables scalable representation learning, but most existing methods align either whole scans with entire reports or local image regions with text fragments. These formulations underuse a key property of radiology reports: findings are organized around anatomical structures, with abnormalities described by organs, disease concepts, locations, and severity-related attributes. We propose OKA-CT, an organ-hierarchical knowledge-augmented framework for CT-report VLP. OKA-CT first converts free-text reports into organ-conditioned knowledge using radiology report parsing and LLM-assisted semantic structuring. The extracted hierarchy is used across two learning stages. Stage~1 injects anatomy-grounded evidence into the CT visual representation through fine-grained organ-conditioned supervision, while Stage~2 uses organ-specific report evidence to guide structured report-CT contrastive learning, where hierarchy-derived semantic soft targets treat non-paired cases with shared organ-level findings as weak semantic positives rather than uniform negatives. A lightweight query-based global branch further aggregates disease-relevant volumetric evidence for whole-scan representation. On CT-RATE and RAD-ChestCT
datasets, OKA-CT achieves zero-shot abnormality diagnosis AUROCs of 84.9 and 72.2, outperforming prior CT VLP baselines.
Retrieval and patch-occlusion analyses further show improved report-image
alignment and stronger sensitivity to disease-associated anatomical regions.
Read the full abstract ▾
All authors Guoliang You, Hongming Li, Yuanwang Zhang, Yong Fan
Computer Vision
Unsupervised Detection of Entry and Exit Regions from Vehicle Trajectories for Camera-Agnostic Turning Movement Counts
Parikshit Singh Rathore, Vishwajeet Pattanaik and Punit Rathore
Turning movement counts are essential for intersection-level traffic management, yet their collection remains predominantly manual due to the cost of per-camera region annotation. This paper presents an unsupervised pipeline that identifies entry and exit regions directly from raw vehicle trajectories extracted via object detection and multi-object tracking, requiring no manual annotation, camera calibration, or prior knowledge of intersection geometry. Unlike trajectory clustering methods that classify individual trajectories using pairwise similarity and must be re-executed on every new batch, the proposed pipeline clusters initial and terminal point locations to produce persistent spatial region polygons that classify future trajectories by point-in-polygon containment at linear cost. The pipeline comprises six sequential steps, five of which introduce configurable parameters evaluated through a systematic statistical analysis spanning 17,100 pipeline executions across 9 surveillance cameras capturing dense heterogeneous traffic in Bengaluru, India, and 10 sequences from the UA-DETRAC
benchmark dataset. Both parametric and nonparametric testing frameworks identify three consistently significant parameters and yield an empirically grounded recommended configuration. Under this configuration, the pipeline achieves a median classification error of 3.4% across all 25 Bengaluru cameras, including 16 held-out locations, with a median per-turning-movement GEH of 2.43. Compared with two trajectory clustering baselines, the proposed pipeline exhibits greater stability across camera views and lower computational cost, at the expense of higher median error. Extended evaluation demonstrates that calibration clips of at least 60 minutes and peak-traffic selection further improve region estimation quality.
Read the full abstract ▾
Computer Vision
DP-Splat: Bayesian Nonparametric Complexity Control for Gaussian Splatting
Aqi Dong
3D Gaussian Splatting represents scenes as finite mixtures of anisotropic Gaussians whose number of components $K$ is set by heuristic density control or user caps. Variational Bayes Gaussian Splatting (VBGS) recast splat fitting as conjugate variational inference, but $K$ remains fixed. We replace the finite symmetric Dirichlet over mixture weights with a truncated stick-breaking Dirichlet-process prior (or a sparse overfitted finite Dirichlet), so that the number of occupied components adapts to the data while every update remains a closed-form coordinate-ascent step; a natural-gradient
stochastic variant scales to large point sets. We give an ELBO monotonicity-and-convergence guarantee, a rigorous truncation-error bound that corrects a large-$\alpha$ approximation in common use which undershoots the exact Ishwaran-James bound, and an honest account of what the fitted number of components estimates. Empirically: (i) the effective complexity $\hat{K}$ adapts to data complexity and recovers the true $K$ within $\pm 1$; (ii) a deconfounded comparison shows the DP prior's contribution is complexity selection, not per-component efficiency: converged DP fits beat single-pass fixed-$K$ VBGS by +2.7 dB yet tie an equally converged fixed-$K$ baseline, and on all eight NeRF-synthetic scenes DP-Splat holds held-out point-space color prediction within 0.33 dB of VBGS with 3.9-7.9x fewer occupied components, while under full rasterization on held-out views it renders 3.4-9.0 dB below the unpruned baseline; (iii) the posterior-predictive color variance is well calibrated, and a seeded matched-capacity ablation attributes that calibration to the conjugate color model rather than to the weight prior; (iv) the ordering suggested by exact-posterior asymptotics reverses under mean-field coordinate ascent: the DP prior resists over-splitting while the sparse finite mixture saturates its truncation.
Read the full abstract ▾
Computer Vision
Design Choices in Splitting-Based Self-Supervised Sparse-View CT Reconstruction
Nadja Gruber, Lukas Neumann, Ander Biguri et al. (+3 more)
Self-supervised data splitting has emerged as a promising paradigm for sparse-view CT reconstruction, enabling training from incomplete measurements without fully sampled ground truth. However, the influence of key design choices, including partitioning strategy, preprocessing, and inference, remains insufficiently understood. In this work, we introduce a unified framework that decomposes splitting-based reconstruction into these three components, enabling controlled comparison of existing methods and two incremental extensions: multi-partition splitting and an alternative inference strategy. Experiments on simulated LoDoPaB-CT data under independent and correlated noise, together with validation on the real-world 2DeteCT
dataset, show that the optimal partitioning strategy strongly depends on the measurement noise structure.
Lattice-based splitting performs favorably under independent noise, whereas angular masking is more robust under correlated noise and real measured data. Multi-partition splitting consistently improves over pure projection-wise splitting in several settings. Complementary perceptual and structural metrics, including LPIPS and HaarPSI, reveal differences between masking strategies that are less apparent from PSNR and SSIM alone. These results provide practical guidelines for designing self-supervised sparse-view CT reconstruction methods and highlight the limitations of common independence assumptions in realistic imaging environments.
Read the full abstract ▾
All authors Nadja Gruber, Lukas Neumann, Ander Biguri, Gyeongha Hwang, Markus Haltmeier, Johannes Schwab
Computer Vision · Cross-listed from cs.RO
3D Scene Graph Prediction: Generating Hierarchical Models from Partially Observed Environments
Siyi Hu, Jared Strader, Hyungtae Lim et al. (+1 more)
Generating realistic 3D indoor scenes is an area of growing interest in computer
vision and robotics. Existing methods, often motivated by applications such as interior design, generally focus on object layout generation within a single room. The generation of high-level scene structure, such as room-level layout and traversability, remains underexplored despite its importance for robotics applications. In this paper, we consider the case where a
robot has explored part of an environment and needs to predict the unexplored parts to support downstream tasks such as exploration or object search. We propose a top-down framework for synthesizing hierarchical 3D scene graphs, including a room layer – describing the floor plan and traversability – and an object layer modeling object layouts within each room. For the room layer, we propose a novel mixed-domain graph
diffusion model jointly predicting room categories, floor boundaries, and traversability between rooms. Via corruption and masking, this model supports partial constraints such as incomplete floor plans, avoiding the need for partially observed training data. For the object layer, we integrate an existing mixed discrete-continuous
diffusion model for joint prediction of object categories, locations, sizes, and orientations within each room given the floor plan. We compare our method with state-of-the-art occupancy-based and LLM-based floor plan generation methods on a standard
benchmark. Compared with an occupancy-based learning baseline, our method generalizes substantially better to out-of-distribution partial floor plans. We also demonstrate our integrated prediction pipeline on real-world scenes from
robot-collected data, enabling prediction beyond explored areas.
Read the full abstract ▾
All authors Siyi Hu, Jared Strader, Hyungtae Lim, Luca Carlone
Computer Vision
X-GuideAR: An Augmented Reality Framework to Mitigate Radiation Exposure during Fluoroscopic Guidance
Mingxu Liu, Zixuan Liu, Ruchen Cai et al. (+5 more)
Achieving optimal screw placement for orthopedic surgeries requires frequent
alignment checks and multiple anatomical views under X-ray – a process known as "fluoro-hunting" that increases radiation exposure to patients and
surgical teams. This work introduces X-GuideAR, an augmented reality (AR) framework for identifying optimal X-ray views, aimed at reducing radiation exposure while ensuring accurate screw placement. To exemplify the benefits of X-GuideAR, we focus on S2 alar-iliac (S2AI) screw placement. Our system provides radiation-free guidance for view acquisition and drilling by generating synthetic X-ray previews that accelerate fluoro-hunting. Once the required anatomical views are identified using these previews, a real X-ray is acquired, and the preview of the drilling trajectory is augmented onto it, facilitating precise screw placement with minimal additional radiation. A preliminary study involving eight S2AI trajectories performed by an expert spine surgeon demonstrated a 62.3% reduction in the number of X-rays. Post-procedure evaluations showed that trajectories done with X-GuideAR supported an average safe screw diameter of 12.95 mm compared to 5.9 mm under the conventional workflow, suggesting improved bony containment and potential biomechanical benefit. X-GuideAR shows great potential to reduce radiation exposure and streamline S2AI screw placement, offering a promising direction toward safer and more efficient surgeries.
Read the full abstract ▾
All authors Mingxu Liu, Zixuan Liu, Ruchen Cai, Yu-Chen Ku, Suxi Gu, Amit Jain, Alejandro Martin-Gomez, Mehran Armand
Computer Vision
AU-Guided Synthetic Video Generation for Micro-Expression Recognition
Pei-Sze Tan, Sailaja Rajanala, Yee-Fan Tan et al. (+2 more)
Micro-expression recognition is limited by the small scale, narrow demographic coverage, and restricted
emotion labels of existing
datasets. We introduce EquiME, a synthetic micro-expression
dataset built from AU-guided image-to-
video generation. EquiME contains 75K
videos generated from 15K source face images across five target
emotions, together with automatically inferred demographic metadata and
video-quality measurements. We evaluate EquiME using frame-pair similarity, spatial variation, and no-reference perceptual-quality metrics, together with cross-
dataset MER experiments on SAMM and CASME II. Models trained on EquiME achieve competitive cross-
dataset performance on SAMM and CASME II and show comparatively low variation across the four evaluated architectures. This paper focuses on the
dataset design, the structured AU-conditioning pipeline used for
video generation, and the empirical evidence needed to assess EquiME as a synthetic MER resource. Project page: https://kirito-blade.github.io/me-vlm/
Read the full abstract ▾
All authors Pei-Sze Tan, Sailaja Rajanala, Yee-Fan Tan, Raphael C.-W. Phan, Huey-Fang Ong
Computer Vision
Diversify Diffusion with Temperature Sampling and Variance-Corrective Time Shifting
Peizhuo Li, Emre Aksan, Alexandru-Eugen Ichim et al. (+2 more)
Diffusion models faithfully reproduce their training distribution, but also inherit its imbalances and leave rare or under-represented modes hard to reach. A natural inference-time remedy is to sample from the high-temperature target $p^{(\gamma)}_0(x) \propto p_0(x)^{\gamma}$ for $0 < \gamma < 1$, which flattens dominant modes and lifts rare ones. However, naive score scaling while correctly reweighting modes also inflates the per-mode variance, breaking the reverse
diffusion process and degrading sample quality. We introduce variance-corrective time shifting, a training-free fix that queries the network at a shifted timestep and scales the resulting score by $\gamma$, canceling the variance inflation while preserving the mode reweighting. The correction turns simple temperature sampling into a practical diversity knob for pretrained
diffusion and flow-matching backbones with no retraining, and we demonstrate consistent gains at minimal cost to sample quality and condition fidelity across DiT, Stable
Diffusion and Motion
Diffusion models. We further show that the timing of the temperature intervention enables coarse-to-fine control: high-noise stages drive compositional diversity across modes, while low-noise stages drive local appearance variation under a fixed composition.
Read the full abstract ▾
All authors Peizhuo Li, Emre Aksan, Alexandru-Eugen Ichim, Thabo Beeler, Olga Sorkine-Hornung
Computer Vision
Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification
Tonmoy Hossain, Atiqur Rahman, Farhana Hossain Swarnali et al. (+1 more)
Medical image classification models are ideally expected to identify diagnostically relevant regions while making predictions, yet standard classification losses rarely provide spatial supervision. Explicit supervision via anatomical shape information, such as
segmentation masks of task-relevant anatomy, has been shown to guide the network toward regions relevant to the target prediction. However, obtaining such masks incurs substantial manual annotation effort and computational overhead. With the advent of
segmentation foundation models that exhibit strong localization of anatomical structures across diverse imaging modalities, we leverage this capability to extract anatomical shape priors without the burden of training a dedicated
segmentation model. In this paper, we propose a new framework, Locus, an anatomical
attention regularization framework that leverages pretrained
segmentation foundation models to guide a classifier's
attention toward diagnostically meaningful anatomical structures across diverse imaging modalities. Instead of enforcing pixel-wise
alignment with the foundation-model-derived mask, we introduce a regularization term that adaptively balances
attention between anatomical (foreground) and background regions, penalizing the classifier when background
attention dominates. We validate Locus on eight diverse medical imaging
datasets spanning dermoscopy, X-ray, histopathology, and cardiac MRI, showing consistent gains in classification performance alongside improved anatomically grounded
attention.
Read the full abstract ▾
All authors Tonmoy Hossain, Atiqur Rahman, Farhana Hossain Swarnali, Miaomiao Zhang
Computer Vision
Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels
Venkanna Babu Guthula, Oswin Krause, Dimitri Gominski et al. (+5 more)
Supervised learning for image
segmentation typically requires spatially aligned image and label sets. When images and labels originate from different sources, the pairing may be misaligned, which can significantly deteriorate the performance of the learned models. This is especially common in remote sensing, when aerial or satellite images are co-registered with labels from another source (e.g., OpenStreetMap). In this work, we propose a novel approach for training on misaligned labels, where we simultaneously learn the label
alignment. Our align and segment (AnS) approach builds on the spatial
transformer module to transform the misaligned labels using an affine transformation to provide a better learning target for a canonical semantic
segmentation network. We prevent shortcut learning of misaligned labels in these semantic
segmentation networks through a self-supervised regularization loss and show that it is complementary to data augmentation, especially for systematically misaligned training data. A decisive characteristic of our AnS approach is that it learns without requiring any golden labels. We experimentally show on both synthetic and real-world data from different cities that our approach enables high-quality building
segmentation and precise label-image
alignment at the same time. Code and derived
datasets are available at https://github.com/venkanna37/align-and-segment
Read the full abstract ▾
All authors Venkanna Babu Guthula, Oswin Krause, Dimitri Gominski, Hui Zhang, Johan Mottelson, Ankit Kariryaa, Nico Lang, Christian Igel
Computer Vision
OmniX: Any-view and Any-time 4D Reconstruction via Feed-forward Trajectory Fields
Yanqin Jiang, Tengfei Wang, Zhengwei Wang et al. (+6 more)
Previous feed-forward 4D reconstruction methods either predict per-frame static point clouds, ignoring foreground motion, or estimate point cloud trajectories while being limited to small camera motions. This restricts their ability to aggregate observations over time and reconstruct complete dynamic scenes under large viewpoint changes. To address this limitation, we propose OmniX, a feed-forward 4D reconstruction framework that predicts dense 3D point trajectories for every pixel from
videos with large camera motion. OmniX decouples dynamic motion modeling from static geometry prediction and represents motion using a compact set of dynamic
tokens. By leveraging the sparse and low-rank structure of 3D motion, these
tokens generate trajectory fields for all pixels across all images while efficiently preserving global interactions. To facilitate training, we further build an automatic UE5-based 4D data engine and introduce a large-scale
dataset containing 80K scenes and 1.28M multi-view
videos with full
geometric annotations. OmniX achieves state-of-the-art performance on dense 3D point trajectory prediction and 3D point tracking, while also demonstrating competitive results on
video depth estimation and camera pose estimation.
Read the full abstract ▾
All authors Yanqin Jiang, Tengfei Wang, Zhengwei Wang, Chenjie Cao, Junta Wu, Wenhan Luo, Weiming Hu, Jin Gao, Chunchao Guo
Computer Vision
3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects
Zhenyu Zhao, Nanshan Jia, Jihyeon Je et al. (+7 more)
Automated evaluation is essential for scaling generative 3D systems, where exhaustive human review is costly and slow. Yet the reliability of an automated judge depends on the full evaluation pipeline, including the
vision-
language model (VLM), asset rendering, visual evidence, task specification, and human reference labels. We introduce 3D-DefectBench, a large-scale
benchmark for rigorous evaluation-pipeline analysis. It complements holistic ratings and pairwise preferences with nine fine-grained binary defects spanning geometry, texture, and
prompt adherence, with optional human severity annotations. Using a balanced factorial design, we vary the VLM, camera protocol, visual input, and
prompt schema across 84 inference designs, and validate the resulting conclusions on a broader set of frontier models. Model choice is the dominant source of variation in agreement with human labels, while other pipeline factors also influence agreement, interact with the model, and can alter the best configuration. A compact six-view RGB protocol performs comparably to denser view sets and configurations augmented with depth or normal channels, making it a strong cost-effective default. Under this fixed design, the best of 12 VLMs still trail trained human labelers, and texture agreement drops sharply from expert-agreement to noisier silver labels. Severity annotations further show that binary judges recover most defects humans flag as severe. These results highlight the importance of evaluating automated judges as complete pipelines and calibrating them across human reference regimes.
Read the full abstract ▾
All authors Zhenyu Zhao, Nanshan Jia, Jihyeon Je, Yifu Tang, Alvin Chan, Michael Spedden, Michael V. Palleschi, Sui Huang, Jingshen Wang, Zeyu Zheng
Computer Vision
h-Flow: Flexible Flow-based Image Editing via Doob's h-Transform
Zehui Guo, Zhen Wang, Junwei Shu et al. (+3 more)
Editing images with pre-trained text-to-image flow models typically requires carefully balancing target
alignment with the desired
prompt and source consistency with the original image. Existing approaches either rely on inversion-based pipelines or heuristic source-to-target trajectory constructions, which often depend on architecture-specific designs or are sensitive to hyperparameters. In this paper, we propose h-Flow, a training-free and theoretically grounded flow-based editing framework. Inspired by Doob's $h$-Transform, we reformulate image editing as conditional generation under multiple terminal events corresponding to source consistency and target
alignment. We first extend the classical $h$-Transform from SDE-based models to the deterministic RF framework by constructing an equivalent SDE with identical marginals. Within this formulation, we design dedicated $h$-functions for source consistency and target
alignment, yielding closed-form reconstruction guidance and velocity-based semantic editing signals. We further introduce a velocity orthogonal decomposition to decouple reconstruction and editing directions, enabling a controllable trade-off between the two objectives. Extensive experiments demonstrate that h-Flow achieves effective, robust, and flexible editing across diverse scenarios. The code will be released soon.
Read the full abstract ▾
All authors Zehui Guo, Zhen Wang, Junwei Shu, Yang Li, Changbo Wang, Long Chen
Computer Vision
Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos
Hao Zheng, Jinyi Huang, Tiantian Zheng et al. (+2 more)
Assembly action understanding is a key enabler for effective human-
robot collaborative assembly, yet it remains challenging due to subtle motions and fine-grained hand-object interactions. We adapt
vision-
language models (VLMs) to this challenging domain with Compositional Context Fine-Tuning (CCFT), a method that decomposes assembly actions into semantic elements (Verb, Object, Tool) and fine-tunes VLMs to recognize each action element using templated question-answering pairs. This approach ensures near-deterministic outputs. To enable efficient and effective multi-task learning under limited data, a Layer-Partitioned Alternating Training (LP-AT) method is presented, which assigns distinct model layers to recognize specific action elements through element-specific low-rank adapters. LP-AT alternates weight updates across element-specific adapters, reducing cross-task interference while enabling per-adapter hyperparameter optimization. Furthermore, we create HA-ViD-VQA and IKEA-ASM-VQA
datasets from existing assembly
video datasets. Extensive experiments on these
datasets demonstrate that our method consistently outperforms strong action recognition baselines while providing interpretable element-level predictions that can support diverse downstream applications.
Read the full abstract ▾
All authors Hao Zheng, Jinyi Huang, Tiantian Zheng, Xun Xu, Tuka Alhanai
Computer Vision
Mixture of Cognitive Experts in Large Vision-Language Models
Robert Wijaya and Ngai-Man Cheung
Large
Vision Language Models (LVLMs) require strong
reasoning over both visual and textual input. Recent work suggests that cognitive elements, especially diverse representations and metacognition, correlate with better performance. Many of the needed perceptual functions are already provided by specialized domain-specific computer
vision models, which act as the perceptual subsystem for detecting objects, localizing them, inferring states, recovering spatial layout, and reading text. The key challenge is to integrate these multi-encoder experts into a trustworthy, interpretable, and coherent representation that improves verifiability and reduces
hallucinations. This is difficult because
vision-
language questions span different cognitive levels, yet most LVLM pipelines apply the same perception-
reasoning routing regardless of the demand of each query. We propose an evidence-driven
multimodal reasoning framework that utilizes a Bloom-inspired taxonomy as a hierarchical
reasoning protocol. The two-stage cognitive verbalization first produces a Literal Evidence Summary by decomposing expert outputs into short, atomic evidence statements. It then performs Bloom Verbalization to turn these evidence items into a staged
reasoning trace, and a lightweight
Reasoning Trace Module quantitatively analyzes the trace to make evidence usage and
reasoning progression explicit. Through this integration, we observed several improvements in perception and
reasoning abilities. Moreover, the trace module provides quantitative evidence that different queries induce different cognitive entry levels and evidence-use trajectories that enable fine-grained analysis.
Read the full abstract ▾
Computer Vision
MAC-Splat: Multi-Attribute Consistency for High-Fidelity Sparse-View Reconstruction
Jinqian Yang, Yichen Wu, Wanhua Li et al. (+4 more)
Reconstructing high-fidelity 3D scenes from sparse-views remains a central problem in generalizable neural rendering. Existing generalizable 3D Gaussian Splatting (3DGS) methods often exhibit
geometric artifacts in sparse-view settings, since supervision based solely on 2D photometric losses cannot resolve depth and correspondence ambiguities. To address this issue, we propose MAC-Splat, a training framework built around direct 3D consistency supervision. MAC-Splat builds on the MASt3R
geometric backbone and a frozen DINOv3 encoder to obtain semantically informed 2D correspondences, which serve as
geometric anchors for 3D supervision. Using these anchors, we define the Multi-Attribute Consistency (MAC) loss. This objective jointly regularizes the 3D attributes of matched Gaussians, including their position, shape, and appearance, by enforcing agreement in a common world coordinate frame. The formulation is robust to outliers and respects the geometry of covariance
matrices, which leads to stable training under sparse-view conditions. Experiments on ScanNet++ show that MAC-Splat outperforms strong baselines, with particularly large gains under different overlap regimes. In particular, it improves average PSNR over Splatt3R by more than 4.5 dB, reduces LPIPS, and maintains performance as the camera pose gap increases. These results indicate that a direct, multi-attribute 3D consistency objective, when combined with high-quality correspondences, is effective for addressing the ill-posed sparse-view reconstruction problem.
Read the full abstract ▾
All authors Jinqian Yang, Yichen Wu, Wanhua Li, Haokun Lin, Renzhen Wang, Xiangchu Feng, Xixi Jia
Computer Vision
Detecting AI-Generated Video: A Vision-Language Dual-View Survey
Dylan Xinming Hou, Juntian Zhang, Xu Gu et al. (+5 more)
The evolving realism of AI-generated
Videos (AIGC-V) is rapidly rendering traditional artifact-centric detection insufficient, necessitating a paradigm shift from low-level inspection to high-level semantic verification. This paper presents a comprehensive survey of AIGC-V detection, reframing the task as Factual Fidelity Verification, which asks whether the events, entities, and physical processes depicted in a
video are consistent with real-world facts. To systematize this rapidly evolving field, we propose a
Vision-
Language Dual-View taxonomy that organizes existing methods into a hierarchical, four-layer landscape, spanning intrinsic cue analysis, spatiotemporal consistency modeling, cross-modal consistency
reasoning, and
language-guided world-level
reasoning. This dual-view framing highlights a fundamental transition from artifact matching in traditional deepfake detection to evidence-based semantic verification enabled by
vision-
language models and agentic
reasoning pipelines. Based on a systematic review of 221 works, we synthesize AIGC-V generation paradigms, survey the landscape of detection methods, and review evaluation metrics and
benchmarks in line with proposed views. Finally, we discuss current challenges and identify promising directions toward robust, explainable, and trustworthy detection.
Read the full abstract ▾
All authors Dylan Xinming Hou, Juntian Zhang, Xu Gu, Yichen Wu, Nils Lukas, Gus Xia, Xiuying Chen, Yuhan Liu
Computer Vision
Toward Efficient Weakly Supervised Semantic Segmentation Using Only Low-Magnification Histopathological Images
Dung Minh Do, Nhat-Thanh Huynh, Duc Minh Huynh et al. (+2 more)
Whole-slide images (WSIs) provide rich tissue-level and cellular-level information, but storing and transmitting high-magnification pathology data is resource-intensive. Moreover, annotating WSIs at the pixel level is labor-intensive and time-consuming. Therefore, it is important to investigate whether low-magnification pathology images with limited annotations (i.e., image-level instead of pixel-level labels) can achieve performance comparable to high-magnification images. This paper presents a systematic
benchmark study on weakly supervised histopathological image
segmentation under different low-resolution storage settings. Starting from high-resolution image patches, we simulate lower-magnification inputs and reconstruct them to the original size using interpolation and deep learning-based reconstruction methods before applying the weakly-supervised
segmentation pipeline. This framework enables a quantitative evaluation of how weakly supervised methods respond to different levels of resolution degradation. Experimental results show that reconstruction quality metrics alone are insufficient to predict downstream
segmentation performance. In particular, the study identifies a critical degradation point where the localization of small-scale structures declines significantly. These findings provide practical guidance for designing efficient digital pathology storage systems while maintaining reliable automated analysis. Code is available at https://github.com/Dung-Dx/LowMagWSS
Read the full abstract ▾
All authors Dung Minh Do, Nhat-Thanh Huynh, Duc Minh Huynh, Doanh C. Bui, Khang Nguyen
Computer Vision
Is Energy Guidance All You Need? Training-Free Norm Injection for Driving World Models
Xiyan Su, Frank Diermeyer and Markus Lienkamp
Driving world models built on large
video-
diffusion backbones generate realistic scenes but are hard to control: enforcing a traffic norm typically means retraining the backbone or conditioning it on hand-built layouts. We ask whether controllability requires training at all. Our experiment shows that a rectified-flow driving world model, which jointly generates future
video and a planned ego trajectory, can have its planned trajectory steered entirely at sampling time by differentiable energy functions that encode driving norms, without knowledge-specific retraining of the
diffusion backbone. Concretely, we demonstrate that a world model built on Open-Sora 2.0 MM-DiT backbone can be steered to brake at a counterfactual target by injecting energy guidance at sampling time. However, we find that the generated
video does not yet follow the steered trajectory through the backbone's joint self-
attention and identify the cross-stream coupling as a crucial requirement for end-to-end-controllable rollouts.
Read the full abstract ▾
Computer Vision
RED-Sphere: Hyperspherical Residual Edge Debiasing for Cross-Population Fundus Disease Domain Generalization
Yan Lin, Ziheng Wang, Shuang Chen et al. (+2 more)
Medical image classifiers are often trained within one source population, yet clinical deployment requires robustness to patients whose appearance, acquisition style, and disease prevalence differ from the source cohort. Existing
fairness and robustness methods often require group supervision or treat appearance variation as an undifferentiated nuisance, which is insufficient when population-correlated low-level cues and lesion evidence share edge and texture structure. We study a strict source-only cross-population setting, where external populations are unseen during optimization, validation, scheduling, hyperparameter and model selection. We propose RED-Sphere, a plug-and-play robustness framework for image classification under unseen population shifts. It estimates shortcut-sensitive nuisance responses with an edge and feature energy prior, attenuates dominant responses through residual soft gating, regularizes masked nuisance views with counterfactual-inspired consistency and separation losses, and predicts labels with normalized spherical prototypes. It favours angular semantic evidence over source-correlated activation magnitude while preserving lesion structure. Although demonstrated on 2D Scanning
Laser Ophthalmoscopy (SLO) fundus classification for Age-Related Macular Degeneration (AMD) and Diabetic Retinopathy (DR), RED-Sphere is not tied to retinal anatomy: the same principle can be adapted with modality-specific nuisance priors wherever appearance shortcuts and semantic evidence are entangled. Under a strict White-only Harvard-FairVision protocol, RED-Sphere improves held-out macro-F1 across all 20 task and backbone comparisons, with average gains of 1.28 and 2.98 F1 points on AMD and DR. Gains in AUC and PR-AUC, visual diagnostics, ablations, and sensitivity analyses further support stronger external semantic
alignment and more stable angular disease geometry.
Read the full abstract ▾
All authors Yan Lin, Ziheng Wang, Shuang Chen, Amir Atapour-Abarghouei, Stephen McGough
Computer Vision
TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation
Sutharsan Mahendran, Darshana Priyasad, Kaushik Roy et al. (+4 more)
Cross-modal
distillation from
Vision Foundation Models (VFMs) to
LiDAR backbones has recently emerged as a self-supervised pretraining strategy that reduces reliance on dense point-wise annotation for 3D scene understanding. However, existing
distillation pipelines typically treat the VFM as a frozen feature source and train a heterogeneous 3D backbone to match fixed image
embeddings, forcing the student to bridge both the modality gap and the cross-architecture gap between dense ViT
token representations and sparse 3D encoders. We propose TOLiD, a self-supervised pretraining method for
LiDAR representation learning that addresses this gap by coupling a
LiDAR backbone with a student
Vision Transformer (ViT) initialized from a frozen VFM teacher and applying supervision over compatible patch-
token representations. TOLiD converts the set of point features within each image patch frustum into a
token using Frustum Pooling followed by Frustum
Attention, and performs
token-level
distillation with visibility masking. For
LiDAR-only deployment, we lift
token features back to per-point representations using masked bilinear sampling to avoid patches that have limited
LiDAR points. We extensively evaluate TOLiD on five heterogeneous
LiDAR datasets and four cross-sensor adaptation pairs, demonstrating improved transfer with frozen backbones and lightweight heads.
Read the full abstract ▾
All authors Sutharsan Mahendran, Darshana Priyasad, Kaushik Roy, Tharindu Fernando, Sridha Sridharan, Clinton Fookes, Peyman Moghadam
Computer Vision
TriCons-Pose: Triangle-Invariant Geometric Consistency Learning for Category-Level Object Pose Estimation
Zuzhi Yang, Bingtao Ma, Shuai Wang et al. (+5 more)
Category-level object pose estimation is a crucial yet challenging task in both academia and industry, and has achieved remarkable success by leveraging keypoint-based correspondence paradigms. However, most existing methods increasingly rely on stronger feature learning while overlooking whether the established correspondences are geometrically stable across diverse perturbations. This often results in fragile pose recovery under intra-class shape variations and occlusions. To tackle this challenge, we develop a novel Triangle-Invariant
Geometric Consistency Learning for Category-Level Object Pose Estimation (TriCons-Pose) to anchor stable keypoints and aggregate pose-invariant cues, yielding reliable canonical mapping and accurate pose estimation. Specifically, a Structure-Consistent Keypoint Detector (SCKD) is designed to identify robust keypoints by enforcing cross-view structural consistency via normalized pairwise distance matching. Moreover, we propose a Pose-Invariant
Geometric Aggregator (PIGA) to augment keypoint representations by injecting triangle-based pose-invariant descriptors into a local-to-global
attention mechanism. The proposed framework is optimized using standard objective functions while incorporating an additional geometry consistency loss. Extensive experiments on REAL275, CAMERA25, and HouseCat6D
datasets demonstrate the effectiveness of the proposed approach.
Read the full abstract ▾
All authors Zuzhi Yang, Bingtao Ma, Shuai Wang, Mounir Kaaniche, Ziwei Li, Zhiming Cheng, Zhidong Zhao, Chenggang Yan
Computer Vision
Water Reflection Detection Using Symmetric Attention
Shuxuan Yao, Chengjia Wang, Jianyuan Sun et al. (+2 more)
Reflections of water pose a significant challenge for computer
vision systems, as standard deep learning models frequently confuse objects with their mirror images, producing spurious false positives and negatives in tasks such as object detection and semantic
segmentation. As a result, detecting reflection axes in natural-water scenes is pivotal for reliable object detection and scene understanding. To mitigate this issue, we leverage the intrinsic imperfect reflective
symmetry of water and introduce a
Symmetry-Aware Water Reflection Detection Network, namely, SAWRD-Net, that couples dihedral group-equivariant convolutions with a
matrix-decomposition decoder in an end-to-end framework. First, dihedral group convolutional layers extract geometry-consistent feature maps that explicitly encode both rotational and mirror
symmetries. A Multi-scale Reflection Equivariant block then aggregates features across scales and employs a symmetric-
attention mechanism to highlight reflection-relevant regions. The proposed
matrix-decomposition decoder factorizes high-dimensional features into compact low-rank parameter and confidence spaces, after which the network directly regresses keypoints on the reflection axis. Then a robust principal component analysis fits the final axis. Evaluated on the largest available water reflection scene data set, SAWRD-Net achieves a true-positive rate of 0.890 against human annotations, outperforming all existing water reflection detectors.
Read the full abstract ▾
All authors Shuxuan Yao, Chengjia Wang, Jianyuan Sun, Junyu Dong, Xinghui Dong
Computer Vision
Traj-VLN: Learning Pixel-Space Interaction via Autoregressive Trajectory Generation
Changfei Fu, Guangcheng Chen, Aoxiang Gu et al. (+3 more)
Benefiting from the powerful priors embedded in large-scale pre-training data and the emerging commonsense
reasoning ability, large
language models (LLMs) have shown unprecedented generalization capabilities in many research fields. Recently, projecting visual
embeddings into the
language space via
vision-
language models (VLMs) to achieve sim-toreal and cross-scene generalization has become a prevailing paradigm in the field of
Vision-and-
Language Navigation in Continuous Environments (VLN-CE). VLN requires an
embodied agent to navigate through unseen environments following natural linguistic instructions. We emphasize that a VLN task can be decomposed into a sequence of sub-tasks, each corresponding to a process of 3D spatial interaction with the environments described by instructions such as "walk to the end of the sofa and turn left." However, such spatial interactions involving moving into the image along the direction of depth sensing are puzzling for VLMs as they were predominantly trained on conversations with RGB images. Rather than incorporating depth or 3D
geometric information-which VLMs rarely encounter during pretrainingwe propose an alternative approach: fine-tuning VLMs to learn
navigation interactions directly in 2D pixel space through autoregressive trajectory generation. Given a linguistic instruction and historical observations, our model sequentially predicts a series of pixel coordinates, drawing a trajectory from the bottom center of the current observation. While prior work has proved that pixel-goal supervision outperforms learning of discrete actions, our experiments further verify that the supervision of pixel-space trajectory significantly enhances VLN performance. Moreover, we demonstrate that our flagship model achieves state-of-the-art level performance with relatively limited computational resources and training data.
Read the full abstract ▾
All authors Changfei Fu, Guangcheng Chen, Aoxiang Gu, Haoxiang Liang, Wenjun Xu, Hong Zhang
Computer Vision · Cross-listed from cs.RO
Action Map Policy: Learning 3D Closed-loop Manipulation via Pixel Classification
Haojie Huang, Zhang Ye, Linfeng Zhao et al. (+7 more)
The action space poses a major challenge in
robot learning, since it is often high-dimensional, can span long time horizons, and frequently admits multi-modal optimal solutions. A good choice of action representation and loss function can help to address these concerns, but there are often trade offs. We propose Action Map Policy (AMP), which casts 3D closed-loop manipulation policy learning as a classification problem in image space. While classification has been an effective formulation in generative
language models, applying it to
robot action learning is difficult because naively discretizing high-dimensional continuous actions explodes the
token vocabulary. Our key idea is to project 3D actions onto the camera image planes and treat each pixel location as a discrete class, thus controlling dimensionality while retaining multi-modality. This method supports millimeter-level precision for high-dimensional actions without requiring a prohibitively large vocabulary, while preserving fine-grained pixel-wise visual signals. Furthermore, it can predict the entire action chunk in a single forward pass, avoiding complex noise scheduling and iterative denoising while achieving substantially faster inference than
diffusion policies. Experiments on various manipulation tasks show that AMP outperforms strong baselines, achieving higher success rates, faster inference, and enhanced spatial
reasoning.
Read the full abstract ▾
All authors Haojie Huang, Zhang Ye, Linfeng Zhao, Boce Hu, Mingxi Jia, Yu Qi, Ahmed Agha, Dian Wang, Robert Platt, Robin Walters
Computer Vision · Cross-listed from cs.LG
On the modality gap and the contrastive loss in multi-modal representation learning
Fabian Mager, Hiba Nassar and Lars Kai Hansen
We study the modality gap in CLIP-style dual-encoder contrastive learning, where image and text
embeddings remain misaligned despite being trained in a shared space. We argue that the gap is induced by a failure of the InfoNCE formulation with independent encoders. We conduct a uni-modal experiment with two independent encoders and identical initialization conditions and find that InfoNCE actively generates a gap at low temperatures. We provide a theoretical analysis of this phenomenon and show that the modality gap is indeed a mode-failure of InfoNCE, but only at low temperatures. We propose a simple modification called xNCE, which uses intermodal as well as intra-modality negative contrastive pairs. xNCE matches
retrieval performance on MS-COCO while consistently reducing the gap even at low temperatures. Notably, xNCE improves zero-shot classification over the InfoNCE baseline across all
benchmarks, whereas high-temperature InfoNCE and regularized InfoNCE both fail to do so, demonstrating that xNCE reduces the modality gap without sacrificing the discriminative geometry needed for transfer.
Read the full abstract ▾
Computer Vision
Effective Synthetic Image Detection via Noise Residual Clustering
Caihui Yan, Gang Cao, Huawei Tian et al. (+2 more)
The rapid advancement of generative artificial intelligence (AI) has made synthetic images remarkably realistic, posing security threats such as misinformation and fraud. It is significant to detect the synthetic image in the manner of passive and blind image authentication. Most existing detectors rely on supervised training with large labeled
datasets, leading to high costs and degraded performance on unknown generative models. To attenuate such deficiencies, we propose a training-free detection method. Specifically, noise residual fingerprints are first extracted by a simple yet effective pre-trained Noiseprint++ model. Then multi-scale features are further extracted from such residual by a frozen
Vision Transformer (ViT), followed by adaptive weighted fusion. Only a few real image samples are used needed to initialize the clustering centers for unsupervised K-Means, distinguishing real and synthetic images without training. Extensive evaluations on four
benchmark datasets show that our proposed scheme achieves an average accuracy of 82.2%, outperforming the state-of-the-art detectors on generalization ability. Superior performance is gained on the popular
diffusion type of synthetic images, and the effectiveness of each module is validated by ablation studies. Source code will be publicly available at https://github.com/multimediaFor/NoiseCluSID.
Read the full abstract ▾
All authors Caihui Yan, Gang Cao, Huawei Tian, Zhen Li, Yuhang Zhai
Computer Vision
Incremental Online Scene Reconstruction by 3D Gaussian Triangulation
Yanjin Zhu, Shaofan Liu and Jianke Zhu
Incremental scene reconstruction is essential for real-world applications. Although 3D Gaussian Splatting shows strong potential, most existing approaches require offline conversion of the optimized Gaussians into an intermediate implicit field for explicit mesh extraction, which hinders seamless integration with downstream tasks. To address this limitation, we propose a novel online framework that incrementally reconstructs and updates high-fidelity explicit meshes by directly triangulating a dense
geometric Gaussian representation, which supports both high-quality rendering and incremental surface reconstruction. Moreover, we present a direct meshing
algorithm that efficiently extracts and updates the mesh from the Gaussian set. To ensure mesh accuracy, we enforce a plane-based pulling constraint that dynamically aligns 3D Gaussian primitives to the approximated local surface. Furthermore, our framework significantly reduces
memory and computational overhead during long-sequence processing by dynamically freezing fully optimized historical regions. Experiments on public
datasets demonstrate that our method outperforms conventional Gaussian-based methods on both rendering quality and reconstruction accuracy.
Read the full abstract ▾