Computer Vision · Cross-listed from cs.CL
NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection
Sanjeev Khatiwada
Out-of-context (OOC) misinformation pairs authentic images with misleading captions to construct false narratives without image manipulation, making detection a problem of
multimodal alignment rather than image forensics. Despite the prevalence and consequences of OOC misinformation in Nepal, no public
benchmark exists for Nepali. We introduce NepOOC, the first publicly available Nepali-dominant
multilingual OOC
benchmark, comprising 1,090 image-caption pairs (545 pristine, 545 OOC) annotated across five typologies (fabricated, miscaptioned, temporal mismatch, geographic mismatch, identity mismatch) with inter-annotator agreement kappa = 0.84. Systematic evaluation of five
multimodal architectures alongside text-only and image-only baselines reveals that caption semantics appear sufficient for strong performance at the current
dataset scale. A text-only mBERT model achieves 94.65+/-0.20% Macro-F1, statistically equivalent to the best
multimodal system (ResNet-50+mBERT, 94.65+/-0.20%; McNemar median p = 1.000, 0/5 seeds significant at alpha = 0.05). Image-only models perform near chance (33-50%), while training-size scaling suggests that
dataset expansion is a more direct path to progress than architectural sophistication or regional specialisation.
Read the full abstract ▾
Computer Vision
Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City
Adrian Cespedes, Marcelo Chincha, Dunant Cusipuma et al. (+3 more)
As Self-Driving Cars continue to expand internationally and use multi-modal systems such as VLMs as a cognitive backbone for their Action models; how well will these systems generalize in new settings, in particular out-of-distribution (OOD) edge-case scenarios in new geographies? In this paper, we study this open question by providing a full factorial analysis with human drivers of Lima, human drivers from New York City, and VLMs and showing them dashcam footage collected from Lima and New York City – prompting them with a variety of questions under a Visual Question Answering (VQA) paradigm. In particular, we pick these two cities as they are highly challenging driving locations where no Self-Driving Car company currently operates in, and ask questions that span 4 categories: Factual, Ratings, Counterfactual and
Reasoning. We find that Humans and VLMs diverge in their responses – though this is modulated by the type of questions asked, and that Humans answer similarly independent of where they are from (Lima/NYC). To our surprise, we did not find a strong difference in terms of answers (Humans or VLMs) that was modulated by geography, likely due to their high out-of-distribution nature. Our
dataset is available at: https://huggingface.co/
datasets/Artificio/robusto-2
Read the full abstract ▾
All authors Adrian Cespedes, Marcelo Chincha, Dunant Cusipuma, Victor Flores-Benites, David Ortega, Arturo Deza
Computer Vision
UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusion
Aryan Das, Koushik Biswas, Moloud Abdar et al. (+1 more)
We introduce UNITY, a Universal-to-Specialized adapter for efficient and scalable composite conditioning in
diffusion based image generation. Unlike prior methods that train separate adapters for each conditioning modality, UNITY jointly learns shared semantics across multiple conditioning types and subsequently specializes without modifying the underlying architecture. The proposed two stage training paradigm consists of a Universal Stage that captures cross modal representations across all conditioning modalities using half of the total training steps, followed by a Specialization Stage that refines modality specific features using the remaining training budget. At the core of UNITY are the Morphable
Attention Flow (MAF) Network and Morph Wrapper modules, which enable channel aware and spatially adaptive feature
alignment through learnable flow fields and
attention based fusion. This constant complexity formulation supports flexible operation under both single and composite conditioning settings while significantly reducing inference latency and
memory consumption. Extensive experiments across multiple
datasets demonstrate that UNITY achieves state of the art image fidelity while maintaining superior
memory efficiency. Code: https://github.com/arya-domain/UNITY
Read the full abstract ▾
All authors Aryan Das, Koushik Biswas, Moloud Abdar, Vinay Kumar Verma
Computer Vision
CogniRoute: Learning to Route Social Evidence in Omni-Modal Models
Yifan Shen, Pei Tian, Xinzhuo Li et al. (+8 more)
Omni-modal models can ingest
video, audio, and text, but unified access to multiple modalities does not guarantee that a model uses the right evidence. This gap is especially pronounced in social
video question answering, where the answer may hinge on a gesture, vocal tone, temporal cue, or mismatch between what is said and what is visually expressed. We introduce CogniRoute, a schema-guided Mixture-of-Experts framework for social omni
reasoning. CogniRoute uses a training-only cognitive schema that factorizes each example by cross-modal relation,
reasoning demand, and temporal scope, and aligns global routing signatures with this structure during supervised fine-tuning. We further introduce route-aware reinforcement learning, which jointly optimizes
token generation and expert allocation using rewards for answer correctness, modality-consistent
reasoning, and cognitive temporal grounding. To support training and evaluation, we construct OmniSocialBench, a diagnostic social
video QA resource with 118K structured training examples, grounded
reasoning traces, schema labels, temporal evidence spans, and a manually verified evaluation split. CogniRoute achieves 59.38% average accuracy on OmniSocialBench, improving over the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points, with the largest gains on questions requiring audio-visual coordination, conflict resolution, and temporally grounded social inference.
Read the full abstract ▾
All authors Yifan Shen, Pei Tian, Xinzhuo Li, Bowen Fang, Shujun Xia, Bingxuan Li, Ana Jojic, Wenming Ye, Xu Cao, James Matthew Rehg, Ismini Lourentzou
Computer Vision · Cross-listed from cs.CL
Scaling Diverse Language Generation for 3D Visual Grounding
Austin T. Wang, Dongchen Yang and Angel X. Chang
Developing robust models for 3D visual grounding (3DVG), the localization of entities in a 3D scene described in natural
language, is important for enabling
agents to correspond spatial
language with objects in the physical world. However, the lack of diverse descriptions at scale prevents models from generalizing beyond simple linguistic patterns. Recent such attempts lack diversity in the constraint types and
language used to ground objects. Captioning methods cannot precisely contrast objects, which is important for visual grounding. We therefore propose ViGiL3D++, a scalable, scene-agnostic method that generates diverse visual grounding queries by combining constraint sampling in scene graphs with the
language generation of LLMs. We show that it has greater diversity over existing scaled
datasets and improves model performance over several 3DVG
benchmarks but also illuminates outstanding limitations of VLMs.
Read the full abstract ▾
Computer Vision
ELDiff: When Evidential Learning Meets Text-to-Image Diffusion
Qingtao Pan, Kai Ye, Zhihao Dou et al. (+2 more)
In multi-object text-to-image (T2I)
diffusion, ensuring semantic consistency between textual
prompts and generated visual content is crucial for image synthesis. However, such consistency constraint is often underemphasized in the denoising process of
diffusion models. Although
token supervised
diffusion models can mitigate this issue by learning object-wise consistency between the image content and object
segmentation maps, it tends to suffer from the problems of
segmentation map bias and semantic overlap conflict, especially when involving multiple objects. In this paper, we propose ELDiff, a new evidential learning-supervised T2I
diffusion model, which leverages the advantages of uncertainty metric and conflict detection to enhance the fault tolerance of unreliable
segmentation maps and suppress semantic conflicts, strengthening object-wise consistency learning. Specifically, a pixel evidence loss is proposed to restrain overconfidence in unreliable labels through evidential regularization, and a
token conflict loss is designed to weaken the contradiction between semantics through optimizing a measured conflict factor. Extensive experiments show that our ELDiff outperforms existing training based and train-free based T2I
diffusion models on SD v1.4, SD v2.1, SDXL, SD v3.5, and Qwen-Image, without requiring additional inference-time manipulations. Notably, ELDiff can be seamlessly extended to the existing training pipeline of T2I
diffusion models. Code can be found at https://github.com/QingtaoPan/ELDiff.
Read the full abstract ▾
All authors Qingtao Pan, Kai Ye, Zhihao Dou, Bing Ji, Shuo Li
Computer Vision
GIM-ENDO: A Multimodal Endoscopic Image and Video Dataset for Gastric Intestinal Metaplasia Morphology and Pathology
Mojgan Forootan, Mahziar Setayeshfar, Ali Darvishi et al. (+2 more)
Gastric intestinal metaplasia (GIM) is a precursor lesion to gastric dysplasia and adenocarcinoma whose early detection is crucial for intervening in the carcinogenesis cascade. Artificial intelligence (AI) holds considerable promise for real-time endoscopic detection and characterization of GIM. However, development of reliable AI models has been constrained by the absence of publicly available, histopathologically validated
datasets that combine detailed endoscopic annotations, histological subtype (complete and incomplete), standardized grading systems, and normal mucosal patterns. GIM-ENDO was designed to fill this gap. The
dataset comprises demographic data, endoscopic findings, histopathological results, and H. pylori status acquired using the Olympus EVIS X1 system with white-light endoscopy (WLE) and image-enhanced endoscopy (IEE), including narrow-band imaging (NBI) and magnifying NBI (M-NBI), along with images and
video clips from 24 patients (22 GIM-positive, 2 normal controls). Annotations cover six primary IEE endoscopic signs – light blue crest (LBC), marginal turbid band (MTB), white opaque substance (WOS), TV pattern (Fusion), atrophy, and map-like erythema (MLE) – plus two additional endoscopic findings (AHP and GA) recorded where present. GIM subtypes (complete and incomplete) are annotated for all GIM-positive cases; OLGA and OLGIM staging are provided where complete histological sampling was available. The
dataset is publicly accessible at https://doi.org/10.5281/zenodo.20707267. For the latest updates and further information regarding this
dataset, readers are referred to the DataBioX website: https://databiox.com A short version of this work has been submitted to MICCAI 2026 Open Data Track.
Read the full abstract ▾
All authors Mojgan Forootan, Mahziar Setayeshfar, Ali Darvishi, Mohammad Tashakoripour, Hamidreza Bolhasani
Computer Vision
PROTON: Prototype-Based Test-Time Online OOD Detection for Medical VLMs
Abhijit Das, Nichula Wasalathilaka, Yifan Lu et al. (+4 more)
Medical
vision-
language models (VLMs) enable zero-shot clinical image classification, yet reliably detecting out-of-distribution (OOD) inputs at deployment remains an open problem. No static scoring method works across all shift types: Maximum Concept Matching (MCM) on FLAIR achieves 76.4% AUROC for far-OOD but only 42.4% for
covariate shifts such as ultra-wide-field fundus images, effectively random. We trace this to a structural mismatch:
covariate-shifted inputs are indistinguishable from in-distribution samples in softmax space, yet occupy distinct regions in the VLM
embedding space. To exploit this untapped signal, we propose
PROTON (PROtotype-based Test-time ONline OOD detection), a lightweight post-hoc module that maintains an online prototype bank from high-confidence test predictions and adaptively fuses prototype distance with MCM scoring via stream-level variance statistics, requiring no model modification, training data, or
prompt engineering. On the ophthalmology
benchmark FLAIR + FIVES,
PROTON improves MCM by +23.9 AUROC on
covariate shift, +8.8 on semantic shift, and +8.1 on far-OOD, making it the only zero-shot method to improve all three without hierarchical
prompts or labeled data. Code is available at https://github.com/GenMI-Lab/
PROTON, and the project page is available at https://genmi-lab.github.io/
PROTON.
Read the full abstract ▾
All authors Abhijit Das, Nichula Wasalathilaka, Yifan Lu, Adinath Dukre, Dwarikanath Mahapatra, Shadab Khan, Imran Razzak
Computer Vision
BELDE: Building a Large-scale Earth-observation Land-cover Dataset for Europe
Ümit Mert Çağlar and Alptekin Temizel
Earth observation imagery plays a critical role in environmental monitoring, urban planning, disaster assessment, and
climate analysis. While multi-spectral sensors are increasingly available, true-color (RGB) imagery remains widely used due to the power, cost, and deployment constraints of many satellite and aerial platforms. However, existing land-cover
segmentation datasets are often limited in geographic coverage, scale, or public accessibility. To bridge this gap, we introduce BELDE (Building a Large-scale Earth-observation Land-cover
Dataset for Europe), a publicly available
dataset tailored for RGB-based remote sensing semantic
segmentation. Constructed from Sentinel-2 true-color images and ESA WorldCover data annotations, BELDE contains 1,088,385 curated image-
segmentation map pairs spanning Europe with 7 land-cover classes at 10 m spatial resolution, making it one of the largest publicly available RGB land-cover
segmentation datasets for Earth observation. To facilitate cross-region generalization studies, we additionally introduce BELDE-K (16,607 pairs) covering the Republic of Korea and BELDE-CA-NV (88,155 pairs) covering California and Nevada in the United States. We establish baseline results using multiple semantic
segmentation architectures and evaluate both in-domain and cross-domain performance. Models trained on BELDE achieve an F1 score of 83.0% on the European test set, while performance decreases to 66.4% on BELDE-CA-NV and 58.3% on BELDE-K, highlighting the challenges posed by out-of-distribution geographic domain shift. By providing a continental-scale RGB
segmentation and evaluation
benchmark, BELDE supports the development of robust and transferable Earth observation models. The
dataset and
benchmark resources will be publicly released.
Read the full abstract ▾
Computer Vision
Go-with-the-Track: Video Compositing and Motion Control with Point Tracking
Koichi Namekata, Yash Kant, Zhizheng Liu et al. (+9 more)
Filmmaking demands precise motion control and reference image compositing – capabilities that existing methods treat separately. Point-track-conditioned image-to-
video models restrict content insertion to the first frame, while reference-to-
video models lack fine-grained spatial-temporal control over how reference content integrates across frames. We present Go-with-the-Track, which unifies both capabilities by jointly conditioning on multiple reference images and reference-anchored point-tracks – extending conventional point-tracks to explicitly establish correspondences between generated frames and reference images, thus enabling precise compositing and motion control throughout the
video. To achieve this, we introduce spatially-aware point-track
embeddings that encode the full sequence of point-track coordinates using a coordinate-wise MLP followed by temporal pooling. This representation captures the spatial characteristics of each point-track (serving as a unique identifier), while the
embedding similarity correlates directly with spatial proximity, enhancing the model's ability to distinguish and associate point-tracks. We inject these point-track
embeddings into a
video diffusion transformer via a lightweight adapter, resolving the pixel-to-patch resolution mismatch while avoiding the substantial motion detail loss inherent in naive point-track subsampling. We use a hybrid training strategy to train jointly on dynamic, static, and synthetic scene
video datasets to boost motion controllability. Experiments demonstrate that Go-with-the-Track achieves superior motion and reference control in a single model and enables new capabilities: multi-reference conditioned
video generation with point-track driven compositing, as well as camera control for both static and dynamic scenes. Project Page: https://eyeline-labs.github.io/Go-with-the-Track/
Read the full abstract ▾
All authors Koichi Namekata, Yash Kant, Zhizheng Liu, Ryan D Burgert, Yuancheng Xu, Kuan Heng Lin, Emmett Steven, Julien Philip, Li Ma, Andrea Vedaldi, Paul Debevec, Ning Yu
Computer Vision
Fine-grained Human Motion Understanding with Language Models
Thomas Markhorst, Zhi-Yi Lin, Jouh Yeong Chew et al. (+2 more)
Fine-grained motion understanding requires knowing not only which poses occur, but also when they occur. Existing motion-
language models either compress motion clips or concatenate pose features without retaining the timing of each pose, making action duration and temporal scale ambiguous under temporal subsampling. We introduce FiGMo, an LLM-based model that represents motion as timestamp-grounded skeletal poses, supporting
reasoning over action order and duration. A unified pose encoder applies this representation to both 2D and 3D skeletons, including poses detected from real
videos, while RGB can optionally provide contextual information. To reason over timestamped pose sequences, the model must understand both individual body configurations and how they change over time. As existing data provides limited pose-level grounding, we construct the required pose-to-motion supervision. Controlled experiments show that timestamps improve temporal localization and preserve physical timing under subsampling, while the constructed supervision drives the main performance gains. FiGMo achieves state-of-the-art results across motion question answering, dense motion captioning, and action recognition with detected poses. Our code is available at: https://github.com/Thomas-Markhorst/FiGMo
Read the full abstract ▾
All authors Thomas Markhorst, Zhi-Yi Lin, Jouh Yeong Chew, Jan van Gemert, Xucong Zhang
Computer Vision
Toward Parking Spot Occupancy Recognition: A Self-Supervised Approach
Luan Marko Kujavski, Rayson Laroca and Paulo Lisboa de Almeida
As urban areas expand, automatic monitoring of parking lots becomes essential for efficient and sustainable cities. This work proposes a self-supervised approach for parking spot occupancy recognition that requires no labeled samples from the target parking lot. Building upon a self-supervised transfer learning fine-tuning protocol, the proposed training strategy consists of two self-supervised stages: first on unlabeled generic data and then on unlabeled target-specific data, followed by supervised fine-tuning using only generic parking lot labels. We adopt SimCLR with a ResNet-50 encoder and evaluate the method under a leave-one-out cross-environment protocol on three public
datasets: PKLot, CNRPark-EXT, and PLds. We also introduce a two-stage deployment strategy in which a Strong General Model is initially deployed, followed by a Specialized Model that incorporates unlabeled images collected during the first N days of deployment in a self-supervised manner. Experimental results show that the Strong General Model alone outperforms supervised and self-supervised baselines, achieving an average accuracy of 97.2%, which further improves to 97.8% with the proposed two-stage strategy. These results demonstrate that self-supervised learning enables a scalable and labelefficient solution for real-world parking occupancy monitoring. Our trained models and source code are publicly available at https://github.com/LoanMaikon/Parking-Spot-Occupancy-Recognition.
Read the full abstract ▾
Computer Vision
FOCA: Future-Oriented Conditioning for Data-Efficient Vision-Language-Action Adaptation
Duc Minh Nguyen, Nghiem Tuong Diep, Binh Gia Nguyen et al. (+20 more)
Vision-
Language-Action (VLA) models enable general-purpose robotic control via large-scale
multimodal pretraining, yet their effectiveness under few-shot imitation learning remains limited. We conduct a systematic stress test of state-of-the-art VLA models and show that performance degrades sharply as demonstrations are reduced, revealing a key weakness of existing adaptation strategies. To address this, we introduce FOCA, a future-oriented conditioning framework for data-efficient VLA adaptation. FOCA combines explicit prediction of task-grounded future interaction
embeddings with implicit
alignment to future goal observations, enabling long-horizon
reasoning in latent space without pixel-level prediction. This formulation naturally supports action-free co-training with synthetic
videos from
video world models and can be interpreted as learning a future-conditioned value-like representation. Extensive experiments demonstrate FOCA achieves 95.7% success with 20 demonstrations on LIBERO, improves 7-12% on RoboCasa, and delivers up to 26% absolute gains on real
robots, establishing a new state of the art in few-shot VLA adaptation.
Read the full abstract ▾
All authors Duc Minh Nguyen, Nghiem Tuong Diep, Binh Gia Nguyen, Trong-Bao Ho, Doanh Le, Tan Q. Nguyen, Thien-Loc Ha, Nhiem Tran, Bao Thach, Nhat X. Tran, Tuan A. Tran, Artur Habuda, Philip Lund Møller, Tran Nguyen Le, Daniel Sonntag, Matthias Niepert, Khoa D. Doan, Vu Duong, Hung Ngo, Minh N. Vu, Duy M. H. Nguyen, An Thai Le, Ngo Anh Vien
Computer Vision
Stochastic Signed Distance Processes
Hiroki Sakuma and Masatoshi Okutomi
Multi-view surface reconstruction is a core problem in computer
vision. One prominent line of work represents the surface implicitly as a signed distance field (SDF), optimizing it based on the photometric loss between rendered and observed pixel colors. These approaches typically employ SDF-based volume rendering to obtain a differentiable relaxation of discontinuous visibility along rays, thereby reducing reliance on silhouette supervision. In this paper, we reformulate SDF-based volume rendering as probabilistic surface rendering, where each pixel color is modeled as a mixture distribution induced by the random first ray-surface intersection. To this end, we introduce
Stochastic Signed Distance Processes (SSDP), which model the SDF along each ray as a
stochastic process, inducing a first-passage-time distribution for each ray. We then derive the first-passage probability for each sampling interval based on
Bayesian filtering, together with its practical approximation for parallel rendering. We further show that NeuS, an existing SDF-based volume rendering method, arises as a special case of our formulation. Experiments on the DTU and MobileBrick
datasets demonstrate that our method outperforms baselines in both surface reconstruction and uncertainty quantification, supporting the effectiveness of our first-passage formulation. Our code is available at https://github.com/skmhrk1209/SSDP.
Read the full abstract ▾
Computer Vision
Translating Inference-Time Control to Radiology Vision-Language Models: Activation Steering for Pneumonia Classification on Chest X-rays
Eduardo Moreno Judice de Mattos Farina, Mateus A. Esmeraldo, Felipe Akio Matsuoka et al. (+2 more)
Inference-time engineering can alter model behavior without fine-tuning. However, its utility for improving diagnostic performance in medical
vision-
language models (VLMs) remains unclear. We aim to evaluate whether Contrastive Activation Addition (CAA) can improve pneumonia classification in chest radiograph VLMs without updating model weights. Three frozen chest radiograph VLMs (MedGemma-4B-IT, NV-Reason-CXR-3B, and CheXOne-3B) were evaluated on the public Kermany pneumonia test set. Classification was based on the logits of the
tokens Yes and No under a binary
prompt. Steering vectors included a 30-pair answer-bias control, a 30-pair pneumonia text contrast, and an image-conditioned contrast derived from 30 pneumonia and 30 normal development images. A deterministic 200-image development set was used for layer and scale selection (100 images) and threshold calibration (100 images). Performance was assessed using ROC-AUC, PR-AUC, F1 score, threshold analyses, reverse-vector controls, random-vector controls, and conditional bootstrap confidence intervals. Fixed-threshold F1 improvements were frequently observed but did not consistently indicate improved diagnostic performance. For MedGemma-4B-IT. NV-Reason-CXR-3B showed the strongest benefit: calibrated F1 improved from 0.7692 in the zero-shot setting to 0.8619 with pneumonia-text steering and to 0.8727 with image-conditioned steering. For CheXOne-3B, pneumonia-text steering increased calibrated F1 from 0.8528 to 0.8666, although the confidence interval crossed zero. On this public pneumonia
benchmark, CAA substantially altered prediction score distributions and operating characteristics without fine-tuning. Meaningful performance gains were observed in one of three evaluated VLMs, suggesting that activation steering may serve as a lightweight approach for adapting medical VLM behavior.
Read the full abstract ▾
All authors Eduardo Moreno Judice de Mattos Farina, Mateus A. Esmeraldo, Felipe Akio Matsuoka, Paulo Eduardo de Aguiar Kuriki, Felipe Campos Kitamura
Computer Vision
From Uncertainty to Stability and Fidelity: Guiding Sparse-View 3D Gaussian Splatting with Fisher Information
Junbao Zhou, Qingshan Xu, Yuan Zhou et al. (+7 more)
3D Gaussian Splatting (3DGS) has emerged as a promising technique for novel view synthesis. However, 3DGS requires dense input views to achieve high-quality rendering. In sparse-view scenarios, 3DGS often prones to overfitting, resulting in noticeable artifacts and degraded rendering quality. Previous methods explore to address this issue by introducing additional priors (e.g. depth priors) or integrating regularization techniques (e.g. Dropout). However, these methods are often applied without principled guidance. In particular, prior-based augmentation typically samples novel viewpoints randomly, while Dropout-based regularization randomly removes Gaussians. The compounded randomness introduces uncertainty and instability, limiting the fidelity of novel view synthesis. In this paper, we propose a novel method for sparse-view 3DGS that incorporates Fisher Information to quantitatively guide the utilization of
geometric priors and regularization. Specifically, our method comprises two key components: (1) Stereo augmentation with Fisher Information. By leveraging Fisher Information, we actively select most informative supporting views and use depth priors to curate reliable pseudo ground truths, which reduces randomness in augmentation and improves stability and rendering fidelity; (2) Uncertainty-aware regularization. We reduce the instability of Dropout-based regularization by using Fisher Information to quantitatively measure the uncertainty of each 3D Gaussian, and adaptively adjust the removal probability, leading to more stable and effective regularization. With these two components, our method effectively mitigates overfitting and improves the stability of optimization in sparse-view 3DGS, resulting in superior rendering fidelity. Extensive experiments show that our method achieves state-of-the-art performance in sparse-view novel view synthesis
benchmarks.
Read the full abstract ▾
All authors Junbao Zhou, Qingshan Xu, Yuan Zhou, Xiaolong Shen, Beier Zhu, Kesen Zhao, Yiming Zeng, Chen Bai, Cheng Lu, Hanwang Zhang
Computer Vision
NeoLoc-68: End-to-end 68-point neonatal facial landmark localisation in neonatal clinical environments
Abdullah Bin-Obaid, Maria M. Cobo, Rebeccah Slater et al. (+2 more)
Facial landmark localisation is a prerequisite for developing automated, non-contact neonatal pain assessment methods. Clinicians use pain scales to judge the severity of pain, many of which rely on facial expression. However, facial landmark detectors trained on adult faces perform poorly in neonatal clinical environments due to frequent occlusions caused by medical equipment, varied head poses, and challenging imaging conditions, including motion blur triggered by sudden pain-related movements. We propose an end-to-end facial landmark detector capable of predicting 68 landmarks on neonatal faces in clinical environments. We combined 37,459 single-face images from 11 public
datasets, standardised to 68-point markup, with 1,123 manually annotated frames from a neonatal research
dataset (totalling over 76,000 landmarks). A YOLO-based keypoint model was adapted to regress the facial landmarks, initialised with weights from a pretrained neonatal face detector. On public
datasets, our proposed model achieved state-of-the-art performance: Normalised Mean Error (NME) = 5.37, Failure Rate (FR) = 12.5%, Area Under the Cumulative Error Curve (AUC) at AUC0.08 = 38.00% and AUC0.1 = 48.70%. On the clinical neonatal test set, before fine-tuning, the model achieved the lowest Detection Failure Rate (DFR) = 5.3% among all baselines and showed strong generalisation. After fine-tuning, performance improved further to NME = 6.36, FR = 22.30%, DFR = 1.77%, AUC0.08 = 29.24% and AUC0.1 = 40.25%. To the best of our knowledge, this represents the first end-to-end 68-point neonatal facial landmark detection model. With further
dataset expansion and refinement, it could support downstream tasks in neonatal health monitoring and pain-related facial analysis.
Read the full abstract ▾
All authors Abdullah Bin-Obaid, Maria M. Cobo, Rebeccah Slater, Lionel Tarassenko, Mauricio Villarroel
Computer Vision
GroundShot: Visually Consistent Multi-Shot Long Video Generation via Entity-Grounded Shot Scheduling
Yixuan Lai, Tianjia Shao, Weijia Dou et al. (+2 more)
Generating visually consistent multi-shot
videos remains an open challenge. As
videos span more shots, inconsistencies can accumulate across shots, causing entities that reappear across shots – characters, objects, and locations – to drift away from how they first appear. We observe that viewers judge consistency by comparing each later appearance of an entity with its first clear appearance; the visual quality of this initial appearance sets the consistency ceiling for all that follows. Motivated by this, we present GroundShot, a training-free, model-agnostic agentic framework for entity-grounded multi-shot generation. GroundShot builds an entity-level visual
memory online from accepted generated shots: it schedules shots' generation order by their expected usefulness as entity references, grounds entities from generated
videos, verifies their reliability before adding them to
memory, and retrieves suitable entity references from
memory before each shot is generated. To evaluate this entity-centered view of consistency, we further introduce GroundBench, a diagnostic
benchmark that measures consistency at the entity level while isolating controlled challenge dimensions. Experiments show that GroundShot improves multi-shot consistency over existing methods while requiring no additional training or model modification.
Read the full abstract ▾
All authors Yixuan Lai, Tianjia Shao, Weijia Dou, Siyu Zhu, Jingdong Wang
Computer Vision · Cross-listed from cs.RO
World Action Models: A Survey
Qiuhong Shen, Shihua Zhang, Yue Liao et al. (+5 more)
World Action Models (WAMs) are
embodied predictive-action models that make a forecast of the future available to action. Recent WAMs repurpose large
video generation models, and a parallel line relies on
language or
vision-
language backbones without a
video-generation core. This rapid expansion has blurred the boundary among broad world models,
video generation models, action-grounded
video world models,
Vision-
Language-Action policies, and WAMs. This survey gives the field a common account. It first clarifies these boundaries, then organizes existing works through two complementary views. The first view asks what each method is required to generate, spanning rendered futures, latent futures, and
video-generation-free action
reasoning. The second view decomposes each method by predictive substrate, backbone, action coupling, and deployment regime. This anatomy supports a unified discussion of interactability, causality, persistence, physical plausibility, and generalization, followed by data, evaluation, and open challenges. Across these axes, a consistent design pattern emerges: WAMs are not simply
video generators with action heads, but predictive-action methods whose design choices trade representational richness against compute,
memory, latency, and action-label cost. The field is moving toward methods that generate less of the future while preserving what control requires. The survey homepage is available at https://world-action-models.github.io/.
Read the full abstract ▾
All authors Qiuhong Shen, Shihua Zhang, Yue Liao, Qi Li, Zhenxiong Tan, Shizun Wang, Shuicheng Yan, Xinchao Wang
Computer Vision
TriMotion: Modality-Agnostic Camera Control for Video Generation
Seunghyun Shin, Jifei Song, Wooseok Jeon et al. (+2 more)
Camera motion control is essential for directing viewpoint changes in generative systems. However, existing methods typically condition the generation process on a single specific modality, such as explicit pose trajectories or reference
videos, limiting their ability to support heterogeneous user inputs. To address this limitation, we present TriMotion, a modality-agnostic framework for camera-controlled
video generation that maps
video, pose, and text inputs, describing the same camera trajectory into a shared motion
embedding space. Learning such a space requires synchronized supervision across modalities. Therefore, we build the Motion Triplet
Dataset by extending a Multi-Cam
Video Dataset with geometry-grounded motion descriptions derived from camera extrinsics. We further introduce a latent motion consistency objective that leverages the motion
embedding space to encourage the generated
video to follow the target camera trajectory directly in latent space, avoiding the cost of pixel-space decoding. Extensive experiments show that TriMotion generates high-quality
videos that accurately follow the target camera trajectories across all three modalities. Beyond standard generation, the shared motion
embedding space also enables flexible applications such as sequential motion composition and cross-modal motion interpolation.
Read the full abstract ▾
All authors Seunghyun Shin, Jifei Song, Wooseok Jeon, Hae-Gon Jeon, Jiankang Deng
Computer Vision
UniSLAD: A Unified Framework for Structural and Logical Industrial Visual Anomaly Detection
Changyi Li, Chao Yang, Yu Xiao et al. (+1 more)
Visual anomaly detection is a fundamental task in industrial automation. While existing approaches have achieved notable progress in identifying structural defects, the detection of logical anomalies remains relatively underexplored. In practice, structural and logical anomalies frequently co-occur in industrial workflows. Therefore, a solution capable of detecting both structural and logical anomalies is crucial for advancing comprehensive anomaly detection research. To address this limitation, we propose a unified framework, termed UniSLAD, which jointly addresses logical and structural anomalies without additional training, enabling a practical solution for dynamic industrial environments. First, we introduce a dual-feature extractor that synergistically integrates a Convolutional Neural Network (CNN) backbone for local texture perception with a
Transformer backbone for global contextual
reasoning, yielding richer and more comprehensive representations. Building on this foundation, we design dual-granularity feature representation modules. At the patch level,
memory banks enhanced by the Mahalanobis Transform (MT) preserve representative features and support more discriminative anomaly scoring. At the image level, distribution maps are aggregated using Lower-Upper Mean (LUM) and Power Mean Pooling (PMP), yielding a more robust global representation than conventional average pooling. Extensive experiments on the two industrial
benchmarks demonstrate that UniSLAD achieves competitive performance in comprehensive anomaly detection, achieving 99.4% and 93.1%, respectively. Furthermore, ablation studies verify the individual contributions and effectiveness of each proposed component.
Read the full abstract ▾
All authors Changyi Li, Chao Yang, Yu Xiao, Kari Tammi
Computer Vision
One Image is All You Need: Agentic One-Shot Image Generation via Text-Based World Models for Long-Tail Spatial Perception
Keqin Zeng, Shuting Su, Shihao Lin et al. (+2 more)
Reliable spatial decision automation, such as autonomous driving and maritime surveillance, critically depends on robust visual perception. However, real-world spatiotemporal data exhibits severe heterogeneity, often manifesting as extreme long-tail distributions for safety-critical scenarios. This data scarcity induces
dataset shift that degrades detection performance and pose safety risks. While synthetic data generation offers a potential solution, existing generative approaches, such as
diffusion models and Generative
Adversarial Networks (GANs), often lack explicit spatial grounding and structural constraints, resulting in spatial and physical inconsistencies in generated scenes. To address these challenges, we introduce WMGen-v1, an agentic text-based world model framework for long-tail spatial data generation. WMGen-v1 employs a Large
Vision-
Language Model (LVLM) to construct a structured scene representation from a single reference image, while a Large
Language Model (LLM) performs guidance-based scene expansion under physical plausibility and commonsense constraints. Subsequently, conditioned on the structured semantic representations produced by this
reasoning process, a
diffusion model generates diverse and physically grounded long-tail training data. Experiments on internal industrial
datasets, ROADWork, and LaRS
benchmarks demonstrate that WMGen-v1 outperforms baseline approaches. Notably, detectors trained solely on WMGen-v1 synthetic data approach real-only performance on aggregate
dataset-level metrics, highlighting its potential to alleviate long-tail data scarcity for downstream spatial perception.
Read the full abstract ▾
All authors Keqin Zeng, Shuting Su, Shihao Lin, Ziyue Li, Rui Zhao
Computer Vision
Mirage: a Clean-Label Backdoor against LiDAR 3D Object Detection
Ziba Parsons and Ang Li
Deep neural network-based
LiDAR 3D object detection serves as a critical perception component in safety-critical autonomous systems. However, recent studies have revealed its vulnerability to
backdoor attacks. Existing attacks typically require white-box access or label modification and focus on
geometric attacks such as object disappearance or bounding-box manipulation. In this paper, we present Mirage, a black-box and clean-label
backdoor attack against deep neural network-based
LiDAR 3DOD. Mirage injects a small number of label-consistent poisoning samples into the training set, causing the model to learn a malicious association between a trigger pattern and an attacker-chosen target class while preserving normal training semantics. As a result, the compromised model behaves normally on benign inputs yet systematically misclassifies triggered objects as the target class during deployment. We evaluate Mirage on multiple state-of-the-art
LiDAR 3DOD models and
benchmark datasets. Experimental results show that Mirage achieves a 73% misclassification success rate with a poisoning rate of only 0.5%, while maintaining detection performance close to that of benign models.
Read the full abstract ▾
Computer Vision
JanusMesh: Fast and Zero-Shot 3D Visual Illusion Generation via Cross-Space Denoising
Siang-Ling Zhang, Huai-Hsun Cheng, Tsung-Ju Yang et al. (+1 more)
Creating 3D visual illusions, a single 3D mesh that reveals entirely different semantics from various viewing angles, is a fascinating but tough challenge. Existing optimization-based methods are slow and can produce oversaturated colors. In contrast, naive stitching approaches fail to produce geometrically coherent objects. This results in visible unnatural seams and semantic leaks. In this paper, we present a fast and training-free framework for generating text-driven 3D visual illusions. Our approach decouples the generation into two stages. First, we propose a cross-space dual-branch denoising process. This process dynamically decodes 3D latents into voxel space for CLIP-guided orientation
alignment and Signed Distance Field (SDF) blending, which ensures seamless
geometric fusion. Second, we introduce a view-conditioned texture synthesis module that projects and aggregates view-specific 2D
diffusion priors onto the fused geometry. Extensive experiments demonstrate that our method generates highly realistic, dual-semantic 3D illusions in just 3-5 minutes. It significantly outperforms existing methods in
geometric integrity, semantic recognizability, and efficiency. Project page: https://siang1105.github.io/JanusMesh.github.io/
Read the full abstract ▾
All authors Siang-Ling Zhang, Huai-Hsun Cheng, Tsung-Ju Yang, Yu-Lun Liu
Computer Vision
TimeProVe: Propose, then Verify for Efficient Long Video Temporal Reasoning in Activities of Daily Living
Arkaprava Sinha, Dominick Reilly, Siddharth Krishnan et al. (+2 more)
Long
Video Question Answering (LVQA) requires identifying sparse, query-relevant evidence within hours-long untrimmed
videos. Existing approaches either process
videos densely with large
vision-
language models (VLMs), incurring prohibitive computational cost, or rely on sparse caption-based
reasoning, which often misses temporally localized and motion-centric evidence. We introduce TimeProVe, a cost-efficient hybrid framework for temporally grounded
reasoning in long
videos. TimeProVe first employs lightweight modules to generate action-grounded answer–evidence hypotheses and subsequently invokes an expensive VLM only for targeted verification. The core of our framework lies in the Action-based Candidate Evidence (ACE) module, which converts temporally localized actions into query-conditioned candidate answers and supporting evidence windows through lightweight LLM
reasoning. We further introduce OpenTSUBench (OTB), an open-ended
benchmark designed to evaluate temporally grounded
reasoning in real-world Activities of Daily Living (ADL) scenarios. Experiments show that TimeProVe outperforms the strongest baseline on OTB by 7.3%, while reducing VLM calls by 75% and inference cost by 93%. Furthermore, without explicit temporal grounding training, TimeProVe achieves competitive performance on Charades-STA, and reaches state-of-the-art results when enhanced with grounding VLMs.
Read the full abstract ▾
All authors Arkaprava Sinha, Dominick Reilly, Siddharth Krishnan, Hieu Le, Srijan Das
Computer Vision
UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning
Wenhao Chi, Arkaprava Sinha, Dominick Reilly et al. (+2 more)
Egocentric video understanding is inherently limited by the narrow perspective of wearable cameras: a single viewpoint, a single modality, a single model cannot capture the full richness of human action. We argue that a truly expressive
egocentric representation must subsume complementary knowledge across viewpoints, modalities, and foundation model representations, yet remain deployable from
egocentric video alone. To this end, we introduce a hierarchical multi-teacher
distillation framework that produces UNIEGO, a unified
egocentric encoder trained with nine teachers spanning ego-exo viewpoints, RGB, depth, and skeleton modalities, and four foundation models. Rather than distilling directly from heterogeneous teachers whose incompatible architectures and feature geometries induce conflicting gradients, our framework interposes a layer of representation-specific Proxy models that translate diverse teacher knowledge into a homogeneous
egocentric space. A second
distillation stage, Selective Proxy
Distillation (SPD), then adaptively selects, for each training sample, the subset of proxies that are both correct and confident, distilling exclusively from reliable supervision and suppressing erroneous signals. SPD is further stabilized by initializing UNIEGO as a learned convex combination of proxy parameters, placing the unified model in a well-conditioned region of the loss landscape before
distillation begins. UNIEGO achieves state-of-the-art performance across three
egocentric video understanding tasks - action recognition,
video retrieval, and action
segmentation on three challenging ego-exo
benchmarks, outperforming naive multi-teacher
distillation baselines and demonstrating that structured, proxy-mediated knowledge transfer yields richer and more discriminative
egocentric representations.
Read the full abstract ▾
All authors Wenhao Chi, Arkaprava Sinha, Dominick Reilly, Hieu Le, Srijan Das
Computer Vision
Thinking in Boxes: 3D Editing in Real Images Made Easy
Pradhaan S Bhat, Naveen Chandra R, Rishubh Parihar et al. (+4 more)
Text and 2D-conditioning interfaces provide weak, ambiguous control over spatial transformations in image editing – particularly under large object motions and camera changes. Prior work has used 3D primitives such as boxes, but only as loose conditioning signals indicating approximate object location rather than specifying the transformation. We instead use 3D boxes as structured specifications: the user provides the input and output boxes of the edit, casting editing as a well-posed geometry problem. This “thinking in boxes” interface, where each box face is color-coded to convey 3D orientation, gives precise control over translation, rotation, scaling, and viewpoint changes in real images while preserving scene and object identity, and recovering previously unseen object regions. To ground transformations in scene appearance, we introduce a depth-aligned planar floor as a global reference frame, shaded with depth-aware cues. Conditioned on this structure, an image generator produces consistent results under large transformations. Trained in two stages – on synthetic multi-object scenes and a small set of real-world
videos from Objectron – the system generalizes to complex, in-the-wild real images. Our method operates directly on real photographs and substantially outperforms recent state-of-the-art methods on large 3D edits.
Read the full abstract ▾
All authors Pradhaan S Bhat, Naveen Chandra R, Rishubh Parihar, Vaibhav Vavilala, R. Venkatesh Babu, D.A. Forsyth, Anand Bhattad
Computer Vision · Cross-listed from cs.LG
Przemyslaw Musialski
We place the
attention token on the group: a
token is an element $g_i$ of a
matrix Lie group $G$ – a bare transformation, with no feature payload and no external action $\rho(g)$ carrying it. To our knowledge this is the first
attention construction whose
tokens are bare
matrix Lie group elements: their score is the closed-form
algebra norm of the relative pose rather than a learned kernel, and it reaches the affine full-frame groups that every irrep- or surjective-exp-based method must exclude. We call it Lie-
Algebra Attention. Once
tokens are group elements, the rest follows with none of the usual representation-theoretic machinery. The relative geometry of a pair is canonical, $g_i^{-1} g_j$, so the pairwise invariant $w_{ij} = \log(g_i^{-1} g_j)$ is intrinsic rather than designed; equivariance under the diagonal $G$-action is tautological, and the cocycle condition holds automatically. The
attention score is the negative squared
algebra norm, $s_{ij} = -\|\log(g_i^{-1} g_j)\|_\lambda^2/\tau$: the canonical proximity kernel under a block-weighted Frobenius inner product, with no irreducible representations, spherical harmonics, Clebsch-Gordan products, or learned kernel. The construction applies to any
matrix Lie group on a chosen logarithm chart containing the relative poses, including the non-compact non-abelian affine groups with scale and shear that no vector-
token attention method reaches: neither the irrep tradition nor surjective-exp methods. Three sequence-completion experiments, on SE(2), SO(3), and Aff(2), bear this out: the closed-form score matches a learned MLP kernel on the same invariant and outperforms it on SE(2), using 50 to 80x fewer score parameters, while a vector-
token baseline breaks invariance by five to twelve orders of magnitude.
Read the full abstract ▾
Computer Vision
Current World Models Lack a Persistent State Core
Jinpeng Lu, Dexu Zhu, Haoyuan Shi et al. (+8 more)
World models are increasingly regarded as a decisive step toward artificial general intelligence, yet modeling the physical world demands more than rendering convincing frames on demand: it requires an internal world state that keeps evolving over time, decoupled from observation, so that objects endure and events run to their conclusions whether or not a camera is watching, much as the moon holds to its orbit when no one is looking. This requirement is a blind spot of existing
benchmarks, which reward surface properties such as fidelity, motion, and camera controllability while never asking whether a generated world keeps evolving once it is unobserved. We introduce WRBench, the first systematic diagnostic
benchmark that treats camera motion as an intervention on observability and resolves evaluation into a human-calibrated chain that asks whether the camera executes the requested interaction, whether the scene stays continuous and identifiable while in view, and whether a returning target remains consistent with the event that was set in motion. Across 9{,}600
videos from 23 models spanning four control paradigms, one finding proves stubborn: current systems maintain the observed world as a tracking shot, resuming a returning target in the state at which it was abandoned rather than advancing the event while it went unseen. Because this failure recurs across control paradigms, model families, and increments of scale, robust world-state
evolution does not follow from cleaner imagery, tighter control, richer
geometric priors, or sheer parameter count We therefore argue that the stability of the physical state kernel and the consistency of worldlines under viewpoint intervention should become first-class objectives of world-model design, so that a world model captures how the world will unfold rather than how the next frame appears.
Read the full abstract ▾
All authors Jinpeng Lu, Dexu Zhu, Haoyuan Shi, Linghan Cai, Guo Tang, Yinda Chen, Jie Cao, Duyu Tang, Yi Zhang, Yong Dai, Xiaozhu Ju