Computer Vision
OctoSense: Self-Supervised Learning for Multimodal Robot Perception
Anthony Bisulco, Jeremy Wang, Kostas Daniilidis et al. (+2 more)
We present OctoSense, an open-source sensor platform with stereo RGB and event cameras,
LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car, and joint angles for a quadruped
robot). The eponymous OctoSense
dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors. We demonstrate multi-modal self-supervised learning using such real-world robotics data, where sensors have different representations, frequencies, latencies and noise. Our approach, a "late-fusion" masked autoencoder, (i) uses modality-specific tokenizers to account for different spatiotemporal characteristics of these sensors, and (ii) caches modality-specific
tokens at inference time to process new measurements as they come. This architecture (i) is fast (6.68 ms and 112 ms on NVIDIA 5090 and Orin NX respectively, to compute the representation), (ii) performs better than existing image-only foundation models on tasks such as estimation of optical flow, depth, semantic
segmentation, and ego-motion (translation, rotation, and steering angle), and (iii) predicts robustly at nighttime or in situations where sensory data is degraded. See our project page for links to the
dataset, code, and supplementary
videos: https://abisulco.com/octosense/.
Read the full abstract ▾
All authors Anthony Bisulco, Jeremy Wang, Kostas Daniilidis, Randall Balestriero, Pratik Chaudhari
Computer Vision
ViQ: Text-Aligned Visual Quantized Representations at Any Resolution
Xumin Yu, Zuyan Liu, Zhenyu Yang et al. (+5 more)
A unified representation for text and
vision is a natural pursuit, as it enables simpler
multimodal modeling and more efficient training. However, representing images as discrete signals in the same way as text inevitably introduces severe information loss. Existing work struggles to balance low-level details and high-level semantics in discrete representations: reconstruction-oriented representations often lack semantic information, whereas semantically stronger features typically suffer from severe loss of detail. We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations while supporting inputs at native resolutions, thereby enabling it to serve as a unified and general discrete representation for arbitrary visual inputs. Our approach structures quantization learning into two stages: text-aligned pre-training and feature discretization. With text-aligned pre-training, we enhance the visual encoder semantic-rich supervision from the pretrained
language model and enable it to process native-resolution visual inputs. During discretization, we propose a proximal representation learning strategy to progressively compact the feature space, along with a position-aware head-wise quantization mechanism that enables flexible processing of arbitrary resolutions. Extensive experiments on
multimodal tasks demonstrate that ViQ achieves competitive performance compared to state-of-the-art
multimodal vision encoders with continuous and high-dimensional visual features, while maintaining high precision in low-level reconstruction. We also show that
multimodal training with visual quantized representations largely improves efficiency, yielding up to 20%-70% acceleration with different base LLMs and training recipes.
Read the full abstract ▾
All authors Xumin Yu, Zuyan Liu, Zhenyu Yang, Yuhao Dong, Shengsheng Qian, Jiwen Lu, Han Hu, Yongming Rao
Computer Vision
See & Sniff: Learning Visuo-Olfactory Representations
Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu et al. (+2 more)
While modern
multimodal models integrate
vision with
language, audio, or touch, olfaction remains largely unexplored due to the lack of paired visuo-olfactory data. We introduce SmellNet-V, a scalable visuo-olfactory
dataset built on the insight that odor identity is largely invariant to visual transformations within a semantic category. This allows us to synthetically pair smell-only samples with semantically aligned in-the-wild web images, converting a unimodal olfactory
dataset into a cross-modal
benchmark without costly co-collection. Building on this
dataset, we propose See & Sniff, a self-supervised framework that learns joint visuo-olfactory representations via dense local
alignment and naturally produces smell saliency maps for spatial grounding of odor sources. We further introduce pixel-level smell localization task and a
benchmark for evaluation. Our method surpasses smell-only baselines by 7% in smell classification from smell alone and generalizes to cross-modal
retrieval and smell localization, establishing visuo-olfactory learning as a new direction in
multimodal perception.
Read the full abstract ▾
All authors Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu, Joon Son Chung, Arda Senocak
Computer Vision
Sculpting NeRF Geometry: Human-Preference Fine-Tuning of a 3D-Aware Face GAN
Archer Moore, Mingming Gong and Liam Hodgkinson
Reinforcement learning from human feedback (RLHF) for 3D generation is now established across a number of works, but most existing pipelines optimise explicit surface representations, often by converting radiance fields into meshes and training heavily on surface-supervised data. We instead fine-tune a pretrained 3D-aware generative model directly from a learned reward over radiance-field density ($\sigma$) values, with no externally supplied mesh or shape prior. The reward model requires no pretraining, trains easily on a small set of preference samples, and yields robust improvement in 3D geometry. Working on an unconditional 3D-aware face GAN (EG3D), our reward reads the continuous 3D density field of the neural radiance field (NeRF) directly and supplies a geometry-only learning signal, requiring neither text conditioning, mesh extraction, nor multi-view rendering. A density-consistency constraint keeps the 2D appearance qualitatively similar while the geometry is reshaped, at a measurable but bounded distributional cost (FID-50k rises from 4.09 to 6.66): the fine-tuned generator, trained from the preferences of a single annotator as a proof of concept, produces face geometries preferred by users in 74.4% of pairwise comparisons.
Read the full abstract ▾
Computer Vision
Exact and Deterministic Patch Descriptor Retrieval via Hierarchical Normalization
Koichi Sato
We present a patch descriptor
retrieval method that returns the exact nearest neighbour – provably identical to exhaustive full-vector search – while evaluating only a small fraction of the database, and does so deterministically: the same (database, query) pair always produces the same result, independent of run order, thread count, or hardware. This contrasts with approximate nearest-neighbour (ANN) approaches such as HNSW and IVF-PQ, which trade exactness for speed and may return different results across runs. The enabling mechanism is Hierarchical Normalization (HN): a normalisation scheme that splits the pre-normalisation feature vector into a K-dim major component (norm sqrt(1-alpha)) and a (128-K)-dim minor component (norm sqrt(alpha)). Since the minor inner product is bounded by alpha (Cauchy-Schwarz on the prescribed norms), the major similarity plus alpha is an admissible upper bound on the full similarity: the search scans the K-dim major component for all entries, then applies full 128-dim evaluation only to entries that cannot be pruned – a provably exact branch-and-bound scan. We train HN-modified HardNet on the notredame split of the UBC patch
dataset and evaluate on trevi and halfdome. With a cache-optimised Structure-of-Arrays layout and K=8, alpha=1/32, the search achieves 13.7x (trevi) / 12.7x (halfdome) speed-up over brute-force 128-dim search, with only 0.4% of entries requiring full evaluation. At K=16, alpha=1/8, FPR@95 rises from 0.0062 to 0.0064 on trevi at 7.2x speed-up, with 98.8% of entries bypassing full evaluation.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
EO-WM: A Physically Informed World Model for Probabilistic Earth Observation Forecasting
Junwei Luo, Shuai Yuan, Zhenya Yang et al. (+3 more)
Earth Observation (EO)
forecasting aims to predict future Earth surface dynamics from satellite observations under changing meteorological conditions. In this paper, we view this task as a partially observed, weather-driven world modeling problem, in which weather acts as a conditioning signal, while
forecasting remains uncertain due to sparse observations and unobserved land-surface states. However, existing methods do not fully capture this setting: deterministic models collapse uncertainty into a single future prediction, while
diffusion-based methods typically treat weather variables as undifferentiated conditioning signals, and existing
benchmarks focus mainly on reconstruction accuracy rather than whether forecasts respond correctly to changed weather forcing.We introduce EO-WM, a
video diffusion transformer for multispectral EO
forecasting. EO-WM incorporates a physically informed conditioning framework that represents meteorological forcing through a climatological baseline, weather anomalies, and cumulative physical stress signals. Specifically, it separates baseline and anomaly through distinct conditioning pathways, and accumulates anomalous forcing over time to capture sustained heat and drought stress. To evaluate weather-response behavior beyond standard metrics, we introduce two diagnostic
benchmarks: an Extreme Summer
Benchmark for severity-aware prediction of vegetation degradation under extreme weather, and a Seasonal Matched-Pair
Benchmark for testing response fidelity under changed weather forcing. Experiments show that EO-WM reduces the error in predicted Normalized Difference Vegetation Index (NDVI) decline amplitude by a relative 5.63% and improves directional hit rate by a relative 7.80%, while remaining competitive on standard pixel-level metrics. The
benchmarks and model will be made open-source at https://github.com/Luo-Z13/EO-WM.
Read the full abstract ▾
All authors Junwei Luo, Shuai Yuan, Zhenya Yang, Yansheng Li, Zhe Liu, Hengshuang Zhao
Computer Vision
CORTEX: A Structured Reasoning Benchmark for Trustworthy 3D Chest CT MLLMs
Hashmat Shadab Malik, Anees Ur Rehman Hashmi, Numan Saeed et al. (+3 more)
Reasoning in
multimodal large
language models (MLLMs) has shown strong promise in medical imaging. However, this
reasoning is usually free-form text judged only by its final answer, making it hard to interpret and verify, especially in 3D radiology, where a diagnosis should be traceable to evidence in the scan. Existing chest CT question-answering
datasets compound this by reducing expert radiology reports to answer-only pairs, dropping the
reasoning that links findings to conclusions and omitting the patient history clinicians rely on. As a result,
reasoning-capable 3D chest CT MLLMs remain out of reach, as neither the structured supervision needed to train them nor the protocol needed to verify their
reasoning yet exists. We introduce CORTEX (Clinically Organized
Reasoning and sTructured EXplanation), a structured
reasoning benchmark for 3D chest CT. For each question, CORTEX restores the missing
reasoning as a four-stage diagnostic trace mirroring a radiologist's workflow: task understanding, visual observation, diagnostic
reasoning, and answer synthesis. We generate these traces using frontier large
language models with broad medical and general-domain knowledge, then filter and verify them with a stage-level evaluation protocol combining automated rubric scoring with expert radiologist review. Crucially, both the
reasoning structure and evaluation rubrics are designed in close collaboration with clinicians. Built on CT-RATE, a large, publicly available chest CT
dataset without
reasoning annotations, CORTEX comprises 76,177 validated
reasoning traces across open-ended VQA, closed-ended VQA, and report generation, providing both the structured supervision and the stage-level evaluation protocol needed to build and evaluate trustworthy
reasoning models for 3D chest CT. Our
dataset and evaluation code is available at https://huggingface.co/
datasets/aneesurhashmi/cortex
Read the full abstract ▾
All authors Hashmat Shadab Malik, Anees Ur Rehman Hashmi, Numan Saeed, Muzammal Naseer, Salman Khan, Christoph Lippert
Computer Vision · Cross-listed from cs.CY
From Celebrities to Anyone: Characterizing AI Nudification Content, Technology, and Community Dynamics on 4chan
Chi Cui, Yixin Wu and Yang Zhang
AI nudification uses generative models to create synthetic non-consensual sexually explicit imagery (SNEACI) of real individuals. Prior work has examined dedicated nudification platforms and model repositories, finding that most targets are female celebrities. However, the anonymous content community, where SNEACI is actively requested, generated, and exchanged, remains unexplored. In this work, we present a large-scale study of AI nudification in the wild, identifying 24,105 SNEACI items. We find a significant shift in target demographics: non-celebrity individuals now account for 55.8% of targets, compared to only 4.7% in prior studies, indicating that AI nudification has expanded from targeting public figures to increasingly harming individuals within users' own social circles. Meanwhile, open-source models dominate production, with Stable
Diffusion family generating 42.7% of images and Wan generating 66.5% of
videos, all driven by thousands of shared fine-tuned models and accessible tutorials. Yet the ecosystem runs on a small cohort of active producers, with the most prolific producing 780 items, drives community engagement, shapes target demographics, and disseminates technical knowledge that lowers barriers for new producers. Our work provides an empirical understanding of how AI nudification operates in the wild, revealing the mechanisms that sustain this ecosystem and highlighting the urgent need for interventions in platform governance, technical safeguards, and affected individual protection.
Read the full abstract ▾
Computer Vision
SatSplatDiff: Geometry-preserving generative refinement for high-fidelity satellite Gaussian Splatting
Jiyong Kim, Shuang Song and Rongjun Qin
Gaussian Splatting has been recently explored for satellite 3D reconstruction, demonstrating flexibility and efficiency in representing radiometrically diverse satellite scenes. However, the limited top viewpoint of satellite imagery results in insufficient supervision on building facades, leaving surface holes and degraded visual fidelity. Generative refinement, which leverages pretrained generative priors to iteratively refine and update the rendered images used as supervision targets, has recently been investigated to improve the visual fidelity of Gaussian-rendered images. However, since these models refine each view independently, the resulting images can generate
hallucinations and break photo-consistency, leading to
geometric degradation. To address these limitations, we propose SatSplatDiff, which aims to minimize
geometric degradation prevalent in generative refinement. Building on photogrammetric DSM initialization and 2DGS-based shadow casting established in our prior work SatSplat, we first introduce
monocular depth supervision and multi-scale
geometric refinement to establish a geometrically accurate and well-regularized surface representation. We then apply shadow-guided generative refinement, where geometrically calculated shadow maps guide the Gaussians to maintain consistency with the underlying geometry, improving visual fidelity while reducing
geometric degradation. Extensive evaluations on the IARPA2016 and DFC2019
datasets demonstrate state-of-the-art performance, reducing
geometric MAE by up to 18% and improving visual fidelity (FID-CLIP) by 28-45% over existing baselines. Our method delivers up to 5x resolution enhancement with minimal
hallucination and sensor-consistent appearance, demonstrating seamless cross-tile consistency and strong scalability for large-scale reconstruction. Source code is available at https://github.com/GDAOSU/SatSplatDiff
Read the full abstract ▾
Computer Vision
LISA: Likelihood Score Alignment for Visual-condition Controllable Generation
Yanghao Wang, Hongxu Chen, Jiazhen Liu et al. (+4 more)
The prevalent dual-branch paradigm, i.e., training a side network to encode visual conditions and fusing its intermediate-layer features to a frozen pretrained main network, has shown remarkable success in visual-condition controllable generation. Despite its widespread adoption, the role of the side branch and its training efficiency remain underexplored. In this paper, we first revisit this mainstream paradigm through the lens of score-based generative modeling: 1) The main network preserves visual perceptual quality by providing a prior unconditional score. 2) The side network steers conditional control by implicitly contributing a likelihood score. Guided by this perspective, we propose LIkelihood Score
Alignment (LISA), an effective regularization method that explicitly aligns the intermediate feature of the side network with an approximated likelihood score. Specifically, we first hook features from a designated layer of the side network and project them into the score latent space by a lightweight decoder. Then, we construct an approximated likelihood score target and calculate the distance between the decoder's output and this target as an additional regularization loss. Finally, we jointly optimize the side network and decoder with both standard
diffusion loss and our regularization loss. Experiments across various image/
video tasks, architectures, and
diffusion/flow models demonstrated that LISA can not only consistently accelerate the training convergence and improve final synthetic results, but also encourage the side network's features to be more disentangled for conditional modeling with negligible additional training cost and zero extra inference cost.
Read the full abstract ▾
All authors Yanghao Wang, Hongxu Chen, Jiazhen Liu, Zhenqi He, Rui Liu, Zhen Wang, Long Chen
Computer Vision
HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models
Jiajun Wu, Haoyu Kang, Yining Sun et al. (+13 more)
Large
vision-
language models (LVLMs) have recently shown immense potential in automated content moderation, sparking growing interest in developing harmful-
video benchmarks. However, we identify two primary limitations in existing works: 1) The multi-layered characteristics of harmful
videos are overlooked. Existing
benchmarks predominantly formulate evaluation as a binary classification task, failing to capture implicit or deep contextual harms. 2) Explanatory rationales are completely absent. Current frameworks measure exclusively whether a model flags a
video correctly rather than explaining why, turning evaluation into a black box where models can succeed through superficial shortcuts. To address these problems, we present HarmVideoBench, a multi-layered diagnostic
benchmark comprising 1,379
videos paired with 4,137 multiple-choice questions. HarmVideoBench
benchmarks three hierarchical dimensions: Observable Evidence, Clip-Internal Meaning, and Beyond-Clip
Reasoning, aiming to evaluate models' deep understanding beyond surface cues with carefully balanced and curated samples. We evaluate 19 leading models on HarmVideoBench to assess their multidimensional understanding of harmful
videos. Moreover, we introduce BCR, a
benchmark-aligned method that predicts
reasoning boundaries and dynamically retrieves context only when needed. Experimental results show that BCR substantially improves the base model's performance in harmful
video understanding, raising the macro average from 61.7 percent to a state-of-the-art 84.4 percent.
Read the full abstract ▾
All authors Jiajun Wu, Haoyu Kang, Yining Sun, Jiacheng Hou, Heng Zhang, Danyang Zhang, Zhenjun Zhao, Haochi Zhang, Leixin Sun, Eric Hanchen Jiang, Yushan Li, Ruiyu Li, Mengkai Huang, Yan Gao, Xu Zhang, Guancheng Wan
Computer Vision
Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks
Yunqi Xue, Zhijiang Li, Philip Torr et al. (+1 more)
Unlike
diffusion-based models that operate in continuous latent spaces, autoregressive unified
multimodal models produce images by sequentially predicting discretized visual
tokens. These
tokens are derived from a codebook that maps
embeddings to quantized visual patterns. The
language-like architecture enables unified
multimodal models to effectively capture text conditional information for generation, making them promising for text-to-image tasks. This also raises an interesting question: how safe are the images generated in such an autoregressive way? In this work, we propose iterative self-improving codebooks for safe autoregressive generation. We leverage the understanding and judgment capabilities of the unified
multimodal model itself to identify unsafe generated images without human annotation. Subsequently, the inherent representations in the codebook are fixed to eliminate harmful mappings. Our method comprises two steps: first, we use the unified model to identify unsafe generations and construct corresponding harmful and safe image-text pairs. These pairs are used to construct the Harmful Space and guide updates to the codebook, thereby eliminating harmful outputs. Second, we perform adaptive fine-tuning on the codebook within the harmless space using safe image-text pairs to ensure the quality of generated images. These two steps are repeated until no further improvement is observed, producing a safety-enhanced model codebook. Without additional external feedback, the safety of models is improved iteratively.
Read the full abstract ▾
All authors Yunqi Xue, Zhijiang Li, Philip Torr, Jindong Gu
Computer Vision
FlameVQA: A Physically-Grounded UAV Wildfire VQA Benchmark with Radiometric Thermal Supervision
Mobin Habibpour, John Spodnik, Niloufar Alipour Talemi et al. (+1 more)
Wildfire monitoring from
UAVs requires reliable
reasoning over complex aerial scenes, where smoke, scale variation, and occlusions often limit RGB-only interpretation. We introduce FlameVQA, a multiple-choice visual question answering
benchmark for
UAV-based wildfire intelligence built on FLAME 3, leveraging paired RGB imagery and radiometric thermal TIFFs for temperature-grounded, safety-critical
reasoning. FlameVQA includes 34 multiple-choice questions per image spanning six operational capability groups, covering tasks such as detection, localization, distribution/coverage estimation, cross-modal
reasoning, and flight planning. To ensure label reliability, we combine MLLM-assisted annotation with deterministic thermal rules and cross-question consistency checks, followed by human
auditing. We also evaluate representative MLLMs on FlameVQA to provide baselines for future work. Results show strong performance when explicit cross-modal cues are available, but notable failures on presence detection under heavy smoke and on coverage estimation. These findings suggest that current MLLMs require domain-specific adaptation to better support disaster and wildfire monitoring. The
dataset and
benchmark code are open-source at github.com/mobiiin/WildFire_VQA
Read the full abstract ▾
All authors Mobin Habibpour, John Spodnik, Niloufar Alipour Talemi, Fatemeh Afghah
Computer Vision · Cross-listed from cs.RO
Proposal-Conditioned Latent Diffusion for Closed-Loop Traffic Scenario Generation
Shubham Vaijanath Phoolari, Aleyna Kara, Christoph Lauer et al. (+1 more)
Closed-loop traffic simulation remains challenging because it must generate interactive multi-
agent behaviors that are scene-consistent and controllable throughout rollout. Prior
diffusion-based approaches achieve strong realism, but their computational cost can hinder deployment in time-constrained replanning loops for autonomous vehicle planning and simulation. We present a
diffusion-based scenario generation framework conditioned on instance-centric scene context and
multimodal proposal priors, with optional test-time guidance for shaping safety-critical behaviors. A compact action-latent representation and proposal-based initialization improve sampling efficiency and reduce per-step runtime without retraining. Experiments on the Waymo Open Motion
Dataset demonstrate a favorable balance among realism, safety, and controllability across diverse interactive scenarios, while showing that test-time guidance enables systematic trade-offs among competing objectives.
Read the full abstract ▾
All authors Shubham Vaijanath Phoolari, Aleyna Kara, Christoph Lauer, Steven Peters
Computer Vision
TMP: Tree-structured Mixed-policy Pruning for Large-scale Image Generation and Editing
Peizhen Zhang, Yang Li, Xunsong Li et al. (+10 more)
Modern image generation model rapidly grows their sizes to meet high-fidelity image synthesis. However, they gradually become unaffordable for their enormous parameter consumption and computation budget that lead to massive resources requirement and gpu
memory footprint. In this paper, we propose TMP, the first Tree-structured Mixed-policy
Pruning framework that generalizes prevalent image tasks (T2I and TI2I) and architectures (Mixture-of-Experts (MoE) and
Diffusion transformer (DiT)). It could be applied to the step-distilled models and contribute as the last stage. We perform experiments upon current open-sourced SOTA HunyuanImage-3.0 instruct and a popular efficient model Z-Image turbo. The proposed
pruning framework manages to compress HunyuanImage 3.0 from 80B to 20B parameters at 75% reduction ratio, sacrificing limited generation quality. We also optimize to enable the inference of the pruned 20B version of HunyuanImage 3.0 on a single 24GB 4090 GPU by engineering skills. The inference script and model weight have been integrated into the existing HunyuanImage3.0 open-source github and huggingface repository. Besides, we prove the efficacy of TMP by compressing Z-Image turbo from 6B to 4B (33% reduction) with negligible degradation.
Read the full abstract ▾
All authors Peizhen Zhang, Yang Li, Xunsong Li, Songtao Liu, Zewen Liu, Qiangqiang Hu, Guotong Guo, Jupeng Ding, Yifu Sun, coopersli, Jian Zhang, Zhao Zhong, Liefeng Bo
Computer Vision
SubdivAR: Autoregressive Next-Scale Prediction for Neural Mesh Subdivision
Huipeng Guo, Zikai Song, Hang Long et al. (+7 more)
Mesh subdivision is a fundamental operation for converting coarse, editable meshes into high-resolution surfaces, with broad applications in digital asset creation. Classical rule-based schemes rely on fixed local refinement rules and often produce over-smoothed surfaces. Recent neural subdivision methods improve detail synthesis, but remain constrained by local modeling and exhibit limited generalizability. We present SubdivAR, a neural mesh subdivision framework based on our proposed Mesh Autoregressive Representation (MAR). MAR arranges meshes at different subdivision levels into an ordered scale sequence, reformulating subdivision as autoregressive next-scale prediction. To support this formulation, we introduce a Hybrid
Topology-Aware
Transformer that combines global semantic
attention with
topology-constrained local feature aggregation. SubdivAR adopts a next-scale coordinate prediction paradigm, regressing vertex offsets at each refinement stage to preserve subdivision
topology while recovering fine-grained
geometric details. To enable reliable learning, we construct FII-40K, a curated
dataset of nearly 40,000 high-quality meshes with multi-level subdivision supervision. Experiments show that SubdivAR outperforms state-of-the-art baselines, reducing Hausdorff Distance and Chamfer Distance by 18.8% and 14.2%, respectively, and demonstrates strong robustness on complex open-surface geometries.
Read the full abstract ▾
All authors Huipeng Guo, Zikai Song, Hang Long, Jielei Zhang, Wenbing Li, Junkai Lin, Tianhao Zhao, Jinshen Zhang, Tianle Guo, Wei Yang
Computer Vision
Pseudo-Text-Conditioned 3D Grounding DINO for Organ Localization in Abdominal CT
Siqi Chen, Han Gong, Keyi Hou et al. (+3 more)
Reliable organ localization in abdominal CT can provide spatial priors for downstream trauma analysis. We propose CT-3GDINO, a lightweight 3D detector that adapts a Grounding-DINO-style query-based architecture to fixed organ localization using frozen pseudo-text class
tokens instead of a real text encoder. The model combines a Swin3D visual backbone, bidirectional feature enhancement, pseudo-text-guided query selection, and a cross-modality decoder to predict normalized 3D boxes for liver, spleen, left kidney, right kidney, and bowel. We train and evaluate on 193 matched RSNA/RATIC CT volumes with
segmentation-derived boxes. The best multi-scale model, trained from scratch, achieves 0.5830 overall top-1 class-wise mAP over 3D IoU thresholds from 0.1 to 0.7, outperforming fixed- and trainable-backbone classification-pretrained variants with 0.5570 and 0.4657 mAP. Performance is strong for coarse localization, with 0.9649 AP at IoU 0.1, but remains limited for strict box
alignment, with 0.1552 AP at IoU 0.7. These results establish CT-3GDINO as an open-source baseline for pseudo-text-conditioned 3D organ localization and motivate future work on localization-aware pretraining, richer
multimodal conditioning, and injury-focused detection.
Read the full abstract ▾
All authors Siqi Chen, Han Gong, Keyi Hou, Jingxuan Yang, Sheethal Bhat, Andreas Maier
Computer Vision
PanoImager: Geometry-Guided Novel View Synthesis and Reconstruction from Sparse Panoramic Views
Zhisong Xu and Takeshi Oishi
Panoramic sensing offers wide field-of-view coverage, yet 3D reconstruction from sparse panoramas remains challenging under rotation-dominant, weak-parallax motion. In such regimes, SfM/SLAM initialization is often ill-conditioned and unreliable. We present PanoImager, an SfM-free framework that combines feed-forward pose/depth priors, geometry-conditioned
diffusion view completion, and depth-guided 3DGS optimization. Given only a few panoramic images, PanoImager decomposes them into local perspective views, synthesizes auxiliary observations to enrich sparse evidence, and stabilizes Gaussian optimization for improved cross-view consistency. Experiments on multiple
benchmarks show improved stability under extreme sparsity, suggesting PanoImager as an offline/background component for map refinement when SfM/SLAM fails to initialize.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQA
Eren Senoglu, Federico Toschi, Nicolo Brunello et al. (+2 more)
Multimodal large
language models (MLLMs) applied to Medical Visual Question Answering (VQA) tend to produce overconfident outputs regardless of actual correctness, and existing verbalized confidence calibration methods, developed primarily for text only LLMs, do not account for the
multimodal nature of medical image understanding. This work proposes a training based framework that finetunes MLLMs to improve their calibration using a composite loss function combining a Brier style calibration term, an anchor regularizer that prevents confidence collapse toward extreme values, a contrastive image text
alignment term, and a KL based model stabilization term. The
alignment signal is derived from a $2 \times 2$ factorial perturbation design that crosses image presence with text integrity, probing the reliance of the model on visual modality input versus
language priors. Finally, a top K KL divergence regularizer is used to protect the answering ability of the model during finetuning. Across three Medical VQA
benchmarks and two architectures (MedGemma 4B IT and Qwen2 VL 7B Instruct), our method reduces calibration error by 60% or more, and improves discrimination by 26% or more, while preserving predictive accuracy. On average across
benchmarks, the technique outperforms prompting based, sampling based, and training based approaches, and ablation experiments confirm that each component of the loss function is indeed necessary for improving the calibration. All code for the experiments is publicly available.
Read the full abstract ▾
All authors Eren Senoglu, Federico Toschi, Nicolo Brunello, Andrea Sassella, Mark James Carman
Computer Vision
On-board Remote-Sensing Foundation Models for Unsupervised Change Detection of Disaster Events
S. Ramírez-Gallego
Remote Sensing Foundation Models (RSFMs) have emerged as a powerful alternative to supervised models for Earth Observation, allowing satellites to autonomously trigger high-resolution captures or adjust tasking parameters upon detecting an anomaly, thereby maximizing the utility of the mission's limited power and computational resources. RSFMs are versatile, unified encoders that optimize onboard storage for multiple orbital applications while ensuring high-fidelity feature extraction. In particular, unsupervised change detection with RSFMs offers a well-informed and transformative path for disaster monitoring without expensive labels. In this paper, we present a novel unsupervised detection method based on ResNet (RSFM) + FPN which identifies a wide spectrum of anomalies by detecting subtle semantic shifts in the latent space between successive orbital passes. By relying on an untrained FPN architecture and its intrinsic priors, the system achieves efficient image-level generation and higher resolution mapping with minimal effort (training-free) compared to previous proposals (patch-based, trained). And by replacing tailored models with RSFMs, we can achieve comparable results through an approach that eliminates the need for bespoke training and extensive development effort and adds customization, while ensuring high-performance generalization across diverse terrains and sensors.
Read the full abstract ▾
Computer Vision
Event-Aware Instructed Assistant for Referring Video Segmentation
Jinyu Liu, Henghui Ding, Shuting He et al. (+1 more)
Existing referring
video segmentation methods often treat a
video as a single event consisting of multiple images, overlooking the fact that a
video typically contains multiple distinct events. Under such a mechanism, the model needs to directly understand all the complex content in the
video and text, which can easily lead to confusion and
hallucinations. To address this issue, we propose to decompose a
video to a set of simple events by learnable Event Query, and understand complex
video content in an event-by-event, easy-to-understand manner. This is based on the observation that natural
language expressions often divide a
video into distinct, text-related segments, each representing a separate event within a compound event. We introduce EVIS, an Event-Aware
Video Instructed
Segmentation Assistant, which utilizes text-guided Event Queries to partition a
video into simple events, extracting event-aware visual-text features to achieve a hierarchical understanding of the
video. Additionally, we propose Object-Pixel-Hybrid Learning, which enables the MLLMs to track targets in long-term
videos by integrating fine-grained pixel features with prior object queries. Extensive experimental results on 5 public
benchmarks demonstrate EVIS's strong performance in addressing the referring
video segmentation task.
Read the full abstract ▾
All authors Jinyu Liu, Henghui Ding, Shuting He, Yu-Gang Jiang
Computer Vision
Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation
Jinyu Liu, Xincheng Shuai, Henghui Ding et al. (+1 more)
Unified
multimodal models capable of both understanding and generation have achieved remarkable strides. However, despite their unified designs, existing evaluations typically assess understanding and generation capabilities in isolation, overlooking the synergy between comprehension and generation. To bridge this gap, we introduce Unison, a comprehensive
benchmark comprising 2,169 high-quality unified task samples, designed to evaluate joint understanding and generation in unified
multimodal models. Unison offers three key strengths: 1) Comprehensive Dimensions: Unison encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable holistic evaluation. 2) Diagnostic Evaluation: it provides both unified and decoupled tracks for understanding and generation, allowing fine-grained attribution of failure modes and quantitative analysis of the gains from unified modeling. 3) Human
Alignment: we also introduce Unison-Judge, an evaluation model well aligned with human judgments to ensure reliable assessment. Based on systematic evaluations of state-of-the-art models on Unison, we uncover critical limitations in current unified
multimodal systems and highlight promising directions for future research. Codes, Unison and Unison-Judge are publicly available at https://github.com/FudanCVL/Unison.
Read the full abstract ▾
All authors Jinyu Liu, Xincheng Shuai, Henghui Ding, Yu-Gang Jiang
Computer Vision
Geometric Gradient Rectification for Safe Open-Set Semi-Supervised Learning
Jiahe Chen, Qian Shao, Qiyuan Chen et al. (+4 more)
Open-set semi-supervised learning aims to leverage unlabeled data that may contain out-of-distribution outliers while maintaining performance on in-distribution classes. Existing methods mainly follow two paradigms: filtering suspicious samples or incorporating unlabeled objectives with soft weighting. We argue that both face a common trade-off: aggressive filtering can discard informative but hard ID samples, whereas utilization can introduce auxiliary gradients that conflict with supervised learning when pseudo labels are wrong. We therefore shift the focus from sample selection to gradient-level control. We propose
Geometric Gradient Rectification (GGR), a plug-in framework that uses the supervised gradient as an anchor and projects conflicting auxiliary gradients onto an admissible region in gradient space. This makes the applied auxiliary update first-order non-opposing within the rectified coordinate block while preserving orthogonal components that may still carry useful representation signals. We further extend GGR with subspace-aware rectification to stabilize the anchor under noisy mini-batch gradients. Experiments on CIFAR and ImageNet
benchmarks show that GGR improves representative OSSL baselines in most settings and yields gains in both closed-set generalization and open-set robustness. Code will be available at https://github.com/JiaheChen2002/GGR.
Read the full abstract ▾
All authors Jiahe Chen, Qian Shao, Qiyuan Chen, Jiaying He, Jintai Chen, Jian Wu, Hongxia Xu
Computer Vision
Computer Vision for MOBA Analytics: A Dataset and Baseline for Visibility Analysis in Dota 2
Ricardo da Rocha Carvalho, Eloísa Oliveira, Luiz Bernardo Martins Kummer et al. (+2 more)
Introduction: Most Multiplayer Online Battle Arena (MOBA) analytics studies rely on structured data, which does not directly capture what each team could actually see during a match. Objective: This work introduces Dota2-Vis, a
video-based
dataset, and a baseline pipeline for visibility analysis in professional Dota 2 matches. Methodology: The
dataset comprises all 144 matches from The International 2025, recorded from both team perspectives, totaling 288 Full HD
videos, together with 2,477 manually annotated minimap images. We evaluate multiple variants of a modern object detector for player-icon detection and use the best-performing model to estimate opponent-visible player presence over time. Results: YOLO11l (large) achieved the best overall performance, reliably identifying player icons even in dense and visually cluttered minimap scenes. The resulting visibility curves reveal player, hero, role, and team-level patterns that complement conventional MOBA analytics, highlighting behavioral differences that are difficult to obtain from structured data alone. The Dota2-Vis
dataset is publicly available at https://github.com/RicardoRCarvalho/dota2-vis/.
Read the full abstract ▾
All authors Ricardo da Rocha Carvalho, Eloísa Oliveira, Luiz Bernardo Martins Kummer, Emerson Cabrera Paraiso, Rayson Laroca
Computer Vision · Cross-listed from cs.AI
Einstein World Models
Munachiso Samuel Nwadike, Zangir Iklassov, Ali Mekky et al. (+2 more)
Does intelligence require the ability to reason about phenomena beyond direct experience? It is natural to suspect that some complex thought cannot be captured through
language alone. However, of particular concern to this work, is whether visualising counterfactual events can complement
language as a mechanism for complex thought. We ask whether LLMs can be trained to utilise such visualisation mechanisms, in a way that benefits their
reasoning abilities. Motivated by this question, we propose Einstein World Models. EWMs are a blueprint for LLM-based
reasoning systems that place visual-temporal rollouts inside the
reasoning trace, allowing them to reason in ways that text alone may not support well. In an EWM, the LLM calls a world-module (not to be confused with a world model), to produce short rollouts of scenes under consideration. The returned rollout is treated not as the answer, but as an inspectable hypothesis that can support later
reasoning. Einstein World Models extend the capability of LLMs for tool calling (such as web search or code execution), into the domain of visual thought experiments.
Read the full abstract ▾
All authors Munachiso Samuel Nwadike, Zangir Iklassov, Ali Mekky, Zayd M. Kawakibi Zuhri, Kentaro Inui
Computer Vision · Cross-listed from cs.AI
Look-Before-Move: Narrative-Grounded World Visual Attention in Dynamic 3D Story Worlds
Jiaming Bian, Bingliang Li, Yuehao Wu et al. (+5 more)
As
embodied AI and world models increasingly operate in dynamic 3D environments, visual perception must move beyond passively interpreting given observations toward actively deciding what to observe. We study this problem through camera planning in dynamic 3D story worlds, where the camera must not only generate smooth motion, but also decide what visual evidence should be acquired before it moves. We formulate this capability as Narrative-Grounded World Visual
Attention, where the camera acts as an
embodied observer that determines what to observe, how to compose the observation, and how to shift
attention over time under narrative intent and physical 3D constraints. To realize this capability, we propose Look-Before-Move, a camera planning framework that separates observation specification from motion execution. It first builds a Semantic Observation Contract to convert directorial intent into executable visual constraints, then performs Monte Carlo Viewpoint Search to find narrative-compliant and geometrically feasible viewpoints, and finally applies Semantic Trajectory Grounding to connect selected viewpoints into continuous, collision-aware, and temporally coherent camera motion. We further construct a dynamic 3D Story World
Benchmark based on StoryBlender, covering 50 stories, 457 scenes, and 1585 shots with animated characters, semantic scene configurations, and executable 3D environments. Experiments show that our framework improves subject perception, intent consistency, and trajectory quality over representative baselines, demonstrating the importance of organizing visual
attention before generating camera motion.
Read the full abstract ▾
All authors Jiaming Bian, Bingliang Li, Yuehao Wu, Pichao Wang, Zhi Wang, Hailan Ma, Huadong Mo, Zhenhong Sun
Computer Vision
Scaling Multi-Reference Image Generation with Dynamic Reward Optimization
Wenwang Huang, Yusen Fu, Junjie Wang et al. (+6 more)
While personalized image generation has achieved remarkable progress, multi-reference image generation (MRIG) remains a challenging task. Most existing
benchmarks fail to adequately evaluate complex MRIG scenarios, hindering further progress in this area. To better assess model performance on complex MRIG tasks, we introduce OmniRef-Bench, a
benchmark that covers complex combinations of reference image types and a large number of reference images. Evaluations on OmniRef-Bench show that mainstream open-source models struggle in complex MRIG scenarios, and their performance deteriorates significantly as the number of mixed-type reference images increases. To address this issue, we propose DyRef, a two-stage training framework. In the first stage, supervised fine-tuning equips the model with the basic capability to handle complex MRIG tasks. In the second stage, we introduce Difficulty-aware Advantage Reweighting (DAR) and Discriminative Reward Scaling (DRS). DAR dynamically adjusts the optimization objective to improve performance when handling a large number of mixed-type reference images. DRS enlarges intra-group reward differences for more effective policy optimization. Experiments demonstrate that DyRef significantly improves the performance of open-source models on OmniRef-Bench and single-image editing
benchmarks, demonstrating the effectiveness and generalization capability of our approach.
Read the full abstract ▾
All authors Wenwang Huang, Yusen Fu, Junjie Wang, Mengfei Huang, Yulin Li, Gan Liu, Jing Cai, Yancheng He, Zhuotao Tian
Computer Vision
TraMP-LLaMA: Generative Interpretability with Decoupled Instruction Tuning for Facial Expression Quality Assessment
Shuchao Duan, Alan Whone, Hossein Rahmani et al. (+2 more)
Existing facial expression quality assessment (FEQA) methods typically produce only a severity score, without explicitly communicating the observable facial motion evidence that supports the prediction. This limits interpretability and makes it difficult to inspect the basis of model outputs in Parkinson's disease assessment. To address this gap, we propose TraMP-LLaMA, a unified
multimodal framework that jointly predicts severity scores and generates structured textual reports from facial motion cues. The framework integrates RGB appearance and landmark trajectory cues, and adopts a decoupled instruction-tuning strategy to reduce task interference between severity prediction and
language generation. To support this task, we further extend the PFED5
dataset with expert-guided textual motion descriptions and construct PFED5-plus. Experiments on PFED5-plus show that TraMP-LLaMA outperforms competitive
video-
language baselines in report generation and achieves the best severity prediction performance among the compared methods under joint multi-expression training, improving Spearman's rank correlation by at least 4.39 percent over all competing methods. The text annotations and code are available at https://github.com/shuchaoduan/TraMP-LLaMA.
Read the full abstract ▾
All authors Shuchao Duan, Alan Whone, Hossein Rahmani, Jun Liu, Majid Mirmehdi
Computer Vision
Focusing on What Matters: Saliency-Harnessing Accurate Routing for Diffusion MoE
Haoyou Deng, Keyu Yan, Chaojie Mao et al. (+4 more)
Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling
diffusion models in visual generation. Recent advancements have focused on adaptively allocating computational resources across diverse
tokens to improve efficiency and performance. However, we identify a routing assignment problem in existing
diffusion MoE frameworks: the router fails to accurately allocate more computational resources to salient
tokens. Our analysis attributes this failure to the router's reliance on noise-corrupted latent features throughout the denoising process. Such
stochastic noise obscures the critical structural and textural information, thereby preventing the router from effectively distinguishing salient
tokens. To address this, we propose SharpMoE, a post-training framework with a saliency-harnessing accurate routing mechanism, which utilizes clean latent features as a noise-free guidance signal for routing. By bypassing the noise-distorted inputs, SharpMoE provides the router with clear saliency guidance, enabling the identification of salient
tokens even in high-noise stages. Furthermore, we introduce a trajectory routing loss to constrain the compute allocation throughout the multi-step denoising trajectory, ensuring precise resource allocation along the generation rollout. Extensive experiments demonstrate that SharpMoE serves as a versatile, plug-and-play solution that further enhances the pretrained, converged MoE models, achieving state-of-the-art performance in visual generation.
Read the full abstract ▾
All authors Haoyou Deng, Keyu Yan, Chaojie Mao, Xiang Wang, Yu Liu, Changxin Gao, Nong Sang
Computer Vision
PortraitGen: Exemplar-Driven GRPO with Dual-Reward Guidance for Photorealistic Portrait Generation
Xiaomin Li, Qian Liang, Yinan Li et al. (+5 more)
Reinforcement Learning like Group Relative Policy Optimization (GRPO) has significantly advanced text-to-image post-training. However, current methods often favor superficial aesthetics, such as over-saturated colors, leaving critical flaws like AI artifacts and biological implausibilities unresolved. We attribute these limitations to two primary factors: (1) The absence of real images during post-training confines GRPO sampling to the original distribution, failing to break inherent generative boundaries; (2) the optimization process lacks specific rewards targeting fine-grained artifacts like overly oily skin and other AI artifacts. To address this, we propose PortraitGen, a novel framework tailored for photorealistic portrait generation. First, we break inherent generative boundaries by directly introducing real images into the GRPO sampling groups, where image inversion is employed to obtain their transition probabilities and latents. Second, to explicitly steer the model toward photorealism, we introduce a complementary dual-reward mechanism: OmniReward for general quality and AI-Portrait for human-centric fidelity. Furthermore, we curate PortraitBench, a comprehensive portrait-centric
benchmark. Extensive experiments demonstrate that PortraitGen significantly outperforms existing baselines, effectively suppressing AI artifacts and achieving unprecedented photorealism.
Read the full abstract ▾
All authors Xiaomin Li, Qian Liang, Yinan Li, Ying Zhang, Chen Li, Jing Lyu, Huchuan Lu, Xu Jia