Computer Vision
From Leaderboard to Deployment: Code Quality Challenges in AV Perception Repositories
Mateus Karvat, Bram Adams and Sidney Givigi
Autonomous vehicle (AV) perception models are typically evaluated solely on
benchmark performance metrics, with limited
attention to code quality, production readiness and long-term maintainability. This creates a significant gap between research excellence and real-world deployment in safety-critical systems subject to international safety standards. To address this gap, we present the first large-scale empirical study of software quality in AV perception repositories, systematically analyzing 178 unique models from the KITTI and NuScenes 3D Object Detection leaderboards. Using static analysis tools (Pylint, Bandit, and Radon), we evaluated code errors, security vulnerabilities, maintainability, and development practices. Our findings revealed that only 7.3% of the studied repositories meet basic production-readiness criteria, defined as having zero critical errors and no high-severity security vulnerabilities. Security issues are highly concentrated, with the top five issues responsible for almost 80% of occurrences, which prompted us to develop a set of actionable guidelines to prevent them. Additionally, the adoption of Continuous Integration/Continuous Deployment pipelines was correlated with better code maintainability. Our findings highlight that leaderboard performance does not reflect production readiness and that targeted interventions could substantially improve the quality and safety of AV perception code.
Read the full abstract ▾
Computer Vision
Sketch2Colab: Sketch-Conditioned Multi-Human Animation via Controllable Flow Distillation
Divyanshu Daiya and Aniket Bera
We present Sketch2Colab, which turns storyboard-style 2D sketches into coherent, object-aware 3D multi-human motion with fine-grained control over
agents, joints, timing, and contacts.
Diffusion-based motion generators offer strong realism but often rely on costly guidance for multi-entity control and degrade under strong conditioning. Sketch2Colab instead learns a sketch-conditioned
diffusion prior and distills it into a rectified-flow student in latent space for fast, stable sampling. To make motion follow storyboards closely, we guide the student with differentiable objectives that enforce keyframes, paths, contacts, and physical consistency. Collaborative motion naturally involves discrete changes in interaction, such as converging, forming contact, cooperative transport, or disengaging, and a continuous flow alone struggles to sequence these shifts cleanly. We address this with a lightweight continuous-time Markov chain (CTMC) planner that tracks the active interaction regime and modulates the flow to produce clearer, synchronized coordination in human-object-human motion. Experiments on CORE4D and InterHuman show that Sketch2Colab outperforms baselines in constraint adherence and perceptual quality while sampling substantially faster than
diffusion-only alternatives.
Read the full abstract ▾
Computer Vision
Leveraging Model Soups to Classify Intangible Cultural Heritage Images from the Mekong Delta
Quoc-Khang Tran, Minh-Thien Nguyen and Nguyen-Khang Pham
The classification of Intangible Cultural Heritage (ICH) images in the Mekong Delta poses unique challenges due to limited annotated data, high visual similarity among classes, and domain heterogeneity. In such low-resource settings, conventional deep learning models often suffer from high variance or overfit to spurious correlations, leading to poor generalization. To address these limitations, we propose a robust framework that integrates the hybrid CoAtNet architecture with model soups, a lightweight weight-space ensembling technique that averages checkpoints from a single training trajectory without increasing inference cost. CoAtNet captures both local and global patterns through stage-wise fusion of convolution and self-
attention. We apply two ensembling strategies - greedy and uniform soup - to selectively combine diverse checkpoints into a final model. Beyond performance improvements, we analyze the ensembling effect through the lens of bias-variance decomposition. Our findings show that model soups reduces variance by stabilizing predictions across diverse model snapshots, while introducing minimal additional bias. Furthermore, using cross-
entropy-based distance metrics and Multidimensional Scaling (MDS), we show that model soups selects geometrically diverse checkpoints, unlike Soft Voting, which blends redundant models centered in output space. Evaluated on the ICH-17
dataset (7,406 images across 17 classes), our approach achieves state-of-the-art results with 72.36% top-1 accuracy and 69.28% macro F1-score, outperforming strong baselines including ResNet-50, DenseNet-121, and ViT. These results underscore that diversity-aware checkpoint averaging provides a principled and efficient way to reduce variance and enhance generalization in culturally rich, data-scarce classification tasks.
Read the full abstract ▾
Computer Vision
Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance
Yiqi Lin, Guoqiang Liang, Ziyun Zeng et al. (+3 more)
Instruction-based
video editing has witnessed rapid progress, yet current methods often struggle with precise visual control, as natural
language is inherently limited in describing complex visual nuances. Although reference-guided editing offers a robust solution, its potential is currently bottlenecked by the scarcity of high-quality paired training data. To bridge this gap, we introduce a scalable data generation pipeline that transforms existing
video editing pairs into high-fidelity training quadruplets, leveraging image generative models to create synthesized reference scaffolds. Using this pipeline, we construct RefVIE, a large-scale
dataset tailored for instruction-reference-following tasks, and establish RefVIE-Bench for comprehensive evaluation. Furthermore, we propose a unified editing architecture, Kiwi-Edit, that synergizes learnable queries and latent visual features for reference semantic guidance. Our model achieves significant gains in instruction following and reference fidelity via a progressive multi-stage training curriculum. Extensive experiments demonstrate that our data and architecture establish a new state-of-the-art in controllable
video editing. All
datasets, models, and code is released at https://github.com/showlab/Kiwi-Edit.
Read the full abstract ▾
All authors Yiqi Lin, Guoqiang Liang, Ziyun Zeng, Zechen Bai, Yanzhe Chen, Mike Zheng Shou
Computer Vision
TerraDiT: Point-Conditioned Diffusion Transformer for Satellite Image Synthesis
Srikumar Sastry, Dan Cher, Brian Wei et al. (+4 more)
We introduce TerraDiT, a
diffusion transformer designed for text-to-satellite image generation with point-based control. Existing controlled satellite image generative models often require pixel-level maps that are time-consuming to acquire, yet semantically limited. To address this limitation, we introduce a novel point-based conditioning framework that controls the generation process through the spatial location of the points and the textual description associated with each point, providing semantically rich control signals. This approach enables flexible, annotation-friendly, and computationally simple inference for satellite image generation. To this end, we introduce an adaptive local
attention mechanism that effectively regularizes the
attention scores based on the input point queries. We systematically evaluate various domain-specific design choices for training TerraDiT, including the selection of satellite image representation for
alignment and geolocation representation for conditioning. Our experiments demonstrate that TerraDiT achieves impressive generation performance, surpassing the state-of-the-art remote sensing generative models. Our models,
dataset, and code are available at https://github.com/mvrl/TerraDiT.
Read the full abstract ▾
All authors Srikumar Sastry, Dan Cher, Brian Wei, Aayush Dhakal, Subash Khanal, Dev Gupta, Nathan Jacobs
Computer Vision
Bridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction
Aniek Eijpe, Soufyan Lakbir, Melis Erdal Cesur et al. (+4 more)
While
multimodal survival prediction models are increasingly more accurate, their complexity often reduces interpretability, limiting insight into how different data sources influence predictions. To address this, we introduce DIMAFx, an explainable
multimodal framework for cancer survival prediction that produces disentangled, interpretable modality-specific and modality-shared representations from histopathology whole-slide images and transcriptomics data. Across multiple cancer cohorts, DIMAFx achieves state-of-the-art performance and improved representation disentanglement. Leveraging its interpretable design and SHapley Additive exPlanations, DIMAFx systematically reveals key
multimodal interactions and the biological information encoded in the disentangled representations. In breast cancer survival prediction, the most predictive features contain modality-shared information, including one capturing solid tumor morphology contextualized primarily by late estrogen response, where higher-grade morphology aligned with pathway upregulation and increased risk, consistent with known breast cancer biology. Key modality-specific features capture microenvironmental signals from interacting adipose and stromal morphologies. These results show that
multimodal models can overcome the traditional trade-off between performance and explainability, supporting their application in precision medicine.
Read the full abstract ▾
All authors Aniek Eijpe, Soufyan Lakbir, Melis Erdal Cesur, Sara P. Oliveira, Angelos Chatzimparmpas, Sanne Abeln, Wilson Silva
Computer Vision
3D Field of Junctions: A Noise-Robust, Training-Free Structural Prior for Volumetric Inverse Problems
Narges Moeini, Namhoon Kim, Justin Romberg et al. (+1 more)
Volume denoising is a foundational problem in computational imaging, as many 3D imaging inverse problems face high levels of measurement noise. Inspired by the strong 2D image denoising properties of Field of Junctions (ICCV 2021), we propose a novel, fully volumetric 3D Field of Junctions (3D FoJ) representation that optimizes a junction of 3D wedges that best explain each 3D patch of a full volume, while encouraging consistency between overlapping patches. In addition to direct volume denoising, we leverage our 3D FoJ representation as a structural prior that: (i) requires no training data, and thus precludes the risk of
hallucination, (ii) preserves and enhances sharp edge and corner structures in 3D, even under low signal to noise ratio (SNR), and (iii) can be used as a drop-in denoising representation via projected or proximal gradient descent for any volumetric inverse problem with low SNR. We demonstrate successful volume reconstruction and denoising with 3D FoJ across three diverse 3D imaging tasks with low-SNR measurements: low-dose X-ray computed
tomography (CT), cryogenic electron
tomography (cryo-ET), and denoising point clouds such as those from
lidar in adverse weather. Across these challenging low-SNR volumetric imaging problems, 3D FoJ outperforms the evaluated classical denoisers, untrained neural denoisers, and denoisers trained only on noisy examples. Code is available at https://github.com/voilalab/3D-Field-of-Junctions.
Read the full abstract ▾
All authors Narges Moeini, Namhoon Kim, Justin Romberg, Sara Fridovich-Keil
Computer Vision
When Bigger is Worse: A Practitioner's Guide to Model Selection Under Data Scarcity
Kwame Mbobda-Kuate and Gabriel Kasmi
Scaling laws assume larger models trained on more data consistently outperform smaller ones – an assumption that drives model selection in computer
vision but remains untested in resource-constrained Earth observation (EO). We conduct a systematic efficiency analysis across three scaling dimensions: model size,
dataset size, and input resolution, on rooftop photovoltaic (PV) detection in Madagascar, yielding 180 training runs across 60 configurations. Optimizing for model efficiency (mAP$_{50}$ per unit of model size), we find a consistent efficiency inversion: YOLO11N achieves the highest efficiency ($22\times$ higher than YOLO11X) with no accuracy penalty: it reaches the second highest absolute mAP$_{50}$ (0.459), outperforming all models except YOLO11S by a margin smaller than run-to-run variance, directly contradicting the scaling prior. Resolution is the dominant resource allocation lever: moving from 416 px to 1280 px at 10 % of the data matches the efficiency gain of collecting the full
dataset at low resolution. These findings are robust to the deployment objective: small high-resolution configurations are Pareto-dominant across all 60 experimental setups in the joint accuracy-throughput space, leaving no tradeoff to resolve. In data-scarce EO, the scaling prior does not just fail: it inverts.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Rethinking Camera Choice: An Empirical Study on Fisheye Camera Properties in Robotic Manipulation
Han Xue, Nan Min, Xiaotong Liu et al. (+5 more)
The adoption of fisheye cameras in robotic manipulation, driven by their exceptionally wide Field of View (FoV), is rapidly outpacing a systematic understanding of their downstream effects on policy learning. This paper presents the first comprehensive empirical study to bridge this gap, rigorously analyzing the properties of wrist-mounted fisheye cameras for imitation learning. Through extensive experiments in both simulation and the real world, we investigate three critical research questions: spatial localization, scene generalization, and hardware generalization. Our investigation reveals that: (1) The wide FoV significantly enhances spatial localization, but this benefit is critically contingent on the visual complexity of the environment. (2) Fisheye-trained policies, while prone to overfitting in simple scenes, unlock superior scene generalization when trained with sufficient environmental diversity. (3) While naive cross-camera transfer leads to failures, we identify the root cause as scale overfitting and demonstrate that hardware generalization performance can be improved with a simple Random Scale Augmentation (RSA) strategy. Collectively, our findings provide concrete, actionable guidance for the large-scale collection and effective use of fisheye
datasets in robotic learning. More results and
videos are available on https://robo-fisheye.github.io/
Read the full abstract ▾
All authors Han Xue, Nan Min, Xiaotong Liu, Wendi Chen, Yuan Fang, Jun Lv, Cewu Lu, Chuan Wen
Computer Vision
OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens
Yiying Yang, Wei Cheng, Sijin Chen et al. (+5 more)
OmniLottie is a versatile framework that generates high quality vector animations from multi-modal instructions. For flexible motion and visual content control, we focus on Lottie, a light weight JSON formatting for both shapes and animation behaviors representation. However, the raw Lottie JSON files contain extensive invariant structural metadata and formatting
tokens, posing significant challenges for learning vector animation generation. Therefore, we introduce a well designed Lottie tokenizer that transforms JSON files into structured sequences of commands and parameters representing shapes, animation functions and control parameters. Such tokenizer enables us to build OmniLottie upon pretrained
vision language models to follow multi-modal interleaved instructions and generate high quality vector animations. To further advance research in vector animation generation, we curate MMLottie-2M, a large scale
dataset of professionally designed vector animations paired with textual and visual annotations. With extensive experiments, we validate that OmniLottie can produce vivid and semantically aligned vector animations that adhere closely to multi modal human instructions.
Read the full abstract ▾
All authors Yiying Yang, Wei Cheng, Sijin Chen, Honghao Fu, Xianfang Zeng, Yujun Cai, Gang Yu, Xingjun Ma
Computer Vision · Cross-listed from cs.IR
NextAds: Towards Next-generation Personalized Video Advertising
Yiyan Xu, Ruoxuan Xia, Wuqiang Zheng et al. (+3 more)
With the rapid growth of online
video consumption,
video advertising has become increasingly dominant in the digital advertising landscape. Yet diverse users and viewing contexts make one-size-fits-all ad creatives insufficient for consistent effectiveness, underlining the importance of personalization. In practice, most personalized
video advertising systems follow a
retrieval-based paradigm, selecting the optimal one from a small set of professionally pre-produced creatives for each user. Such static and finite inventories limit both the granularity and the timeliness of personalization, and prevent the creatives from being continuously refined based on online user feedback. Recent advances in generative AI make it possible to move beyond
retrieval toward optimizing
video creatives in a continuous space at serving time. In this light, we propose NextAds, a generation-based paradigm for next-generation personalized
video advertising. We articulate its key principles and conceptualize an agentic workflow for closed-loop creative optimization, spanning planning, generation, verification, and reflection. We further explore how personalized creatives can evolve across impressions through feedback-driven refinement and progressive storytelling. To further ground this
vision, we formalize two representative tasks and conduct exploratory experiments on corresponding lightweight
benchmarks, revealing both the promise and limitations of existing GenAI. Moreover, we discuss potential research opportunities and the broader implications of increasingly personalized and native advertising, highlighting issues that warrant
attention from the community and aiming to provide insights for future research and responsible deployment. Our code,
benchmarks, and demonstrations are available at https://anonymous.4open.science/w/NextAdsProjectPage/.
Read the full abstract ▾
All authors Yiyan Xu, Ruoxuan Xia, Wuqiang Zheng, Fengbin Zhu, Wenjie Wang, Fuli Feng
Computer Vision
OnlineX: Unified Online 3D Reconstruction and Understanding with Active-to-Stable State Evolution
Chong Xia, Fangfu Liu, Yule Wang et al. (+2 more)
Recent advances in generalizable 3D Gaussian Splatting (3DGS) have enabled rapid 3D scene reconstruction within seconds, eliminating the need for per-scene optimization. However, existing methods primarily follow an offline reconstruction paradigm, lacking the capacity for continuous reconstruction, which limits their applicability to online scenarios such as robotics and VR/AR. In this paper, we introduce OnlineX, a feed-forward framework that reconstructs both 3D visual appearance and
language fields in an online manner using only streaming images. A key challenge in online formulation is the cumulative drift issue, which is rooted in the fundamental conflict between two opposing roles of the
memory state: an active role that constantly refreshes to capture high-frequency local geometry, and a stable role that conservatively accumulates and preserves the long-term global structure. To address this, we introduce a decoupled active-to-stable state
evolution paradigm. Our framework decouples the
memory state into a dedicated active state and a persistent stable state, and then cohesively fuses the information from the former into the latter to achieve both fidelity and stability. Moreover, we jointly model visual appearance and
language fields and incorporate an implicit Gaussian fusion module to enhance reconstruction quality. Experiments on mainstream
datasets demonstrate that our method consistently outperforms prior work in novel view synthesis and semantic understanding, showcasing robust performance across input sequences of varying lengths with real-time inference speed.
Read the full abstract ▾
All authors Chong Xia, Fangfu Liu, Yule Wang, Yize Pang, Yueqi Duan
Computer Vision
SimRecon: SimReady Compositional Scene Reconstruction from Real Videos
Chong Xia, Kai Zhu, Zizhuo Wang et al. (+3 more)
Compositional scene reconstruction seeks to create object-centric representations rather than holistic scenes from real-world
videos, which is natively applicable for simulation and interaction. Conventional compositional reconstruction approaches primarily emphasize on visual appearance and show limited generalization ability to real-world scenarios. In this paper, we propose SimRecon, a framework that realizes a "Perception-Generation-Simulation" pipeline towards cluttered scene reconstruction, which first conducts scene-level semantic reconstruction from
video input, then performs single-object generation, and finally assembles these assets in the simulator. However, naively combining these three stages leads to visual infidelity of generated assets and physical implausibility of the final scene, a problem particularly severe for complex scenes. Thus, we further propose two bridging modules between the three stages to address this problem. To be specific, for the transition from Perception to Generation, critical for visual fidelity, we introduce Active Viewpoint Optimization, which actively searches in 3D space to acquire optimal projected images as conditions for single-object completion. Moreover, for the transition from Generation to Simulation, essential for physical plausibility, we propose a Scene Graph Synthesizer, which guides the construction from scratch in 3D simulators, mirroring the native, constructive principle of the real world. Extensive experiments on the ScanNet
dataset validate our method's superior performance over previous state-of-the-art approaches.
Read the full abstract ▾
All authors Chong Xia, Kai Zhu, Zizhuo Wang, Fangfu Liu, Zhizheng Zhang, Yueqi Duan
Computer Vision
Stereo-Inertial Poser: Towards Metric-Accurate Shape-Aware Motion Capture Using Sparse IMUs and a Single Stereo Camera
Tutian Tang, Xingyu Ji, Yutong Li et al. (+3 more)
Recent advancements in visual-inertial motion capture systems have demonstrated the potential of combining
monocular cameras with sparse inertial measurement units (IMUs) as cost-effective solutions, which effectively mitigate occlusion and drift issues inherent in single-modality systems. However, they are still limited by metric inaccuracies in global translations stemming from
monocular depth ambiguity, and shape-agnostic local motion estimations that ignore anthropometric variations. We present Stereo-Inertial Poser, a real-time motion capture system that leverages a single stereo camera and six IMUs to estimate metric-accurate and shape-aware 3D human motion. By replacing the
monocular RGB with stereo
vision, our system resolves depth ambiguity through calibrated baseline geometry, enabling direct 3D keypoint extraction and body shape parameter estimation. IMU data and visual cues are fused for predicting drift-compensated joint positions and root movements, while a novel shape-aware fusion module dynamically harmonizes anthropometry variations with global translations. Our end-to-end pipeline achieves over 200 FPS without optimization-based post-processing, enabling real-time deployment. Quantitative evaluations across various
datasets demonstrate state-of-the-art performance. Qualitative results show our method produces drift-free global translation under a long recording time and reduces foot-skating effects.
Read the full abstract ▾
All authors Tutian Tang, Xingyu Ji, Yutong Li, MingHao Liu, Wenqiang Xu, Cewu Lu
Computer Vision
LiftAvatar: Kinematic-Space Completion for Expression-Controlled 3D Gaussian Avatar Animation
Hualiang Wei, Shunran Jia, Jialun Liu et al. (+1 more)
We present LiftAvatar, a new paradigm that completes sparse
monocular observations in kinematic space (e.g., facial expressions and head pose) and uses the completed signals to drive high-fidelity avatar animation. LiftAvatar is a fine-grained, expression-controllable large-scale
video diffusion Transformer that synthesizes high-quality, temporally coherent expression sequences conditioned on single or multiple reference images. The key idea is to lift incomplete input data into a richer kinematic representation, thereby strengthening both reconstruction and animation in downstream 3D avatar pipelines. To this end, we introduce (i) a multi-granularity expression control scheme that combines shading maps with expression coefficients for precise and stable driving, and (ii) a multi-reference conditioning mechanism that aggregates complementary cues from multiple frames, enabling strong 3D consistency and controllability. As a plug-and-play enhancer, LiftAvatar directly addresses the limited expressiveness and reconstruction artifacts of 3D Gaussian Splatting-based avatars caused by sparse kinematic cues in everyday
monocular videos. By expanding incomplete observations into diverse pose-expression variations, LiftAvatar also enables effective prior
distillation from large-scale
video generative models into 3D pipelines, leading to substantial gains. Extensive experiments show that LiftAvatar consistently boosts animation quality and quantitative metrics of state-of-the-art 3D avatar methods, especially under extreme, unseen expressions.
Read the full abstract ▾
All authors Hualiang Wei, Shunran Jia, Jialun Liu, Wenhui Li
Computer Vision
A 3D mesh convolution-based autoencoder for geometry compression
Germain Bregeon, Marius Preda, Radu Ispas et al. (+1 more)
In this paper, we introduce a novel 3D mesh convolution-based autoencoder for geometry compression, able to deal with irregular mesh data without requiring neither preprocessing nor
manifold/watertightness conditions. The proposed approach extracts meaningful latent representations by learning features directly from the mesh faces, while preserving connectivity through dedicated pooling and unpooling operations. The encoder compresses the input mesh into a compact base mesh space, which ensures that the latent space remains comparable. The decoder reconstructs the original connectivity and restores the compressed geometry to its full resolution. Extensive experiments on multi-class
datasets demonstrate that our method outperforms state-of-the-art approaches in both 3D mesh geometry reconstruction and latent space classification tasks. Code available at: github.com/germainGB/MeshConv3D
Read the full abstract ▾
All authors Germain Bregeon, Marius Preda, Radu Ispas, Titus Zaharia
Computer Vision · Cross-listed from cs.AI
Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy
Jiahao Huang, Fengyan Lin, Xuechao Yang et al. (+4 more)
The development of affective
multimodal language models (MLMs) has long been constrained by a gap between low-level perception and high-level interaction, leading to fragmented affective capabilities and limited generalization. To bridge this gap, we propose a cognitively inspired three-level hierarchy that organizes affective tasks according to their cognitive depth-perception, understanding, and interaction-and provides a unified conceptual foundation for advancing affective modeling. Guided by this hierarchy, we introduce Nano-EmoX, a small-scale multitask MLM, and P2E (Perception-to-Empathy), a curriculum-based training framework. Nano-EmoX integrates a suite of omni-modal encoders, including an enhanced facial encoder and a fusion encoder, to capture key
multimodal affective cues and improve cross-task transferability. The outputs are projected into a unified
language space via heterogeneous adapters, empowering a lightweight
language model to tackle diverse affective tasks. Concurrently, P2E progressively cultivates emotional intelligence by aligning rapid perception with chain-of-thought-driven empathy. To the best of our knowledge, Nano-EmoX is the first compact MLM (2.2B) to unify six core affective tasks across all three hierarchy levels, achieving state-of-the-art or highly competitive performance across multiple
benchmarks, demonstrating excellent efficiency and generalization. The code is available at https://github.com/waHAHJIAHAO/Nano-EmoX.
Read the full abstract ▾
All authors Jiahao Huang, Fengyan Lin, Xuechao Yang, Chen Feng, Kexin Zhu, Xu Yang, Zhide Chen
Computer Vision · Cross-listed from cs.IR
Efficient and High-Fidelity Omni Modality Retrieval
Chuong Huynh, Manh Luong and Abhinav Shrivastava
Multimodal retrieval is the task of aggregating information from queries across heterogeneous modalities to retrieve desired targets. State-of-the-art
multimodal retrieval models can understand complex queries, yet they are typically limited to two modalities: text and
vision. This limitation impedes the development of universal
retrieval systems capable of comprehending queries that combine more than two modalities. To advance toward this goal, we present OmniRet, the first
retrieval model capable of handling complex, composed queries spanning three key modalities: text,
vision, and audio. Our OmniRet model addresses two critical challenges for universal
retrieval: computational efficiency and representation fidelity. First, feeding massive
token sequences from modality-specific encoders to Large
Language Models (LLMs) is computationally inefficient. We therefore introduce an
attention-based resampling mechanism to generate compact, fixed-size representations from these sequences. Second, compressing rich omni-modal data into a single
embedding vector inevitably causes information loss and discards fine-grained details. We propose
Attention Sliced Wasserstein Pooling to preserve these fine-grained details, leading to improved omni-modal representations. OmniRet is trained on an aggregation of approximately 6 million query-target pairs spanning 30
datasets. We
benchmark our model on 13
retrieval tasks and a MMEBv2 subset. Our model demonstrates significant improvements on composed query, audio and
video retrieval tasks, while achieving on-par performance with state-of-the-art models on others. Furthermore, we curate a new Audio-Centric
Multimodal Benchmark (ACM). This new
benchmark introduces two critical, previously missing tasks-composed audio
retrieval and audio-visual
retrieval to more comprehensively evaluate a model's omni-modal
embedding capacity.
Read the full abstract ▾
Computer Vision
FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding
Yiweng Xie, Bo He, Junke Wang et al. (+3 more)
This paper presents FluxMem, a training-free framework for efficient streaming
video understanding. FluxMem adaptively compresses redundant visual
memory through a hierarchical, two-stage design: (1) a Temporal Adjacency Selection (TAS) module removes redundant visual
tokens across adjacent frames, and (2) a Spatial Domain Consolidation (SDC) module further merges spatially repetitive regions within each frame into compact representations. To adapt effectively to dynamic scenes, we introduce a self-adaptive
token compression mechanism in both TAS and SDC, which automatically determines the compression rate based on intrinsic scene statistics rather than manual tuning. Extensive experiments demonstrate that FluxMem achieves new state-of-the-art results on existing online
video benchmarks, reaching 76.4 on StreamingBench and 67.2 on OVO-Bench under real-time settings, while reducing latency by 69.9% and peak GPU
memory by 34.5% on OVO-Bench. Furthermore, it maintains strong offline performance, achieving 73.1 on MLVU while using 65% fewer visual
tokens.
Read the full abstract ▾
All authors Yiweng Xie, Bo He, Junke Wang, Xiangyu Zheng, Ziyi Ye, Zuxuan Wu
Computer Vision
A Detection-Gated Pipeline for Robust Glottal Area Waveform Extraction and Clinical Pathology Assessment
Harikrishnan Unnikrishnan and Rita Patel
We present a fully automated, two-stage modular glottal area
segmentation framework for high-speed videoendoscopy (HSV) designed for accuracy, generalizability, and real-time playback. Our detection-gated pipeline combines a YOLOv8n glottis localizer with a U-Net segmenter; the localizer defines a tight crop to ensure a consistent field of view and gates the output to reduce spurious
segmentations during glottal closure. The models were trained on the GIRAFE (N=600) and BAGLS (N=55,750)
datasets. Cross-
dataset portability was evaluated by benchmarking GIRAFE-trained models on the BAGLS test set without fine-tuning. In these evaluations, the pipeline achieved a Dice Similarity Coefficient (DSC) of 0.745 (87% of the in-domain ceiling). On in-distribution test sets, the system achieved DSCs of 0.81 (GIRAFE) and 0.856 (BAGLS), outperforming or competing with state-of-the-art methods. An exploratory clinical study of 40 subjects demonstrated that the glottal area Coefficient of Variation (CV distinguished healthy from pathological function (p=0.006). The system processes ~35 frames per second on commodity hardware, enabling interactive clinical review. This design supports uniform extraction of laryngeal kinematic measures across varying acquisition settings. Code, weights, and software are available at https://github.com/hari-krishnan/openglottal.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
$\pi$-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs
Siting Wang, Xiaofeng Wang, Zheng Zhu et al. (+7 more)
Flow-based
vision-
language-action (VLA) models have emerged as a powerful paradigm for
embodied control, but multi-step denoising makes policy-gradient online reinforcement learning (RL) difficult because the final-action likelihood induced by the sampler is not directly tractable. We identify an exploration–supervision mismatch within the denoising trajectory, where
stochastic differential equation (SDE) sampling explores a wider local
stochastic transition space while supervision derived from the final denoised output does not directly use the adjacent transitions observed during sampling. We propose $\pi$-StepNFT (Step-wise Negative-aware Fine-Tuning), a flow-native online RL framework that couples wider
stochastic exploration with finer step-wise supervision without computing the final-action likelihood. $\pi$-StepNFT treats observed SDE transitions as supervision units, using variance-normalized transition errors to rank mirrored positive and negative velocity branches and map rollout feedback to local velocity updates. On LIBERO, starting from shared few-shot SFT initializations, $\pi$-StepNFT raises average success from 57.6% to 92.0% for $\pi_0$ and from 77.1% to 94.0% for $\pi_{0.5}$. On ManiSkill, it achieves the highest reported mean success among the evaluated RL baselines on five test-only visual perturbations. Our project homepage is available at https://wangst0181.github.io/pi-StepNFT/.
Read the full abstract ▾
All authors Siting Wang, Xiaofeng Wang, Zheng Zhu, Minnan Pei, Xinyu Cui, Cheng Deng, Jian Zhao, Guan Huang, Haifeng Zhang, Jun Wang
Computer Vision
From Pixels to Patches: Pooling Strategies for Earth Embeddings
Isaac Corley, Caleb Robinson, Inbal Becker-Reshef et al. (+1 more)
Geospatial foundation models increasingly expose pixel-level
embedding products that can be downloaded and reused without access to the underlying encoder. In this setting, downstream tasks with patch- or region-level labels require a post-hoc aggregation step that maps dense pixel
embeddings to a single representation. The default choice, mean pooling, discards within-patch variability and can underperform under spatial distribution shift. To study this setting, we introduce EuroSAT-Embed: 81,000
embedding GeoTIFFs derived from three foundation models: AlphaEarth, OlmoEarth, and Tessera. Using these fixed
embedding products, we
benchmark 11 training-free pooling methods and 2 train-set-fitted baselines under both random and geographically disjoint test splits. Richer pooling schemes reduce the geographic generalization gap by over 50% relative to mean pooling and improve accuracy by up to 6% on spatial splits. We recommend a three-tier strategy: (1) mean as a baseline, (2) stats pooling (min/max/mean/std) as the default at 4x the
embedding dimension, and (3) covariance pooling for peak accuracy. Across all three
embedding products, simple distributional statistics improve spatial-split performance over mean pooling.
Read the full abstract ▾
All authors Isaac Corley, Caleb Robinson, Inbal Becker-Reshef, Juan M. Lavista Ferres
Computer Vision
MMNavAgent: Multi-Magnification WSI Navigation Agent for Clinically Consistent Whole-Slide Analysis
Zhengyang Xu, Han Li, Jingsong Liu et al. (+10 more)
Recent AI
navigation approaches aim to improve Whole-Slide Image (WSI) diagnosis by modeling spatial exploration and selecting diagnostically relevant regions, yet most operate at a single fixed magnification or rely on predefined magnification traversal. In clinical practice, pathologists examine slides across multiple magnifications and selectively inspect only necessary scales, dynamically integrating global and cellular evidence in a sequential manner. This mismatch prevents existing methods from modeling cross-magnification interactions and adaptive magnification selection inherent to real diagnostic workflows. To these, we propose a clinically consistent Multi-Magnification WSI
Navigation Agent (MMNavAgent) that explicitly models multi magnification interaction and adaptive magnification selection. Specifically, we introduce a Cross-Magnification
navigation Tool (CMT) that aggregates contextual information from adjacent magnifications to enhance discriminative representations along the
navigation path. We further introduce a Magnification Selection Tool (MST) that leverages
memory-driven
reasoning within the
agent framework to enable interactive and adaptive magnification selection, mimicking the sequential decision process of pathologists. Extensive experiments on a public
dataset demonstrate improved diagnostic performance, with 1.45% gain of AUC and 2.93% gain of BACC over a non-
agent baseline. Code will be public upon acceptance.
Read the full abstract ▾
All authors Zhengyang Xu, Han Li, Jingsong Liu, Linrui Xie, Xun Ma, Xin You, Shihui Zu, Ayako Ito, Xinyu Hao, Hongming Xu, Shaohua Kevin Zhou, Nassir Navab, Peter J. Schüffler
Computer Vision
ORGAN: Object-Centric Representation Learning using Cycle Consistent Generative Adversarial Networks
Joël Küchler, Ellen van Maren, Vaiva Vasiliauskaitė et al. (+3 more)
Although data generation is often straightforward, extracting information from data is more difficult. Object-centric representation learning can extract information from images in an unsupervised manner. It does so by segmenting an image into its subcomponents: the objects. Each object is then represented in a low-dimensional latent space that can be used for downstream processing. Object-centric representation learning is dominated by autoencoder architectures (AEs). Here, we present ORGAN, a novel approach for object-centric representation learning, which is based on cycle-consistent Generative
Adversarial Networks instead. We show that it performs similarly to other state-of-the-art approaches on synthetic
datasets, while at the same time being the only approach tested here capable of handling more challenging real-world
datasets with many objects and low visual contrast. Complementing these results, ORGAN creates expressive latent space representations that allow for object manipulation. Finally, we show that ORGAN scales well both with respect to the number of objects and the size of the images, giving it a unique edge over current state-of-the-art approaches.
Read the full abstract ▾
All authors Joël Küchler, Ellen van Maren, Vaiva Vasiliauskaitė, Katarina Vulić, Reza Abbasi-Asl, Stephan J. Ihle
Computer Vision
WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories
Yisu Zhang, Chenjie Cao, Tengfei Wang et al. (+4 more)
Recent advances in foundational
Video Diffusion Models (VDMs) have yielded significant progress. Yet, despite the remarkable visual quality of generated
videos, reconstructing consistent 3D scenes from these outputs remains challenging, due to limited camera controllability and inconsistent generated content when viewed from distinct camera trajectories. In this paper, we propose WorldStereo, a novel framework that bridges camera-guided
video generation and 3D reconstruction via two dedicated
geometric memory modules. Formally, the global-
geometric memory enables precise camera control while injecting coarse structural priors through incrementally updated point clouds. Moreover, the spatial-stereo
memory constrains the model's
attention receptive fields with 3D correspondence to focus on fine-grained details from the
memory bank. These components enable WorldStereo to generate multi-view-consistent
videos under precise camera control, facilitating high-quality 3D reconstruction. Furthermore, the flexible control branch-based WorldStereo shows impressive efficiency, benefiting from the distribution matching distilled VDM backbone without joint training. Extensive experiments across both camera-guided
video generation and 3D reconstruction
benchmarks demonstrate the effectiveness of our approach. Notably, we show that WorldStereo acts as a powerful world model, tackling diverse scene generation tasks (whether starting from perspective or panoramic images) with high-fidelity 3D results. Models will be released.
Read the full abstract ▾
All authors Yisu Zhang, Chenjie Cao, Tengfei Wang, Xuhui Zuo, Junta Wu, Jianke Zhu, Chunchao Guo
Computer Vision
NICO-RAG: Multimodal Hypergraph Retrieval-Augmented Generation for Understanding the Nicotine Public Health Crisis
Manuel Serna-Aguilera, Raegan Anderes, Page Dobbs et al. (+1 more)
The nicotine addiction public health crisis continues to be pervasive. In this century alone, the tobacco industry has released and marketed new products in an aggressive effort to lure new and young customers for life. Such innovations and product development, namely flavored nicotine or tobacco such as nicotine pouches, have undone years of anti-tobacco campaign work. Past work is limited both in scope and in its ability to connect large-scale data points. Thus, we introduce the Nicotine Innovation Counter-Offensive (NICO)
Dataset to provide public health researchers with over 200,000
multimodal samples, including images and text descriptions, on 55 tobacco and nicotine product brands. In addition, to provide public health researchers with factual connections across a large-scale
dataset, we propose NICO-RAG, a
retrieval-augmented generation (RAG) framework that can retrieve image features without incurring the high-cost of
language models, as well as the added cost of processing image
tokens with large-scale
datasets such as NICO. At construction time, NICO-RAG organizes image- and text-extracted entities and relations into hypergraphs to produce as factual responses as possible. This joint
multimodal knowledge representation enables NICO-RAG to retrieve images for query answering not only by visual similarity but also by the semantic similarity of image descriptions. Experimentals show that without needing to process additional
tokens from images for over 100 questions, NICO-RAG performs comparably to the state-of-the-art RAG method adapted for images.
Read the full abstract ▾
All authors Manuel Serna-Aguilera, Raegan Anderes, Page Dobbs, Khoa Luu
Computer Vision · Cross-listed from cs.RO
Fabian Schmidt, Karol Fedurko, Markus Enzweiler et al. (+1 more)
While
multimodal large
language models (MLLMs) provide advanced
reasoning for autonomous driving, translating their discrete semantic knowledge into continuous trajectories remains a fundamental challenge. Existing methods often rely on unimodal planning heads that inherently limit their ability to represent
multimodal driving behavior. Furthermore, most generative approaches frequently condition on one-hot encoded actions, discarding the nuanced navigational uncertainty critical for complex scenarios. To resolve these limitations, we introduce LAD-Drive, a generative framework that structurally disentangles high-level intention from low-level spatial planning. LAD-Drive employs an action decoder to infer a probabilistic meta-action distribution, establishing an explicit belief state that preserves the nuanced intent typically lost by one-hot encodings. This distribution, fused with the vehicle's kinematic state, conditions an action-aware
diffusion decoder that utilizes a truncated denoising process to refine learned motion anchors into safe, kinematically feasible trajectories. Extensive evaluations on the LangAuto
benchmark demonstrate that LAD-Drive achieves state-of-the-art results, outperforming competitive baselines by up to 59% in Driving Score while significantly reducing route deviations and collisions. We will publicly release the code and models on https://github.com/iis-esslingen/lad-drive.
Read the full abstract ▾
All authors Fabian Schmidt, Karol Fedurko, Markus Enzweiler, Abhinav Valada
Computer Vision
Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT
Simon Ging, Philipp Arnold, Sebastian Walter et al. (+6 more)
Recent 3D CT
vision-
language models align volumes with reports via contrastive pretraining, but typically rely on limited public data and provide only coarse global supervision. We train a 3D CT
vision-
language model on 98k report-volume pairs (50k patients) collected at a single hospital, combined with public
datasets, using SigLIP-style contrastive pretraining together with
prompt-based disease supervision in the shared
vision-text
embedding space. On CT-RATE, our model achieves state-of-the-art text-to-image
retrieval (R@10 31.5 vs. 22.2) and competitive disease classification (AUC 83.8 vs. 83.8), with consistent results on Rad-ChestCT (AUC 77.0 vs. 77.3). We further observe that radiologists routinely reference specific images within their reports (e.g., “series X, image Y”), linking textual descriptions to precise axial locations. We automatically mine 262k such snippet-slice pairs and introduce the task of intra-scan snippet localization – predicting the axial depth referred to by a text snippet – reducing mean absolute error to 36.3 mm at 12 mm feature resolution, compared with 67.0 mm for the best baseline. Adding this localization objective leaves
retrieval and classification broadly unchanged within confidence bounds, yielding a single unified model for
retrieval, classification, and intra-scan grounding.
Read the full abstract ▾
All authors Simon Ging, Philipp Arnold, Sebastian Walter, Hani Alnahas, Hannah Bast, Elmar Kotter, Jiancheng Yang, Behzad Bozorgtabar, Thomas Brox
Computer Vision · Cross-listed from cs.CL
MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning
Jiachun Li, Shaoping Huang, Zhuoran Jin et al. (+5 more)
Recent progress in the
reasoning capabilities of
multimodal large
language models (MLLMs) has empowered them to address more complex tasks such as scientific analysis and mathematical
reasoning. Despite their promise, MLLMs'
reasoning abilities across different scenarios in real life remain largely unexplored and lack standardized
benchmarks for evaluation. To address this gap, we introduce MMR-Life, a comprehensive
benchmark designed to evaluate the diverse
multimodal multi-image
reasoning capabilities of MLLMs across real-life scenarios. MMR-Life consists of 2,646 multiple-choice questions based on 19,108 images primarily sourced from real-world contexts, comprehensively covering seven
reasoning types: abductive, analogical, causal, deductive, inductive, spatial, and temporal. Unlike existing
reasoning benchmarks, MMR-Life does not rely on domain-specific expertise but instead requires models to integrate information across multiple images and apply diverse
reasoning abilities. The evaluation of 37 advanced models highlights the substantial challenge posed by MMR-Life. Even top models like GPT-5 achieve only 58% accuracy and display considerable variance in performance across
reasoning types. Moreover, we analyze the
reasoning paradigms of existing MLLMs, exploring how factors such as thinking length,
reasoning method, and
reasoning type affect their performance. In summary, MMR-Life establishes a comprehensive foundation for evaluating, analyzing, and improving the next generation of
multimodal reasoning systems.
Read the full abstract ▾
All authors Jiachun Li, Shaoping Huang, Zhuoran Jin, Chenlong Zhang, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
Computer Vision
MAP-Diff: Multi-Anchor Guided Diffusion for Progressive 3D Whole-Body Low-Dose PET Denoising
Peiyuan Jing, Chun-Wun Cheng, Liutao Yang et al. (+7 more)
Low-dose Positron Emission
Tomography (PET) reduces radiation exposure but suffers from severe noise and quantitative degradation.
Diffusion-based denoising models achieve strong final reconstructions, yet their reverse trajectories are typically unconstrained and not aligned with the progressive nature of PET dose formation. We propose MAP-Diff, a multi-anchor guided
diffusion framework for progressive 3D whole-body PET denoising. MAP-Diff introduces clinically observed intermediate-dose scans as trajectory anchors and enforces timestep-dependent supervision to regularize the reverse process toward dose-aligned intermediate states. Anchor timesteps are calibrated via degradation matching between simulated
diffusion corruption and real multi-dose PET pairs, and a timestep-weighted anchor loss stabilizes stage-wise learning. At inference, the model requires only ultra-low-dose input while enabling progressive, dose-consistent intermediate restoration. Experiments on internal (Siemens Biograph
Vision Quadra) and cross-scanner (United Imaging uEXPLORER)
datasets show consistent improvements over strong CNN-,
Transformer-, GAN-, and
diffusion-based baselines. On the internal
dataset, MAP-Diff improves PSNR from 42.48 dB to 43.71 dB (+1.23 dB), increases SSIM to 0.986, and reduces NMAE from 0.115 to 0.103 (-0.012) compared to 3D DDPM. Performance gains generalize across scanners, achieving 34.42 dB PSNR and 0.141 NMAE on the external cohort, outperforming all competing methods.
Read the full abstract ▾
All authors Peiyuan Jing, Chun-Wun Cheng, Liutao Yang, Zhenxuan Zhang, Thiago V. Lima, Klaus Strobel, Antoine Leimgruber, Angelica Aviles-Rivero, Guang Yang, Javier A. Montoya-Zegarra