Computer Vision · Cross-listed from cs.RO
PointAction: 3D Points as Universal Action Representations for Robot Control
Mutian Tong, Han Jiang, Qiao Feng et al. (+2 more)
Video-Action Models (VAMs) leverage the broad visual dynamics captured by pre-trained
video diffusion models, offering a promising path toward generalizable
robot manipulation. However, RGB-only
video rollouts are not directly actionable: they leave metric 3D motion, contact geometry, and fine-grained spatial constraints under-specified, making action grounding ambiguous. Meanwhile, scaling action supervision across diverse tasks and embodiments remains costly. We present PointAction, a framework that bridges
video predictions to
robot actions through explicit point-based 4D modeling. PointAction fine-tunes a foundation
video generation model to jointly predict future RGB frames and dynamic 3D pointmaps, producing temporally consistent 3D motion of task-relevant scene geometry. These point dynamics serve as a structured, embodiment-agnostic action interface, which a
diffusion-based action decoder maps to executable
robot actions. By using metric 3D point dynamics as the interface between
video prediction and control, PointAction reduces the ambiguity of RGB-only action grounding and supports transfer across tasks and embodiments with limited action supervision. Experiments show that PointAction achieves state-of-the-art 4D generation quality on
robot scenes, outperforms existing baselines in simulation, and generalizes to two real
robot arms unseen during pretraining.
Read the full abstract ▾
All authors Mutian Tong, Han Jiang, Qiao Feng, Lingjie Liu, Jiatao Gu
Computer Vision · Cross-listed from eess.IV
SEAOTTER: Sensor Embedded Autoencoding with One-Time Transcode for Efficient Reconstruction
Dan Jacobellis and Neeraja J. Yadwadkar
In robotics systems, vast amounts of visual data are easily captured at high resolution using low-cost, low-power hardware. Yet, limited bandwidth and on-device compute resources prevent full utilization when transmitted via conventional codecs like JPEG/MPEG. Newer codecs, like AV1/AVIF, improve the rate-distortion trade-off, but demand far more resources for encoding, impractical without custom ASICs. Recent asymmetric autoencoders deliver high quality under extreme power and bandwidth constraints, but add prohibitive decoding cost and use bespoke formats that ignore decades of infrastructure built around standards like JPEG. To address these limitations, we introduce a compression framework for cloud robotics based on a Sensor Embedded Autoencoder paired with a One-Time Transcode for Efficient Reconstruction (SEAOTTER). Because the sensor, cloud, and consumer stages face very different power and bandwidth budgets, SEAOTTER combines the compactness of a learned latent with the broad usability of a standard JPEG file. Since naive transcoding degrades performance, we propose a learnable JPEG color and quantization transform that enables increased accuracy for global, dense, and
vision-
language-based perception. Using SEAOTTER, we train both general-purpose and task-aware transcoding pipelines for a pre-trained, frozen encoder. At a compression ratio of 200:1 and compared to AVIF, we observe 7 times faster encoding, 3.5 times faster decoding, and +8% ImageNet top-1 accuracy, while retaining compatibility with JPEG infrastructure. Our code is available at https://github.com/UT-SysML/seaotter .
Read the full abstract ▾
Computer Vision
Adaptive Causal Alignment for High-Confidence Adversarial Training
Zhiming Luo, Kejia Zhang, Yingxin Lai et al. (+3 more)
Inverse
adversarial training leverages high-confidence predictions to stabilize robust learning, yet we uncover a critical paradox: high confidence often stems from overfitting to non-causal background correlations rather than intrinsic object semantics. Our investigation reveals that visual context functions as a dual-natured signal, serving as either a necessary supportive prior or a spurious confounder. This insight renders existing blind suppression strategies flawed, as they inevitably lead to severe Feature Loss. To resolve this, we propose High-Confidence Causally Aligned Training (HICAT), a unified framework that establishes a Semantic
Equilibrium. Operating on a “Measure-Debias-Align” pipeline, HICAT integrates a Learnable Background-Bias
Estimator (LBBE) to adaptively diagnose context utility. Guided by this diagnosis, an Adaptive Debiasing mechanism performs
surgical logit rectification, complemented by a geometrically grounded Foreground Logit Orthogonal Enhancement (FLOE) loss to enforce rigorous feature disentanglement. Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet-1K demonstrate that HICAT consistently improves over matched baselines across diverse architectures (CNNs and ViTs) while significantly reducing the robust generalization gap.
Read the full abstract ▾
All authors Zhiming Luo, Kejia Zhang, Yingxin Lai, Junwei Wu, Juanjuan Weng, Shaozi Li
Computer Vision
GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB images
Jiahao Sun, Dingkun Wei, Zehong Shen et al. (+3 more)
Converting multi-view RGB observations into simulation-ready 3D environments remains challenging because current reconstruction pipelines produce monolithic scene representations without explicit physical structure. They are typically defined up to an arbitrary global rotation and entangle rigid foreground objects with background geometry, which hinders stable physical interaction. Existing solutions often recover interactivity by replacing reconstructed objects with retrieved CAD assets, but this introduces a slow
retrieval-and-replacement stage and weakens scene-specific
geometric fidelity. We propose GARDEN, an RGB-only framework that reformulates reconstruction as physically-grounded scene factorization and outputs a structured hybrid scene representation. The key idea is to use
gravity as a universal physical prior: we first align the reconstruction to a unified
Gravity-View frame to resolve gauge ambiguity, then recover object-centric rigid meshes with accurate 6-DoF placement, and finally remove duplicate object geometry from the background through conditional 3D point classification. The resulting representation combines explicit rigid bodies with a decoupled background, enabling direct physics simulation while preserving visual realism. Experiments on both simulated and real multi-view scenes show that GARDEN improves object placement reliability, disentanglement quality, and rendering-simulation efficiency compared with
retrieval-based baselines. Project page: https://sunjiahaovo.github.io/garden/
Read the full abstract ▾
All authors Jiahao Sun, Dingkun Wei, Zehong Shen, Hongyu Zhou, Yujun Shen, Liang Li
Computer Vision
Benchmarking Visual State Tracking in Multimodal Video Understanding
Sihyun Yu, Nanye Ma, Pinzhi Huang et al. (+8 more)
Understanding a
video requires more than recognizing isolated moments, as humans continuously track entities, states, and events over time. This capacity for visual state tracking is fundamental to
video understanding, yet remains underexplored in current evaluations of
Multimodal Large
Language Models (MLLMs). We introduce Visual STAte Tracking
benchmark (VSTAT), a
video-based
benchmark designed to diagnose visual state tracking in MLLMs. VSTAT consists of 834 clips drawn from both synthetic and real-world
videos, paired with 1,500 questions that cannot be answered from any single frame or short segment, requiring continuous perception and integration of events across the entire
video stream. Despite their strong performance on existing
video benchmarks, we find that state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines. To analyze this gap, we compare MLLMs' thinking traces with the underlying
video stream to understand why and when MLLMs fail on VSTAT. We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track. Finally, our preliminary evaluation suggests that recent agentic approaches, including MLLM-based
video agents and coding
agents, do not readily resolve these failures, still falling short on VSTAT.
Read the full abstract ▾
All authors Sihyun Yu, Nanye Ma, Pinzhi Huang, Hyunseok Lee, Shusheng Yang, June Suk Choi, Ellis Brown, Oscar Michel, Boyang Zheng, Jinwoo Shin, Saining Xie
Computer Vision
PatchScene: Patch-based Voxel Diffusion for Large-Scale Scene Completion
Qingdong Xu, Jiajun Zhu, Shilin Zhu et al. (+4 more)
We propose PatchScene, a novel
diffusion-based framework for large-scale
LiDAR scene completion. Unlike existing methods that rely on global latent representations or dense voxel grids, PatchScene adopts a patch-based voxel
diffusion paradigm that explicitly generates fine-grained geometry within localized 3D regions. To ensure coherent reconstruction at both spatial and temporal scales, we introduce a confidence-guided spatio-temporal fusion mechanism that integrates overlapping patches and adjacent frames in a unified generative process. Furthermore, we design an Annular-Flow
diffusion strategy that leverages the radial density pattern of
LiDAR scans to progressively propagate high-fidelity information from near-range to far-range regions, enabling spatially unbounded scene completion. Extensive experiments on the SemanticKITTI
benchmark demonstrate that PatchScene achieves state-of-the-art performance across all standard metrics, surpassing previous approaches in both
geometric accuracy and temporal consistency. Remarkably, the model trained on 20 m
LiDAR ranges generalizes effectively to 50 m scenes without retraining, highlighting its strong scalability and generalization capability for real-world autonomous driving applications. Project page: https://patchscene.github.io/
Read the full abstract ▾
All authors Qingdong Xu, Jiajun Zhu, Shilin Zhu, Xinjing He, Chao Lu, Huanran Wang, Jiyao Zhang
Computer Vision
Bootstrap Your Generator: Unpaired Visual Editing with Flow Matching
Yoad Tewel, Yuval Atzmon, Gal Chechik et al. (+1 more)
Modern generative models possess a deep understanding of visual content, yet training them for image editing typically requires massive
datasets of paired examples. This limits scalability, especially for
video editing where collecting paired data is prohibitively expensive. We propose Bootstrap Your Generator (ByG), a general framework for unpaired training of flow matching editing models. It leverages the base model's knowledge without any external signal. Our approach pairs instruction-following cues extracted from the frozen model with cycle-consistency for structure preservation. To make this tractable, we propose to route gradients from downstream losses over clean predictions to noisy training states. We demonstrate state-of-the-art results on challenging data-scarce image and
video editing scenarios. Extensive evaluations and user studies show that our method effectively generalizes to unseen domains and outperforms supervised baselines trained on millions of samples. Analysis reveals that our gradient routing bridges the train-inference gap, and extracting semantic cues from a base model provides a robust training signal that obviates the need for external reward models.
Read the full abstract ▾
All authors Yoad Tewel, Yuval Atzmon, Gal Chechik, Lior Wolf
Computer Vision
SparseStreet: Sparse Gaussian Splatting for Real-Time Street Scene Simulation
Qingpo Wuwu, Xiaobao Wei, Peng Chen et al. (+6 more)
While 3D Gaussian Splatting has shown promising results in street scene reconstruction, existing methods require massive numbers of Gaussian primitives to capture fine details, leading to prohibitive storage costs and slow rendering speeds. We observe that dynamic objects (e.g., vehicles and pedestrians) demand high-fidelity representations to maintain temporal consistency, while static background regions often contain substantial redundancy. Motivated by this, we propose SparseStreet, a general compression framework specifically designed for street scenes. First, we introduce a node-based learnable
pruning strategy that systematically removes low-contributing Gaussian primitives while preserving visually critical regions. Second, after the scene representation stabilizes, we apply background compression, further reducing redundancy in static regions. Our method effectively preserves the geometry and appearance of dynamic objects while significantly reducing the total number of Gaussian primitives. Extensive experiments on the Waymo and nuScenes demonstrate that SparseStreet achieves up to 80% compression ratio with minimal quality degradation, enabling resource-efficient, high-fidelity dynamic scene reconstruction. Project website: https://sparsestreet.github.io/.
Read the full abstract ▾
All authors Qingpo Wuwu, Xiaobao Wei, Peng Chen, Nan Huang, Zhongyu Zhao, Hao Wang, Ming Lu, Ningning Ma, Shanghang Zhang
Computer Vision · Cross-listed from cs.LG
MAdam: Metric-Aware Multi-Objective Adam
Fengbei Liu, Rachit Saluja, Sunwoo Kwak et al. (+5 more)
Multi-objective optimization (MOO) underlies many machine learning problems, yet MOO solvers across the loss-balancing, gradient-balancing, and Pareto-based families almost universally hand their reconciled directions to Adam~\citep{kingma2015adam}. We show this coupling introduces two systematic gaps between the solver's intent and the optimizer's execution. The first is a weighting mismatch: Adam's second-moment denominator entangles the time-varying preference vector with gradient statistics, marginalizing the preference into a history average and collapsing distinct Pareto trade-offs toward a near-uniform mixture. The second is a
geometric mismatch: Adam's adaptive metric distorts the Euclidean geometry MOO solvers assume, turning aligned objectives into apparent conflicts. To resolve both jointly, we introduce MAdam (Metric-Aware Multi-Objective Adam), a drop-in wrapper that leaves both solver and optimizer unchanged. MAdam preconditions the reconciled direction by the preference-conditioned
curvature of the scalarized objective; on this whitened input, Adam's second moment collapses to identity, so the realized update is governed by the preference-conditioned metric. Across multi-task learning, Pareto-front recovery, physics-informed neural networks, and medical imaging, MAdam improves over Adam in aggregate for every solver family. Our code is available at https://github.com/lfb-1/MAdam.
Read the full abstract ▾
All authors Fengbei Liu, Rachit Saluja, Sunwoo Kwak, Ruibo Wang, Ruining Deng, Heejong Kim, Johannes C. Paetzold, Mert R. Sabuncu
Computer Vision
An Attention-Based Denoising Model for Diffusion Weighted Imaging
Prithviraj Verma, Pawan Kumar, Chandan Deshani et al. (+1 more)
Diffusion-weighted imaging (DWI) is used for whole-body cancer screening, but it typically requires a long acquisition time. When the scan time is reduced, the image quality often suffers, leading to increased noise in the scans. Magnitude reconstruction in DWI introduces signal-dependent Rician noise, which makes denoising more challenging for conventional convolution-based methods. To address this limitation, we propose a noise-aware
attention-driven denoising framework that integrates hierarchical Swin
Transformer window
attention with
transformer-based multi-dimensional gated refinement for DWI restoration. The model incorporates explicit noise-level conditioning and residual reconstruction to enable adaptive suppression of heteroscedastic noise across a wide range of corruption levels. Experimental evaluation on corrupted DWI scans demonstrates strong restoration performance. Our model achieves a mean PSNR of 33.69~dB and SSIM of 0.8539 across noise levels from 1% to 15%, while maintaining stable behavior under severe noise conditions. These results indicate that
attention-guided contextual modeling combined with channel-adaptive refinement provides a robust and generalizable solution for DWI denoising.
Read the full abstract ▾
All authors Prithviraj Verma, Pawan Kumar, Chandan Deshani, Prasun Chandra Tripathi
Computer Vision
Electromagnetic Navigation for Femoral Osteotomy Using High-Accuracy X-ray-to-CT Registration
Roman Flepp, Arend Nieuwland, Bastian Sigrist et al. (+3 more)
Accurate execution of preoperative plans in corrective femoral osteotomies remains challenging. Current techniques are limited by variable accuracy, invasiveness, and radiation exposure, with free-hand methods and patient-specific instrumentation (PSI) often requiring >30 and >6 fluoroscopic images, respectively. We present an integrated, electromagnetic tracking (EMT)-based
navigation system for femoral osteotomies that minimizes dissection and intraoperative fluoroscopy. The system couples CT-based preoperative planning with one-time intraoperative C-arm calibration and accurate X-ray-to-CT registration from two fluoroscopic images acquired at initialization. This enables real-time, fluoroscopy-free EMT
navigation of the saw blade and bone fragments relative to the preoperative plan, and is compatible with uniplanar and biplanar osteotomies. In a feasibility study using 18 synthetic femora, EMT guidance significantly outperformed free-hand execution in total angular error ($(3.05 \pm 0.75)^\circ$ vs.\ $(6.32 \pm 2.36)^\circ$, $p=0.031$), assuming the same minimal
surgical exposure for both. No EMT-guided trials exceeded the >5{\deg} clinical threshold, whereas free-hand produced 4 outliers of 6 trials. The system achieved statistical equivalence ($\pm 2^\circ$, $\pm 2,\text{mm}$) to PSI for total angular ($p \le 0.02$) and total translational ($p=0.048$) errors, with no significant differences in user questionnaire scores. By transferring preoperative plans using only two fluoroscopic images while matching PSI accuracy without additional
surgical exposure, the proposed system motivates subsequent cadaveric and clinical validation.
Read the full abstract ▾
All authors Roman Flepp, Arend Nieuwland, Bastian Sigrist, Philipp Fürnstahl, Lilian Calvet, Thomas Dreher
Computer Vision
OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs
Yifei Li, Pengyiang Liu, Yuhang Zang et al. (+4 more)
Multimodal agents in robotics, AR, and autonomous driving must reason about places and layouts from continuous
egocentric streams, often using evidence outside the current view. Existing
benchmarks either evaluate offline over full
videos or target events rather than spatial structure. We introduce OVO-S-Bench, a fully human-annotated
benchmark for streaming spatial intelligence, comprising 1,680 questions over 348 source
videos. Annotation involves 12 trained annotators (each also serving as a blind cross-reviewer) across roughly 804 person-hours of multi-round quality assurance. Each question carries a query timestamp and an evidence interval, and at evaluation, the model sees only the prefix preceding the query. Questions span four levels of increasing abstraction: instantaneous
egocentric perception, spatiotemporal context tracking, generative spatial
reasoning, and allocentric spatial mapping. Across 38 proprietary and open-source MLLMs, Gemini-3.1-Pro trails human experts evaluated under the same prefix-access protocol by 33 points (59.2 vs. 92.2), with allocentric spatial mapping as the dominant bottleneck. Notably, streaming and spatially fine-tuned MLLMs underperform their own backbones. We further find that chain-of-thought
reasoning amplifies spatial errors when ungrounded in the stream. By exposing these limitations, OVO-S-Bench establishes a demanding testbed for next-generation streaming spatial MLLMs.
Read the full abstract ▾
All authors Yifei Li, Pengyiang Liu, Yuhang Zang, Zhongyue Shi, Qi Fu, Hongye Hao, Jiwen Lu
Computer Vision
CoralBay: A Self-Supervised CT Foundation Model
Ioannis Gatopoulos, Nicolas Känzig, Sebastian Otálora et al. (+1 more)
Self-supervised learning has enabled large-scale pre-training on 2D natural images, producing general-purpose visual representations that transfer effectively across tasks. However, many medical imaging modalities, such as CT scans, are inherently three-dimensional and differ fundamentally from natural images in both structure and semantics. Volumetric modalities capture spatial continuity, organ anatomy, and intensity-based tissue properties (e.g., Hounsfield Units), which are not adequately modeled by 2D pre-training. To bridge this gap, we introduce CoralBay, a self-
distillation framework that extends DINO by using a hierarchical 3D Swin backbone and applying self-
distillation to concatenated multi-scale features, enabling data-efficient self-supervised learning of rich spatial representations that encode both global semantics and fine-grained local structure. As a result, CoralBay transfers effectively to a wide range of downstream radiological tasks, demonstrating strong and consistent performance across diverse anatomical targets. In addition, we contribute to the open-source \eva framework by introducing a public, reproducible 3D radiology leaderboard that unifies multiple
datasets and establishes a standardized
benchmark for evaluating volumetric representation learning methods.
Read the full abstract ▾
All authors Ioannis Gatopoulos, Nicolas Känzig, Sebastian Otálora, Fei Tang
Computer Vision
Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs
Wei Ding, Yudong Zhang, Ruobing Xie et al. (+3 more)
As foundation models scale toward fusing more heterogeneous visual streams, understanding how diverse encoders interact under joint training becomes a prerequisite for principled design. Yet large
vision-
language models (LVLMs) currently lack the tools to do so, and parameter-efficient encoder configurations remain hard to identify before training. To re-examine encoder roles under joint training, on the 16-
benchmark Cambrian-1 suite we retrain and evaluate all 31 non-empty subsets of five common
vision encoders under a unified pipeline (~20k GPU-hours total), and report three findings. First, retraining each subset from scratch reveals encoder rankings that differ from those obtained by masking encoders on a fixed checkpoint, including which encoder ranks first overall. Second, we decompose each encoder's contribution into two axes, Capacity, the score an encoder reaches on its own, and Necessity, the drop when it is removed from the full pool. The two axes are not interchangeable. Pairing the two highest-Capacity encoders is suboptimal, while pairing a high-Capacity anchor with an adaptive complement matches the full five-encoder model. Adding further encoders beyond this pair yields only marginal gains. Third, at fixed parameter count, per-encoder pre-projector effective rank explains the residual score variation. The strongest pairs combine an anchor whose rank survives joint training with a complement whose rank expands under it, suggesting that higher-rank, less-collapsed projector inputs correspond to a more favorable optimization regime at the encoder-projector interface. Together, the Capacity-Necessity decomposition and the pre-projector rank analysis, along with comprehensive evaluation through retraining, expose a methodological gap in multi-encoder LVLM design, and offer concrete primitives for closing it.
Read the full abstract ▾
All authors Wei Ding, Yudong Zhang, Ruobing Xie, Xingwu Sun, Jiansheng Chen, Yu Wang
Computer Vision
MLP Splatting: Object-Centric Neural Fields
Shinjeong Kim, Yuzhou Cheng, Xin Kong et al. (+2 more)
3D representations are fundamental for scene rendering, understanding, and interaction. Approaches like Neural Radiation Fields and 3D Gaussian Splatting achieve impressive photorealistic novel-view synthesis, but lack the ability to easily decompose scene elements into object level, requiring additional
segmentation or grouping. We present MLP-Splatting, a method that enables scene decomposition via a few expressive light-field primitives while providing photorealistic novel-view synthesis. MLP-Splatting models each primitive as an independent compact MLP with localized spatial support that predicts radiance and opacity. In contrast to low-level Gaussian primitives or a single global radiance field, our neural primitives provide greater expressive capacity while remaining spatially localized. Rendering is performed through efficient sparse volumetric compositing over ray-primitive interactions. Our primitives are supervised using RGB supervision alone, which yields primitives that represent local scene regions often corresponding to objects or object parts, enabling interactive object-level editing without
segmentation masks by selecting a handful of primitives. Our method, augmented with optional semantic feature
distillation, enables open-vocabulary scene interaction and open-set instant
segmentation. Compared to state-of-the-art semantic 3DGS methods, we achieve substantially lower
memory usage (1/7$\times$) and faster rendering (5$\times$). Project Page: https://shinjeongkim.com/mlp-splatting
Read the full abstract ▾
All authors Shinjeong Kim, Yuzhou Cheng, Xin Kong, Paul H. J. Kelly, Andrew J. Davison
Computer Vision
Seg2Track++: Probabilistic Track Validation and Data Association for Multi-Object Tracking and Segmentation
Diogo Mendonça, Tiago Barros, Cristiano Premebida et al. (+1 more)
Autonomous systems require robust Multi-Object Tracking and
Segmentation (MOTS) to operate reliably in dynamic environments, ensuring consistent object identities and precise mask-level delineation. Foundation models such as SAM2 have shown strong zero-shot generalization for
segmentation, but their direct application to MOTS is limited by unreliable track association and false-positive propagation. This work introduces Seg2Track++, a framework that integrates instance
segmentation with SAM2 and a novel track management module to perform zero-shot MOTS with enhanced temporal consistency. Tracks are associated using Mask Centroid Distance (MCD) and Confidence-Aware Cost Modulation (CCM), while Probabilistic Track Validation (PTV) employs a Bernoulli filter to validate track existence and suppress ghost tracks. Experimental results on KITTI MOTS demonstrate improved identity preservation, reduced false-positive propagation, and robust track management without fine-tuning.
Read the full abstract ▾
All authors Diogo Mendonça, Tiago Barros, Cristiano Premebida, Urbano J. Nunes
Computer Vision
DyaPlex: Full-Duplex Speech-Motion Model for Dyadic Interaction
Koki Nagano, Hongyu Liu, Seonwook Park et al. (+9 more)
We present DyaPlex, a streaming, full-duplex
speech-and-motion model designed for dyadic interaction. To capture the continuous and reciprocal nature of human communication, this full-duplex capability empowers the
agent to simultaneously perceive and generate both
speech and physical motion in a streaming fashion. At its core, our method leverages the strong priors of a foundational full-duplex
speech model and integrates a novel motion pathway, thereby achieving fully synchronized multi-modal interaction. Specifically, we design a dual-tower
Transformer architecture that preserves the zero-shot conversational
reasoning of a frozen base
speech model while constructing a deeply coupled, streaming motion pathway. By introducing a unified dyadic
token interleaving mechanism and guiding cross-
attention via a time-aligned
speech-motion RoPE, our model effectively aligns autoregressive motions with rich latent
speech features. Trained on the 4,000-hour Seamless Interaction
dataset, our model effectively captures cross-speaker dependencies and establishes new state-of-the-art performance across both monadic and dyadic human interaction
benchmarks.
Read the full abstract ▾
All authors Koki Nagano, Hongyu Liu, Seonwook Park, Tianye Li, Amrita Mazumdar, Christian Jacobsen, Shengze Wang, Michael Stengel, Rajarshi Roy, Ka Chun Cheung, Simon See, Shalini De Mello
Computer Vision
Visual Instruction Tuning Aligns Modalities through Abstraction
Luis Palacios, Lorenzo Basile, Diego Doimo et al. (+1 more)
Visual instruction tuning effectively adapts a pre-trained Large
Language Model (LLM) to process image information alongside text. Yet, it remains unclear how visual features are embedded into the layer-wise hierarchy of abstractions of the LLM backbone. Across a diverse set of
vision-
language architectures, we show that instruction tuning primarily serves as a bridge,
embedding visual features directly into the intermediate semantic layers of the LLM, bypassing the early layers devoted to unimodal processing. With probing analyses and causal interventions, we show that these intermediate layers are the semantic core of
vision-
language processing and play a critical role in the performance on a broad set of
multimodal benchmarks. In addition, by comparing the geometry of semantically equivalent visual and textual representations, we find that fine-tuning extends and strengthens the existing abstraction phase, aligning visual features with pre-existing textual ones. Finally, we confirm the functional role of this localized
alignment by restricting fine-tuning to intermediate layers alone: this strategy preserves the performance of full fine-tuning on
vision-centric
benchmarks while reducing training time. Our results suggest that
multimodal integration is a localized phenomenon driven by the repurposing of the internal abstraction engine of the LLM.
Read the full abstract ▾
All authors Luis Palacios, Lorenzo Basile, Diego Doimo, Alberto Cazzaniga
Computer Vision
Unified Video-Action Joint Denoising for Dexterous Action and Data Generation
Dingrui Wang, YuAn Wang, Jinkun Liu et al. (+4 more)
Recent world action models leverage
video foundation models by aligning broad visual-dynamics priors with executable
robot actions. We revisit this
alignment from a distributional perspective. Existing formulations typically narrow the aligned prior into an observation-conditioned policy distribution over future actions. In contrast, we keep the distribution broader by modeling the joint space of interaction
videos and executable hand trajectories under multiple conditioning regimes. We propose Donk, a unified
video-action denoising model for dexterous hands. With
language, an initial image, and the initial hand state, Donk samples future
videos and bimanual MANO trajectories as an action policy. Without the image condition, the same denoising architecture samples paired
video-action rollouts from a text-conditioned distribution, turning the aligned
video prior into a data engine. Across action,
video, and text-only generation evaluations, Donk improves dexterous trajectory accuracy, preserves strong
video fidelity, and produces smooth text-conditioned action rollouts under the same unified training recipe.
Read the full abstract ▾
All authors Dingrui Wang, YuAn Wang, Jinkun Liu, Yue Zhang, Mattia Piccinini, Yu Sun, Johannes Betz
Computer Vision
Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?
Luc P.J. Sträter and Hazel Doughty
Parameter-efficient finetuning (PEFT) and probing enable adaptation of foundation models using only a small number of trainable parameters. This makes these techniques attractive for
video understanding where annotation and computation are expensive. However,
video PEFT has focused on adapting image-pretrained models, while standard PEFT methods can also be applied to
video representations. These settings are rarely compared and both confine temporal
reasoning to a single component of the model, leaving open how temporal context should be distributed across backbone, PEFT, and probe. In this work, we provide a systematic study of model adaptation strategies for
video understanding. We evaluate methods across appearance-focused, motion-focused and spatially dense settings, with a particular emphasis on scenarios with limited data where parameter efficiency is most beneficial. Our results provide new insights into PEFT and probing across task types and demonstrate the importance of temporal context allocation for effective
video adaptation.
Read the full abstract ▾
Computer Vision
Conditional Latent Diffusion Model with Fourier-based Motion Modelling for Virtual Population Synthesis
Shaokun Lan, Haoran Dou, Jinghan Huang et al. (+5 more)
In-silico trials of medical devices require the generation of virtual populations of anatomies. In cardiovascular applications, virtual anatomy is typically represented as a 3D+t mesh sampled from a generative model. However, most existing mesh generators focus on static anatomy, while sequence models often lack explicit periodicity. To this end, we propose 4D F-MeshLDM, a conditional generative framework comprising a convolutional mesh VAE to encode meshes, a structural latent space that parameterises motion using a truncated
Fourier series, and a
diffusion prior that learns the latent distribution over
Fourier coefficient
tokens. By conditioning the
diffusion process on clinical
covariates via affine modulation, we enable controllable synthesis. Sampling
tokens and performing inverse
Fourier synthesis yield cycle-consistent latent trajectories, which can be decoded into 3D+t cardiac mesh sequences. Experiments on 5,000 UK Biobank subjects demonstrate that 4D F-MeshLDM outperforms state-of-the-art baselines in anatomical fidelity and achieves near-zero cycle closure error. Furthermore, the generated cohorts accurately preserve clinical functional indices, highlighting the potential of our framework for reliable in-silico cardiac trials.
Read the full abstract ▾
All authors Shaokun Lan, Haoran Dou, Jinghan Huang, Arezoo Zakeri, Fengming Lin, Zherui Zhou, Jinming Duan, Alejandro F. Frangi
Computer Vision
TeX-1500: A Paired Real-World LWIR Hyperspectral Dataset and Benchmark for Temperature-Emissivity-Texture Decomposition
Cheng Dai, Jiale Lin, Hongyi Xu et al. (+3 more)
Temperature-emissivity-texture (TeX) decomposition seeks to recover object heat state, material spectral response, and visible-like
geometric texture from long-wave infrared hyperspectral imaging (LWIR HSI). Existing TeX pipelines are mainly scene-specific inverse solvers, and the lack of paired LWIR HSI-TeX supervision has limited learning-based decomposition. To address this gap, we introduce TeX-1500, a large-scale paired LWIR HSI-TeX
dataset and
benchmark for supervised HSI-to-TeX decomposition. TeX-1500 contains 1,522 calibrated real-scene pairs from DARPA Invisible Headlights (DARPA IH) pushbroom imagery and our FTIR acquisitions, covering five locations, four seasons, diverse acquisition times, heterogeneous wavelength layouts, and two sensor families. Each sample stores a calibrated valid-band radiance cube, calibrated wavelength positions, and aligned temperature, emissivity, and texture supervision constructed through a consistent restoration and TeX-construction protocol. We further provide TeX-UNet, a simple wavelength-aware baseline that maps calibrated HSI bands and wavelength positions to TeX fields. Experiments on the held-out DARPA IH pushbroom scenes and zero-/few-shot transfer to FTIR scenes show that TeX-1500 provides usable paired supervision and a measurable
benchmark for data-driven physical-property-centered thermal perception.
Read the full abstract ▾
All authors Cheng Dai, Jiale Lin, Hongyi Xu, Bingxuan Song, Ziyang Xie, Fanglin Bao
Computer Vision
Template Collapse and Information-Theoretic Limits in Camera rPPG Pulse Morphology Restoration
Achraf Ben Ahmed
Objective: Consumer face camera remote photoplethysmography (rPPG) enables passive cardiovascular monitoring, but whether single-cycle waveform morphology encoding arterial stiffness biomarkers is recoverable from this measurement has not been characterised. Methods: We evaluated 16 architectures spanning six families on 153 subjects across three
datasets, introducing cross-subject Pearson r to distinguish subject-specific recovery from template collapse. Results: No architecture recovered subject-specific morphology (cross-subject r range 0.773–0.9999; ground-truth ceiling 0.601). Supervised Contrastive (SupCon) converged to log N = 4.844, constituting the strongest available empirical evidence that no discriminative morphological structure is extractable from single-cycle rPPG by the encoder families tested. The VAE decoder restores population-level harmonic content absent from the rPPG input (H2/H1: 0.310 output vs. 0.275 input), generalising zero-shot to UBFC (r = +0.708); a directional
hallucination gap (p = 0.150) suggests partial signal reading. Anti-collapse objectives fail when input carries no discriminative structure. Significance: Consumer cameras cannot encode individual arterial morphology; cross-subject r is a necessary collapse diagnostic for waveform reconstruction
benchmarks.
Read the full abstract ▾
Computer Vision
Beyond Compression: Quantifying Spectral Accessibility in Vision Representations
Akayou A. Kitessa and Yijun Zhao
Vision-
language models map visual features into a shared
embedding space through learned projection layers, yet it remains unclear how these transformations alter the structure of visual information. This study examines changes in representation through spatial-frequency accessibility, measured by the linear recoverability of band-limited
Fourier energy from model representations. To isolate effects beyond dimensionality reduction, we introduce Residual Spectral Loss (RSL), which evaluates changes relative to a dimension-matched random projection baseline. To reduce confounding effects from optimization, the analysis uses pretrained models with all parameters frozen. The experimental results show consistent frequency-dependent changes in accessibility across CLIP and DINOv2 on ImageNet and MS-COCO
datasets. Spectral accessibility follows a non-monotonic trajectory across depth, peaking at intermediate layers before decreasing toward the output representation. The final transformation differs across architectures: CLIP's learned projection is spectrally neutral, with changes explained by compression, whereas DINOv2's [CLS] pooling induces a structured loss across the spectrum. These findings identify intermediate layers and pooling mechanisms as primary drivers of spectral transformation in modern
vision encoders.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Hashmat Shadab Malik, Muzammal Naseer and Salman Khan
Multimodal Large
Language Models integrate visual perception into
language reasoning, introducing a continuous attack surface susceptible to
adversarial attacks. Prior work on MLLM robustness has focused largely on English-centric tasks, leaving
multilingual behaviour unexplored. We address this gap through a systematic study of
adversarial robustness and
multimodal safety across 12 diverse
languages, evaluating open-source MLLMs that acquire
multilingual capability through instruction tuning. Gradient-based attacks reveal a transferable
multilingual vulnerability:
adversarial images optimized in one
language continue to induce failure in others, demonstrating strong cross-lingual transferability.
Multilingual safety further varies with how effectively a model retrieves or interprets harmful instructions. When harmful intent is issued through text,
languages with stronger linguistic grounding more often elicit misuse-enabling responses, while weaker
languages produce fewer unsafe outputs. When embedded in the image as typographic content, English scripts are reliably recognised and followed, whereas non-English scripts are rarely parsed by the
vision encoder. Lower-resource
languages may therefore appear safer, but this is an artefact of comprehension and visual-grounding failures rather than genuine
alignment, a phenomenon we term safety-by-failure. In contrast, MLLMs that build
multilingual capability throughout their training stages rather than only at instruction tuning, such as Qwen3-VL, exhibit genuine cross-lingual safety, maintaining active refusal across
languages rather than masking comprehension failure. Shallow
multilingual adaptation, such as fine-tuning on translated instruction data, may produce surface-level understanding that creates illusory safety in low-resource
languages; deeper integration across training stages leads to genuine
multilingual safety
alignment.
Read the full abstract ▾
Computer Vision
Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting
Georgios Tsoumplekas, Stella Bounareli and Vasileios Argyriou
Low-Rank Adaptation (LoRA) successfully enables personalization in text-to-image generation by adapting pre-trained
diffusion models to specific visual concepts and styles. However, extending such models to multi-concept customization remains challenging. Naively combining multiple LoRA weights or their outputs often leads to interference among concepts, resulting in degraded visual quality and reduced fidelity to the reference images of individual concepts. This paper proposes a simple yet effective approach for multi-concept customization by optimally combining the outputs of multiple LoRA modules. We leverage the relative importance of each concept during generation, as inferred from its corresponding
prompt tokens and introduce two methods, W-Switch and W-Composite, that employ a
prompt-aware importance weighting strategy in which each LoRA is weighted according to the semantic influence of its trigger words in the target
prompt. In addition, we extend existing quantitative evaluation metrics by proposing a new image-based similarity evaluation framework that assesses image fidelity and identity preservation through comparisons between real-world reference images and automatically segmented concept regions from generated images. We evaluate our approach on the ComposLoRA testbed and demonstrate consistent improvements over existing state-of-the-art methods in terms of visual quality, identity preservation and compositionality. Qualitative evaluations, including a Large
Language Model (LLM) based assessment and a user study, further validate the effectiveness of the proposed methods and align with the newly introduced quantitative image-based metrics. Our code is available at https://github.com/GeorgeTsoumplekas/
Prompt-Aware-Multi-LoRA-Composition.
Read the full abstract ▾
Computer Vision
SLU-2K: A Question-Based Benchmark for Semantic Evaluation of Sign Language Translation
Zeno Testa, Antonino Furnari, Lorenzo Baraldi et al. (+1 more)
Sign
Language Translation (SLT) is typically evaluated with surface-form metrics such as BLEU and ROUGE, which reward lexical overlap but do not directly measure whether a translation preserves the meaning of the source sign sequence. This is in contrast with the final objective of integrating SLT in assistive technology. In this work, we shift the focus from Sign
Language Translation (SLT) to Sign
Language Understanding (SLU), with particular emphasis on semantic understanding. Specifically, we evaluate systems based on their ability to correctly recover, from the input
video, key semantic aspects of the original sentence, such as actions taking place and facts about people and objects. To enable this evaluation systematically, we propose SLU-2K, a
dataset of 2,350 closed-ended
video question-answer pairs based on the popular PHOENIX-2014T and CSL-Daily
datasets. To obtain SLU-2K, we propose and extensively evaluate an automated data generation pipeline which produces questions across 7 categories, namely actions, locations, numbers, objects, people, time, and weather conditions. We show the potential of SLU-2K by evaluating popular
Multimodal Large
Language Models (MLLMs) and two representative state-of-the-art systems, MMSTL and SpaMo. Our results show that MLLMs reach near-random performance, highlighting the need for a more systematic integration of SLU in current AI systems. Furthermore, state-of-the-art translation systems carefully fine-tuned on in-domain data still exhibit a substantial semantic gap, with results ranging from 56.7% to 75.2%. These findings suggest that current SLT evaluation protocols overestimate true understanding and that future progress should be measured not only by fluency and n-gram overlap, but also by semantic correctness. Code,
prompts, and
benchmark files are available at https://github.com/ZenoTsT/SLU-2K
Read the full abstract ▾
All authors Zeno Testa, Antonino Furnari, Lorenzo Baraldi, Natalia Díaz-Rodríguez
Computer Vision
AmbientEye: A Dataset for Pupil Segmentation under Natural Ambient Infrared Illumination
Mingyu Han, Hyunyoung Han, Nitheekulawatn Thommakoon et al. (+4 more)
Eye tracking is essential for smart glasses, as it provides insight into user
attention for ambient intelligence applications. However, most existing eye-tracking systems rely on active infrared (IR) illumination, creating practical barriers to all-day outdoor use due to power consumption. In this paper, we investigate whether passive IR cameras alone, without any active IR light source, can enable reliable pupil detection in unconstrained outdoor environments, where ambient sunlight serves as the sole illumination source. To support this investigation, we introduce AmbientEye, a large-scale
dataset of 2,606,225 eye images collected from 35 participants from 19 countries. It is captured outdoors under natural sunlight with two off-axis camera configurations and two sun-orientation conditions. We provide high-quality pupil annotation through SAM2 automatic
segmentation, followed by refinement by human annotators. We
benchmark a state-of-the-art pupil
segmentation algorithm on our
dataset and compare its performance with that on existing
datasets under controlled IR illumination. Results reveal a substantial drop in pupil
segmentation performance from 0.928 on controlled IR
datasets to 0.767 on AmbientEye. This performance gap highlights the challenge of the ambient-light setting. This positions AmbientEye as a first
benchmark for an unexplored and highly practical eye-tracking scenario.
Read the full abstract ▾
All authors Mingyu Han, Hyunyoung Han, Nitheekulawatn Thommakoon, Gangtae Park, Jieun Han, Xucong Zhang, Ian Oakley
Computer Vision
Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models
Glenn Jocher, Jing Qiu, Mengyu Liu et al. (+3 more)
Real-time
vision demands models that are accurate, efficient, and simple to deploy across diverse hardware. The YOLO family has become widely deployed for this reason, yet most YOLO detectors still rely on non-maximum suppression at inference, carry heavy detection heads due to Distribution Focal Loss, require long training schedules, and can leave the smallest objects without positive label assignments. We present Ultralytics YOLO26, a unified real-time
vision model family that addresses these limitations through coordinated architecture and training advances. YOLO26 uses a dual-head design for native NMS-free end-to-end inference and removes DFL entirely, yielding a lighter head with unconstrained regression range. Its training pipeline combines MuSGD, a hybrid
Muon-SGD optimizer adapted from large
language model training; Progressive Loss, which shifts supervision toward the inference-time head; and STAL, a label assignment strategy that guarantees positive coverage for small objects. Beyond detection, YOLO26 introduces task-specific head and loss designs for instance
segmentation, pose estimation, and oriented detection, producing consistent gains across tasks and scales. The family spans five scales (n/s/m/l/x) and supports detection, instance
segmentation, pose estimation, classification, and oriented detection in a single pipeline, with an open-vocabulary extension, YOLOE-26, for text-, visual-, and
prompt-free inference. Across all scales, YOLO26 achieves 40.9-57.5 mAP on COCO at 1.7-11.8 ms T4 TensorRT latency, advancing the accuracy-latency Pareto front over prior real-time detectors, while YOLOE-26x reaches 40.6 AP on LVIS minival under text prompting. Code and models are available at https://github.com/ultralytics/ultralytics.
Read the full abstract ▾
All authors Glenn Jocher, Jing Qiu, Mengyu Liu, Shuai Lyu, Fatih Cagatay Akyon, Muhammet Esat Kalfaoglu
Computer Vision
Qwen-Image-Flash: Rethinking the Training Recipe for Few-Step Distillation
Tianhe Wu, Zikai Zhou, Kun Yan et al. (+23 more)
Few-step
distillation has emerged as a critical component in the development of advanced visual generative foundation models, substantially reducing inference overhead while enabling real-time generation and cost-efficient deployment across a broad range of practical scenarios. However, prior work has predominantly focused on advancing training objectives, while comparatively overlooking the training recipe, which has become increasingly critical in the era of large-scale foundation models. In this work, we systematically revisit the training recipe under the well-established distribution matching
distillation (DMD) framework for both text-to-image generation and image editing, focusing on three key dimensions: training data composition, teacher guidance within DMD, and task mixture. Our empirical analysis reveals several non-obvious and counterintuitive phenomena, ultimately motivating the development of Qwen-Image-Flash. These findings highlight that effective few-step
distillation depends not only on carefully designed objectives, but also on a principled training recipe.
Read the full abstract ▾
All authors Tianhe Wu, Zikai Zhou, Kun Yan, Kaiyuan Gao, Lihan Jiang, Jiahao Li, Jie Zhang, Ningyuan Tang, Shengming Yin, Xiaoyue Chen, Xiao Xu, Yilei Chen, Yuxiang Chen, Yan Shu, Yixian Xu, Yanran Zhang, Zihao Liu, Zhendong Wang, Zekai Zhang, Deqing Li, Liang Peng, Yi Wang, Zeke Xie, Jingren Zhou, Bo Zheng, Chenfei Wu