Computer Vision · Cross-listed from cs.RO
Tactile Modality Fusion for Vision-Language-Action Models
Charlotte Morissette, Amin Abyaneh, Wei-Di Chang et al. (+5 more)
We propose TacFiLM, a lightweight modality-fusion approach that integrates visual-tactile signals into
vision-
language-action (VLA) models. While advances in VLAs have introduced
robot policies that are both generalizable and semantically grounded, these models mainly rely on
vision-based perception.
Vision alone, however, cannot capture the complex interaction dynamics that occur during contact-rich manipulation, including contact forces, surface friction, compliance, and shear. While recent attempts to integrate tactile signals into VLA models often increase complexity through
token concatenation or large-scale pretraining, the heavy computational demands of behaviour models necessitate lightweight fusion strategies. To address these challenges, TacFiLM outlines a post-training finetuning approach that conditions intermediate visual features on pretrained tactile representations using feature-wise linear modulation (FiLM). Experimental results on insertion and drawer opening tasks demonstrate consistent improvements in success rate, direct task performance, completion time, and force stability across both in-distribution and out-of-distribution tasks. Together, these results support our method as an effective approach to integrating tactile signals into VLA models, improving contact-rich manipulation behaviours. Project page: https://charliem7.github.io/projects/TacFilm/
Read the full abstract ▾
All authors Charlotte Morissette, Amin Abyaneh, Wei-Di Chang, Anas Houssaini, David Meger, Hsiu-Chin Lin, Jonathan Tremblay, Gregory Dudek
Computer Vision
Texel Splatting: Perspective-Stable 3D Pixel Art
Dylan Ebert
Rendering 3D scenes as pixel art requires that discrete pixels remain stable as the camera moves. Existing methods snap the camera to a grid. Under orthographic projection, this works: every pixel shifts by the same amount, and a single snap corrects all of them. Perspective breaks this. Pixels at different depths drift at different rates, and no single snap corrects all depths. Texel splatting avoids this entirely. Scene geometry is rendered into a cubemap from a fixed point in the world, and each texel is splatted to the screen as a world-space quad. Cubemap indexing gives rotation invariance. Grid-snapping the origin gives translation invariance. The primary limitation is that a fixed origin cannot see all geometry; disocclusion at probe boundaries remains an open tradeoff.
Read the full abstract ▾
Computer Vision
Medical Image Spatial Grounding with Semantic Sampling
Andrew Seohwan Yu, Mohsen Hariri, Kunio Nakamura et al. (+3 more)
Vision language models (VLMs) have shown significant promise in visual grounding for images as well as
videos. In medical imaging research, VLMs represent a bridge between object detection and
segmentation, and report understanding and generation. However, spatial grounding of anatomical structures in the three-dimensional space of medical images poses many unique challenges. In this study, we examine image modalities, slice directions, and coordinate systems as differentiating factors for
vision components of VLMs, and the use of anatomical, directional, and relational terminology as factors for the
language components. We then demonstrate that visual and textual prompting systems such as labels, bounding boxes, and mask overlays have varying effects on the spatial grounding ability of VLMs. To enable measurement and reproducibility, we introduce MIS-Ground, a
benchmark that comprehensively tests a VLM for vulnerabilities against specific modes of Medical Image Spatial Grounding. We release MIS-Ground to the public at github.com/asy51/mis-ground. In addition, we present MIS-SemSam, a low-cost, inference-time, and model-agnostic optimization of VLMs that improves their spatial grounding ability with the use of Semantic Sampling. We find that MIS-SemSam improves the accuracy of Qwen3-VL-32B on MIS-Ground by 13.06%.
Read the full abstract ▾
All authors Andrew Seohwan Yu, Mohsen Hariri, Kunio Nakamura, Mingrui Yang, Xiaojuan Li, Vipin Chaudhary
Computer Vision · Cross-listed from cs.DC
Covariance-Guided Resource Adaptive Learning for Efficient Edge Inference
Ahmad N. L. Nabhaan, Zaki Sukma, Rakandhiya D. Rachmanto et al. (+4 more)
For deep learning inference on edge devices, hardware configurations achieving the same throughput can differ by 2$\times$ in power consumption, yet operators often struggle to find the efficient ones without exhaustive profiling. Existing approaches often rely on inefficient static presets or require expensive offline profiling that must be repeated for each new model or device. To address this problem, we present CORAL, an online optimization method that discovers near-optimal configurations without offline profiling. CORAL leverages distance covariance to statistically capture the non-linear dependencies between hardware settings, e.g., DVFS and concurrency levels, and performance metrics. Unlike prior work, we explicitly formulate the challenge as a throughput-power co-optimization problem to satisfy power budgets and throughput targets simultaneously. We evaluate CORAL on two NVIDIA Jetson devices across three object detection models ranging from lightweight to heavyweight. In single-target scenarios, CORAL achieves 96% $\unicode{x2013}$ 100% of the optimal performance found by exhaustive search. In strict dual-constraint scenarios where baselines fail or exceed power budgets, CORAL consistently finds proper configurations online with minimal exploration.
Read the full abstract ▾
All authors Ahmad N. L. Nabhaan, Zaki Sukma, Rakandhiya D. Rachmanto, Muhammad Husni Santriaji, Byungjin Cho, Arief Setyanto, In Kee Kim
Computer Vision
A comprehensive multimodal dataset and benchmark for ulcerative colitis scoring in endoscopy
Noha Ghatwary, Jiangbei Yue, Ahmed Elgendy et al. (+8 more)
Ulcerative colitis (UC) is a chronic mucosal inflammatory condition that places patients at increased risk of colorectal cancer. Colonoscopic surveillance remains the gold standard for assessing disease activity, and reporting typically relies on standardised endoscopic scoring metrics. The most widely used is the Mayo Endoscopic Score (MES), with some centres also adopting the Ulcerative Colitis Endoscopic Index of Severity (UCEIS). Both are descriptive assessments of mucosal inflammation (MES: 0 to 3; UCEIS: 0 to 8), where higher values indicate more severe disease. However, computational methods for automatically predicting these scores remain limited, largely due to the lack of publicly available expert-annotated
datasets and the absence of robust benchmarking. There is also a significant research gap in generating clinically meaningful descriptions of UC images, despite image captioning being a well-established computer
vision task. Variability in endoscopic systems and procedural workflows across centres further highlights the need for multi-centre
datasets to ensure algorithmic robustness and generalisability. In this work, we introduce a curated multi-centre, multi-resolution
dataset that includes expert-validated MES and UCEIS labels, alongside detailed clinical descriptions. To our knowledge, this is the first comprehensive
dataset that combines dual scoring metrics for classification tasks with expert-generated captions describing mucosal appearance and clinically accepted
reasoning for image captioning. This resource opens new opportunities for developing clinically meaningful
multimodal algorithms. In addition to the
dataset, we also provide benchmarking using convolutional neural networks,
vision transformers, hybrid models, and widely used
multimodal vision-
language captioning
algorithms.
Read the full abstract ▾
All authors Noha Ghatwary, Jiangbei Yue, Ahmed Elgendy, Hanna Nagdy, Ahmed Galal, Hayam Fathy, Hussein El-Amin, Venkataraman Subramanian, Noor Mohammed, Gilberto Ochoa-Ruiz, Sharib Ali
Computer Vision
ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inference
Surendra Pathak and Bo Han
While Large
Vision-
Language Models (LVLMs) demonstrate exceptional multi-modal capabilities, the quadratic computational cost of processing high-resolution visual
tokens remains a critical bottleneck. Though recent
token reduction strategies attempt to accelerate inference, such methods inadequately exploit
attention values and fail to address
token redundancy. More critically, they overlook the “
attention shift” phenomenon inherent in LVLMs, which skews
token attention scores. In this work, we propose ASAP, a novel training-free, KV-Cache-compatible
pruning recipe that comprehensively addresses these limitations. First, we mitigate the
attention shift by utilizing a dynamic bidirectional soft
attention mask, ensuring the selection of genuinely informative
tokens rather than naive
attention-based selection. Second, we posit that high semantic redundancy within the
token set degrades performance. We therefore introduce a weighted soft merging component that merges semantically similar
tokens, preserving only the most feature-dense visual patches for subsequent layers. ASAP achieves virtually lossless compression of visual context, retaining 99.02% of the original LLaVA-NeXT-7B performance while aggressively slashing computational FLOPs by ~80%.
Read the full abstract ▾
Computer Vision
Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoders
Jiaming Chu, Tao Wang and Lei Jin
Variational Autoencoder (VAE) encoders play a critical role in modern generative models, yet their computational cost often motivates the use of knowledge
distillation or quantification to obtain compact alternatives. Existing studies typically believe that the model work better on the samples closed to their training data distribution than unseen data distribution. In this work, we report a counter-intuitive phenomenon in VAE encoder
distillation: a compact encoder distilled only at low resolutions exhibits poor reconstruction performance at its native resolution, but achieves dramatically improved results when evaluated at higher, unseen input resolutions. Despite never being trained beyond $256^2$ resolution, the distilled encoder generalizes effectively to $512^2$ resolution inputs, partially inheriting the teacher model's resolution preference.We further analyze latent distributions across resolutions and find that higher-resolution inputs produce latent representations more closely aligned with the teacher's
manifold. Through extensive experiments on ImageNet-256, we show that simple resolution remapping-upsampling inputs before encoding and downsampling reconstructions for evaluation-leads to substantial gains across PSNR, MSE, SSIM, LPIPS, and rFID metrics. These findings suggest that VAE encoder
distillation learns resolution-consistent latent
manifolds rather than resolution-specific pixel mappings. This also means that the high training cost on
memory, time and high-resolution
datasets are not necessary conditions for distilling a VAE with high-resolution image reconstruction capabilities. On low resolution
datasets, the
distillation model still could learn the detailed knowledge of the teacher model in high-resolution image reconstruction.
Read the full abstract ▾
Computer Vision
Interp3R: Continuous-time 3D Geometry Estimation with Frames and Events
Shuang Guo, Filbert Febryanto, Lei Sun et al. (+2 more)
In recent years, 3D visual foundation models, pioneered by pointmap-based approaches such as
DUSt3R, have attracted a lot of interest, achieving impressive accuracy and strong generalization across diverse scenes. However, these methods are inherently limited to recovering scene geometry only at the discrete time instants when images are captured, leaving the scene
evolution during the blind time between consecutive frames largely unexplored. We introduce Interp3R, to the best of our knowledge, the first method that enhances pointmap-based models to estimate depth and camera poses at arbitrary time instants. It leverages asynchronous event data to interpolate pointmaps produced by frame-based models, enabling temporally continuous
geometric representations. Depth and camera poses are then jointly recovered by aligning the interpolated pointmaps together with those predicted by the underlying frame-based models into a consistent spatial framework. We train Interp3R exclusively on a synthetic
dataset, yet demonstrate strong generalization across six
datasets, both synthetic and real. Compared with the best two-stage baseline, Interp3R reduces absolute relative depth error by 15%-32% on DSEC and absolute trajectory error by up to 51% on EDS.
Read the full abstract ▾
All authors Shuang Guo, Filbert Febryanto, Lei Sun, Luc Van Gool, Guillermo Gallego
Computer Vision
LatSearch: Latent Reward-Guided Search for Faster Inference-Time Scaling in Video Diffusion
Zengqun Zhao, Ziquan Liu, Yu Cao et al. (+5 more)
The recent success of inference-time scaling in large
language models has inspired similar explorations in
video diffusion. In particular, motivated by the existence of "golden noise" that enhances
video quality, prior work has attempted to improve inference by optimising or searching for better initial noise. However, these approaches have notable limitations: they either rely on priors imposed at the beginning of noise sampling or on rewards evaluated only on the denoised and decoded
videos. This leads to error accumulation, delayed and sparse reward signals, and prohibitive computational cost, which prevents the use of stronger search
algorithms. Crucially, stronger search
algorithms are precisely what could unlock substantial gains in controllability, sample efficiency and generation quality for
video diffusion, provided their computational cost can be reduced. To fill in this gap, we enable efficient inference-time scaling for
video diffusion through latent reward guidance, which provides intermediate, informative and efficient feedback along the denoising trajectory. We introduce a latent reward model that scores partially denoised latents at arbitrary timesteps with respect to visual quality, motion quality, and text
alignment. Building on this model, we propose LatSearch, a novel inference-time search mechanism that performs Reward-Guided Resampling and
Pruning (RGRP). In the resampling stage, candidates are sampled according to reward-normalised probabilities to reduce over-reliance on the reward model. In the
pruning stage, applied at the final scheduled step, only the candidate with the highest cumulative reward is retained, improving both quality and efficiency. We evaluate LatSearch on the VBench-2.0
benchmark and demonstrate that it consistently improves
video generation across multiple evaluation dimensions compared to the baseline Wan2.1 model.
Read the full abstract ▾
All authors Zengqun Zhao, Ziquan Liu, Yu Cao, Shaogang Gong, Zhensong Zhang, Jifei Song, Jiankang Deng, Ioannis Patras
Computer Vision
VLA-Thinker: Boosting Vision-Language-Action Models through Thinking-with-Image Reasoning
Chaoyang Wang, Wenrui Bao, Sicheng Gao et al. (+5 more)
Vision-
Language-Action (VLA) models have shown promising capabilities for
embodied intelligence, but most existing approaches rely on text-based chain-of-thought
reasoning where visual inputs are treated as static context. This limits the ability of the model to actively revisit the environment and resolve ambiguities during long-horizon tasks. We propose VLA-Thinker, a thinking-with-image
reasoning framework that models perception as a dynamically invocable
reasoning action. To train such a system, we introduce a two-stage training pipeline consisting of (1) an SFT cold-start phase with curated visual Chain-of-Thought data to activate structured
reasoning and tool-use behaviors, and (2) GRPO-based reinforcement learning to align complete
reasoning-action trajectories with task-level success. Extensive experiments on LIBERO and RoboTwin 2.0
benchmarks demonstrate that VLA-Thinker significantly improves manipulation performance, achieving 97.5% success rate on LIBERO and strong gains across long-horizon robotic tasks. Project and Codes: https://cywang735.github.io/VLA-Thinker/ .
Read the full abstract ▾
All authors Chaoyang Wang, Wenrui Bao, Sicheng Gao, Bingxin Xu, Yu Tian, Yogesh S. Rawat, Yunhao Ge, Yuzhang Shang
Computer Vision
Expanding mmWave Datasets for Human Pose Estimation with Unlabeled Data and LiDAR Datasets
Zhuoxuan Peng, Boan Zhu, Xingjian Zhang et al. (+2 more)
Current millimeter-wave (mmWave)
datasets for human pose estimation (HPE) are scarce and lack diversity in both point cloud (PC) attributes and human poses, hindering the generalization ability of their trained models. On the other hand, unlabeled mmWave HPE data and diverse
LiDAR HPE
datasets are readily available. We propose EMDUL, a novel approach to expand the volume and diversity of an existing mmWave
dataset using unlabeled mmWave data and
LiDAR datasets. EMDUL consists of two independent modules, namely a pseudo-label
estimator to annotate unlabeled mmWave data, and a closed-form converter that translates an annotated
LiDAR PC to its mmWave counterpart. Expanding the original
dataset with both
LiDAR-converted and pseudo-labeled mmWave PCs significantly boosts the performance and generalization ability of all the examined HPE models, reducing 15.1% and 18.9% error for in-domain and out-of-domain settings, respectively. Code is available at https://github.com/Shimmer93/EMDUL.
Read the full abstract ▾
All authors Zhuoxuan Peng, Boan Zhu, Xingjian Zhang, Wenying Li, S.-H. Gary Chan
Computer Vision
Unlocking the Latent Canvas: Eliciting and Benchmarking Symbolic Visual Expression in LLMs
Yiren Zheng, Shibo Li, Jiaming Liu et al. (+2 more)
Current
multimodal approaches predominantly treat visual generation as an external process, relying on pixel rendering or code execution, thereby overlooking the native visual representation capabilities latent within Large
Language Models (LLMs). In this work, we unlock this potential through ASCII art, a compact, efficient, and text-native visual format. We introduce SVE-ASCII, a unified framework designed to elicit and
benchmark Symbolic Visual Expression directly within the pure text space. To address the scarcity of systematic resources, we construct ASCIIArt-7K, a high-quality
dataset synthesized via a novel "Seed-and-Evolve" pipeline that augments human-curated anchors through in-context stylistic editing. We further implement a unified instruction-tuning strategy that jointly optimizes for both Generation (Text-to-ASCII) and Understanding (ASCII-to-Text). Crucially, our experiments reveal a critical phenomenon regarding task duality: while it is established that perception aids generation, we provide compelling evidence that generative training significantly enhances visual comprehension. This confirms a mutually reinforcing cycle in symbolic visual processing, a relationship previously hypothesized but rarely empirically demonstrated in the visual domain. We release our
dataset, the ASCIIArt-Bench
benchmark, and the SVE-ASCII model, establishing a robust baseline for native text-based visual intelligence.
Read the full abstract ▾
All authors Yiren Zheng, Shibo Li, Jiaming Liu, Haofan Wang, Yiren Song
Computer Vision · Cross-listed from cs.LG
Trust-Region Noise Search for Black-Box Alignment of Diffusion and Flow Models
Niklas Schweiger, Daniel Cremers and Karnik Ram
Optimizing the noise samples of
diffusion and flow models is an increasingly popular approach to align these models to target rewards at inference time. However, we observe that these approaches are usually restricted to differentiable or cheap reward models, the formulation of the underlying pretrained generative model, or are
memory/compute inefficient. We instead propose a simple trust-region based search
algorithm (TRS) which treats the pre-trained generative and reward models as a black-box and only optimizes the source noise. Our approach achieves a good balance between global exploration and local exploitation, and is versatile and easily adaptable to various generative settings and reward models with minimal hyperparameter tuning. We evaluate TRS across text-to-image, molecule and
protein design tasks, and obtain significantly improved output samples over the base generative models and other inference-time
alignment approaches which optimize the source noise sample, or even the entire reverse-time sampling noise trajectories in the case of
diffusion models. Our source code is publicly available.
Read the full abstract ▾
Computer Vision
Mapping Dark-Matter Clusters via Physics-Guided Diffusion Models
Diego Royo, Brandon Zhao, Adolfo Muñoz et al. (+2 more)
Galaxy clusters are powerful probes of astrophysics and cosmology through gravitational
lensing: the clusters' mass, dominated by 85% dark matter, distorts background light. Yet, mass reconstruction lacks the scalability and large-scale
benchmarks to process the hundreds of thousands of clusters expected from forthcoming wide-field surveys. We introduce a fully automated method to reconstruct cluster surface mass density from photometry and gravitational
lensing observables. Central to our approach is DarkClusters-15k, our new
dataset of 15,000 simulated clusters with paired mass and photometry maps, the largest
benchmark to date, spanning multiple
redshifts and simulation frameworks. We train a plug-and-play
diffusion prior on DarkClusters-15k that learns the statistical relationship between mass and light, and draw posterior samples constrained by weak- and strong-
lensing observables; this yields principled reconstructions driven by explicit physics, alongside well-calibrated uncertainties. Our approach requires no expert tuning, runs in minutes rather than hours, achieves higher accuracy, and matches expertly-tuned reconstructions of the MACS 1206 cluster. We release our method and DarkClusters-15k to support development and benchmarking for upcoming wide-field cosmological surveys.
Read the full abstract ▾
All authors Diego Royo, Brandon Zhao, Adolfo Muñoz, Diego Gutierrez, Katherine L. Bouman
Computer Vision · Cross-listed from cs.RO
R3DP: Real-Time 3D-Aware Policy for Embodied Manipulation
Yuhao Zhang, Wanxi Dong, Yue Shi et al. (+13 more)
Embodied manipulation requires accurate 3D understanding of objects and their spatial relations to plan and execute contact-rich actions. While large-scale 3D
vision models provide strong priors, their computational cost incurs prohibitive latency for real-time control. We propose Real-time 3D-aware Policy (R3DP), which integrates powerful 3D priors into manipulation policies without sacrificing real-time performance. A core innovation of R3DP is the asynchronous fast-slow collaboration module, which seamlessly integrates large-scale 3D priors into the policy without compromising real-time performance. The system maintains real-time efficiency by querying the pre-trained slow system (VGGT) only on sparse key frames, while simultaneously employing a lightweight Temporal Feature Prediction Network (TFPNet) to predict features for all intermediate frames. By leveraging historical data to exploit temporal correlations, TFPNet explicitly improves task success rates through consistent feature estimation. Additionally, to enable more effective multi-view fusion, we introduce a Multi-View Feature Fuser (MVFF) that aggregates features across views by explicitly incorporating camera intrinsics and extrinsics. R3DP offers a plug-and-play solution for integrating large models into real-time inference systems. We evaluate R3DP against multiple baselines across different visual configurations. R3DP effectively harnesses large-scale 3D priors to achieve superior results, outperforming single-view and multi-view DP by 32.9% and 51.4% in average success rate, respectively. Furthermore, by decoupling heavy 3D
reasoning from policy execution, R3DP achieves a 44.8% reduction in inference time compared to a naive DP+VGGT integration.
Read the full abstract ▾
All authors Yuhao Zhang, Wanxi Dong, Yue Shi, Yi Liang, Jingnan Gao, Qiaochu Yang, Yaxing Lyu, Zhixuan Liang, Yibin Liu, Congsheng Xu, Xianda Guo, Wei Sui, Yaohui Jin, Xiaokang Yang, Yanyan Xu, Yao Mu
Computer Vision
Stefan Englmeier, Katharina Winter and Fabian B. Flohr
Autonomous driving systems depend on on models that can reason about high-level scene contexts and accurately predict the dynamics of their surrounding environment.
Vision-
Language Models (VLMs) have recently emerged as promising tools for decision-making and scene understanding, offering strong capabilities in contextual
reasoning. However, their limited spatial comprehension constrains their effectiveness as end-to-end driving models. World Models (WM) internalize environmental dynamics to predict future scene
evolution. Recently explored as ego-motion predictors and foundation models for autonomous driving, they represent a promising direction for addressing key challenges in the field, particularly enhancing generalization while maintaining dynamic prediction. To leverage the complementary strengths of context-based decision making and prediction, we propose WorldVLM: A hybrid architecture that unifies VLMs and WMs. In our design, the high-level VLM generates behavior commands to guide the driving WM, enabling interpretable and context-aware actions. We evaluate conditioning strategies and provide insights into the hybrid design challenges.
Read the full abstract ▾
Computer Vision
Refining 3D Medical Segmentation with Verbal Instruction
Kangxian Xie, Jiancheng Yang, Nandor Pinter et al. (+3 more)
Accurate 3D anatomical
segmentation is essential for clinical diagnosis and
surgical planning. However, automated models frequently generate suboptimal shape predictions due to factors such as limited and imbalanced training data, inadequate labeling quality, and distribution shifts between training and deployment settings. A natural solution is to iteratively refine the predicted shape based on the radiologists' verbal instructions. However, this is hindered by the scarcity of paired data that explicitly links erroneous shapes to corresponding corrective instructions. As an initial step toward addressing this limitation, we introduce CoWTalk, a
benchmark comprising 3D arterial anatomies with controllable synthesized anatomical errors and their corresponding repairing instructions. Building on this
benchmark, we further propose an iterative refinement model that represents 3D shapes as vector sets and interacts with textual instructions to progressively update the target shape. Experimental results demonstrate that our method achieves significant improvements over corrupted inputs and competitive baselines, highlighting the feasibility of
language-driven clinician-in-the-loop refinement for 3D medical shapes modeling.
Read the full abstract ▾
All authors Kangxian Xie, Jiancheng Yang, Nandor Pinter, Chao Wu, Behzad Bozorgtabar, Mingchen Gao
Computer Vision
Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
He Li, Yuhui Zhang, Xiaohan Wang et al. (+2 more)
The paper demonstrate that simple adjustments of the fine-tuning recipes of
multimodal large
language models (MLLM) are sufficient to mitigate catastrophic forgetting. On visual question answering, we design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs. Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images. However, we uncover a distinct form of forgetting in settings with in-distribution images and out-of-distribution text. We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy that combines
datasets and tasks. Finally, we demonstrate that this approach naturally extends to continual learning, outperforming existing methods with complex auxiliary mechanisms. In general, our findings challenge the prevailing assumptions by highlighting the inherent robustness of MLLMs and providing practical guidelines for adapting them while preserving their general capabilities.
Read the full abstract ▾
All authors He Li, Yuhui Zhang, Xiaohan Wang, Kaifeng Lyu, Serena Yeung-Levy
Computer Vision
V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar et al. (+6 more)
We present V-JEPA 2.1, a family of self-supervised models that learn dense, high-quality visual representations for both images and
videos while retaining strong global scene understanding. The approach combines four key components. First, a dense predictive loss uses a masking-based objective in which both visible and masked
tokens contribute to the training signal, encouraging explicit spatial and temporal grounding. Second, deep self-supervision applies the self-supervised objective hierarchically across multiple intermediate encoder layers to improve representation quality. Third, multi-modal tokenizers enable unified training across images and
videos. Finally, the model benefits from effective scaling in both model capacity and training data. Together, these design choices produce representations that are spatially structured, semantically coherent, and temporally consistent. Empirically, V-JEPA 2.1 achieves state-of-the-art performance on several challenging
benchmarks, including 7.71 mAP on Ego4D for short-term object-interaction anticipation and 40.8 Recall@5 on EPIC-KITCHENS for high-level action anticipation, as well as a 20-point improvement in real-
robot grasping success rate over V-JEPA-2 AC. The model also demonstrates strong performance in robotic
navigation (5.687 ATE on TartanDrive), depth estimation (0.307 RMSE on NYUv2 with a linear probe), and global recognition (77.7 on Something-Something-V2). These results show that V-JEPA 2.1 significantly advances the state of the art in dense visual understanding and world modeling.
Read the full abstract ▾
All authors Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran, Koustuv Sinha, Mike Rabbat, Yann LeCun, Nicolas Ballas, Adrien Bardes
Computer Vision
Wi-Spike: A Low-power WiFi Human Multi-action Recognition Model with Spiking Neural Networks
Nengbo Zhang, Yao Ying, Lu Wang et al. (+3 more)
WiFi-based human action recognition (HAR) has gained significant
attention due to its non-intrusive and
privacy-preserving nature. However, most existing WiFi sensing models predominantly focus on improving recognition accuracy, while issues of power consumption and energy efficiency remain insufficiently discussed. In this work, we present Wi-Spike, a bio-inspired spiking neural network (SNN) framework for efficient and accurate action recognition using WiFi channel state information (CSI) signals. Specifically, leveraging the event-driven and low-power characteristics of SNNs, Wi-Spike introduces spiking convolutional layers for spatio-temporal feature extraction and a novel temporal
attention mechanism to enhance discriminative representation. The extracted features are subsequently encoded and classified through spiking fully connected layers and a voting layer. Comprehensive experiments on three
benchmark datasets (NTU-Fi-HAR, NTU-Fi-HumanID, and UT-HAR) demonstrate that Wi-Spike achieves competitive accuracy in single-action recognition and superior performance in multi-action recognition tasks. As for energy consumption, Wi-Spike reduces the energy cost by at least half compared with other methods, while still achieving 95.83% recognition accuracy in human activity recognition. More importantly, Wi-Spike establishes a new state-of-the-art in WiFi-based multi-action HAR, offering a promising solution for real-time, energy-efficient edge sensing applications.
Read the full abstract ▾
All authors Nengbo Zhang, Yao Ying, Lu Wang, Kaishun Wu, Jieming Ma, Fei Luo
Computer Vision
LongVidSearch: An Agentic Benchmark for Multi-hop Evidence Retrieval Planning in Long Videos
Rongyi Yu, Chenyuan Duan and Wentao Zhang
Long
video question answering (Long-
Video QA) increasingly relies on agentic tool use to retrieve evidence from long
videos. In realistic settings, this process often requires multi-hop
retrieval, where
agents must iteratively gather multiple discontinuous evidence clips. However, existing long-
video benchmarks are largely static: they rarely enforce strict multi-hop
retrieval and typically lack a standardized evidence-access interface, making it difficult to separate failures in
retrieval planning from those in answer generation. To address this gap, we introduce LongVidSearch, a
benchmark for evaluating agentic multi-hop evidence
retrieval planning in long
videos under standardized access constraints. LongVidSearch enforces
retrieval necessity: a Hop-k question requires exactly k necessary evidence clips, and removing any single clip renders the question unsolvable. The
benchmark contains 3,000 questions over 447 long
videos (average length 26 minutes), covering four
reasoning categories: State Mutation, Causal Inference, Global Summary, and Visual Tracking, with 2-hop, 3-hop, and 4-hop evidence requirements. To ensure fair and controlled evaluation, all
agents interact with LongVidSearch through a unified tool interface, which fixes the
retrieval backend and isolates the
agent's ability to formulate queries and plan iterative
retrieval. In addition to answer accuracy, we measure tool-call cost to analyze the accuracy-efficiency trade-off under identical access conditions. We evaluate VideoAgent-style QA
agents with multiple backbone LLMs using three-judge majority voting. GPT-5 achieves the highest accuracy (42.43), outperforming Gemini 3 Pro (30.97) and GPT-4o (19.20), yet remaining below 50 %, highlighting the difficulty of multi-hop
retrieval planning. With gold evidence clips, performance becomes near-perfect, confirming
retrieval planning as the primary bottleneck.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.HC
Inclusive AI for Group Interactions: Predicting Gaze-Direction Behaviors in People with Intellectual and Developmental Disabilities
Giulia Huang, Maristella Matera and Micol Spitale
Artificial
agents that support human group interactions hold great promise, especially in sensitive contexts such as well-being promotion and therapeutic interventions. However, current systems struggle to mediate group interactions involving people who are not neurotypical. This limitation arises because most AI detection models (e.g., for turn-taking) are trained on data from neurotypical populations. This work takes a step toward inclusive AI by addressing the challenge of eye contact detection, a core component of non-verbal communication, with and for people with Intellectual and Developmental Disabilities. First, we introduce a new
dataset, Multi-party Interaction with Intellectual and Developmental Disabilities (MIDD), capturing atypical gaze and engagement patterns. Second, we present the results of a comparative analysis with neurotypical
datasets, highlighting differences in class imbalance, speaking activity, gaze distribution, and interaction dynamics. Then, we evaluate classifiers ranging from SVMs to FSFNet, showing that fine-tuning on MIDD improves performance, though notable limitations remain. Finally, we present the insights gathered through a focus group with six therapists to interpret our quantitative findings and understand the practical implications of atypical gaze and engagement patterns. Based on these results, we discuss data-driven strategies and emphasize the importance of feature choice for building more inclusive human-centered tools.
Read the full abstract ▾
Computer Vision
Uni-MDTrack: Learning Decoupled Memory and Dynamic States for Parameter-Efficient Visual Tracking in All Modality
Wenrui Cai, Zhenyi Lu, Yuzhe Li et al. (+4 more)
With the advent of
Transformer-based one-stream trackers that possess strong capability in inter-frame relation modeling, recent research has increasingly focused on how to introduce spatio-temporal context. However, most existing methods rely on a limited number of historical frames, which not only leads to insufficient utilization of the context, but also inevitably increases the length of input and incurs prohibitive computational overhead. Methods that query an external
memory bank, on the other hand, suffer from inadequate fusion between the retrieved spatio-temporal features and the backbone. Moreover, using discrete historical frames as context overlooks the rich dynamics of the target. To address the issues, we propose Uni-MDTrack, which consists of two core components:
Memory-Aware Compression
Prompt (MCP) module and Dynamic State Fusion (DSF) module. MCP effectively compresses rich
memory features into
memory-aware
prompt tokens, which deeply interact with the input throughout the entire backbone, significantly enhancing the performance while maintaining a stable computational load. DSF complements the discrete
memory by capturing the continuous dynamic, progressively introducing the updated dynamic state features from shallow to deep layers, while also preserving high efficiency. Uni-MDTrack also supports unified tracking across RGB, RGB-D/T/E, and RGB-
Language modalities. Experiments show that in Uni-MDTrack, training only the MCP, DSF, and prediction head, keeping the proportion of trainable parameters around 30%, yields substantial performance gains, achieves state-of-the-art results on 10
datasets spanning five modalities. Furthermore, both MCP and DSF exhibit excellent generality, functioning as plug-and-play components that can boost the performance of various baseline trackers, while significantly outperforming existing parameter-efficient training approaches.
Read the full abstract ▾
All authors Wenrui Cai, Zhenyi Lu, Yuzhe Li, Yongchao Feng, Jinqing Zhang, Qingjie Liu, Yunhong Wang
Computer Vision
End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction
Haoyu Zhang, Wei Zhai, Yuhang Yang et al. (+2 more)
Monocular 4D human-object interaction (HOI) reconstruction - recovering a moving human and a manipulated object from a single RGB
video - remains challenging due to depth ambiguity and frequent occlusions. Existing methods often rely on multi-stage pipelines or iterative optimization, leading to high inference latency, failing to meet real-time requirements, and susceptibility to error accumulation. To address these limitations, we propose THO, an end-to-end Spatial-Temporal
Transformer that predicts human motion and coordinated object motion in a forward fashion from the given
video and 3D template. THO achieves this by leveraging spatial-temporal HOI tuple priors. Spatial priors exploit contact-region proximity to infer occluded object features from human cues, while temporal priors capture cross-frame kinematic correlations to refine object representations and enforce physical coherence. Extensive experiments demonstrate that THO operates at an inference speed of 31.5 FPS on a single RTX 4090 GPU, achieving a >600x speedup over prior optimization-based methods while simultaneously improving reconstruction accuracy and temporal consistency. The project page is available at: https://nianheng.github.io/THO-project/
Read the full abstract ▾
All authors Haoyu Zhang, Wei Zhai, Yuhang Yang, Yang Cao, Zheng-Jun Zha
Computer Vision
GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos
Minghan Li, Tongna Chen, Tianrui Lv et al. (+3 more)
Existing text-to-
video retrieval benchmarks often permit matching from static appearance, making it difficult to determine whether models capture state changes and final outcomes. We introduce GenState-AI, a
benchmark comprising 956 AI-generated
videos and 195 verified state-change queries across Toy, Cartoon, Home, and Manipulation domains. Its construction uses
prompt-controlled variants that target either conflicting temporal/state outcomes or semantic substitutions. Following human verification, all valid
videos contribute to subset-level
retrieval pools, while complete query-specific triplets support diagnostic hard-negative ordering. We evaluate VCF-Lik, Qwen3-VL, and LanguageBind using both standard Recall@K and triplet-based consistency metrics. Qwen3-VL (8B) achieves the best macro-average R@1 and triplet consistency, while LanguageBind obtains the highest semantic discrimination accuracy. However, stronger
retrieval does not consistently yield correct hard-negative ordering, and Qwen3-VL and LanguageBind remain weak on Manipulation triplets, where VCF-Lik performs substantially better. GenState-AI therefore provides a controlled diagnostic testbed for distinguishing global
retrieval quality from fine-grained temporal and semantic grounding in AI-generated
videos.
Read the full abstract ▾
All authors Minghan Li, Tongna Chen, Tianrui Lv, Yishuai Zhang, Suchao An, Guodong Zhou
Computer Vision
Deep EM with Hierarchical Latent Label Modelling for Multi-Site Prostate Lesion Segmentation
Wen Yan, Yipei Wang, Shiqi Huang et al. (+5 more)
Label variability is a major challenge for prostate lesion
segmentation. In multi-site
datasets, annotations often reflect centre-specific contouring protocols, causing
segmentation networks to overfit to local styles and generalise poorly to unseen sites in inference. We treat each observed annotation as a noisy observation of an underlying latent 'clean' lesion mask, and propose a hierarchical expectation-maximisation (HierEM) framework that alternates between: (1) inferring a voxel-wise posterior distribution over the latent mask, and (2) training a CNN using this posterior as a soft target and estimate site-specific sensitivity and specificity under a hierarchical prior. This hierarchical prior decomposes label-quality into a global mean with site- and case-level deviations, reducing site-specific bias by penalising the likelihood term contributed only by site deviations. Experiments on three cohorts demonstrate that the proposed hierarchical EM framework enhances cross-site generalisation compared to state-of-the-art methods. For pooled-
dataset evaluation, the per-site mean DSC ranges from 29.50% to 39.69%; for leave-one-site-out generalisation, it ranges from 27.91% to 32.67%, yielding statistically significant improvements over comparison methods (p<0.039). The method also produces interpretable per-site latent label-quality estimates (sensitivity alpha ranges from 31.5% to 47.3% at specificity beta approximates 0.99), supporting post-hoc analyses of cross-site annotation variability. These results indicate that explicitly modelling site-dependent annotation can improve cross-site generalisation.
Read the full abstract ▾
All authors Wen Yan, Yipei Wang, Shiqi Huang, Natasha Thorley, Mark Emberton, Vasilis Stavrinides, Yipeng Hu, Dean Barratt
Computer Vision
Histo-MExNet: A Unified Framework for Real-World, Cross-Magnification, and Trustworthy Breast Cancer Histopathology
Enam Ahmed Taufika, Md Ahasanul Arafatha, Abhijit Kumar Ghoshb et al. (+2 more)
Accurate and reliable histopathological image classification is essential for breast cancer diagnosis. However, many deep learning models remain sensitive to magnification variability and lack interpretability. To address these challenges, we propose Histo-MExNet, a unified framework designed for scaleinvariant and uncertainty-aware classification. The model integrates DenseNet, ConvNeXt, and EfficientNet backbones within a gated multi-expert architecture, incorporates a prototype learning module for example-driven interpretability, and applies physics-informed regularization to enforce morphology preservation and spatial coherence during feature learning. Monte Carlo Dropout is used to quantify predictive uncertainty. On the BreaKHis
dataset, Histo-MExNet achieves 96.97% accuracy under multi-magnification training and demonstrates improved generalization to unseen magnification levels compared to single-expert models, while uncertainty estimation helps identify out-of-distribution samples and reduce overconfident errors, supporting a balanced combination of accuracy, robustness, and interpretability for clinical decision support.
Read the full abstract ▾
All authors Enam Ahmed Taufika, Md Ahasanul Arafatha, Abhijit Kumar Ghoshb, Md. Tanzim Rezab, Md Ashad Alamc
Computer Vision
G-ZAP: A Generalizable Zero-Shot Framework for Arbitrary-Scale Pansharpening
Zhiqi Yang, Shan Yin, Jingze Liang et al. (+1 more)
Pansharpening aims to fuse a high-resolution panchromatic (PAN) image and a low-resolution multispectral (LRMS) image to produce a high-resolution multispectral (HRMS) image. Recent deep models have achieved strong performance, yet they typically rely on large-scale pretraining and often generalize poorly to unseen real-world image pairs. Prior zero-shot approaches improve real-scene generalization but require per-image optimization, hindering weight reuse, and the above methods are usually limited to a fixed scale. To address this issue, we propose G-ZAP, a generalizable zero-shot framework for arbitrary-scale pansharpening, designed to handle cross-resolution, cross-scene, and cross-sensor generalization. G-ZAP adopts a feature-based implicit neural representation (INR) fusion network as the backbone and introduces a multi-scale, semi-supervised training scheme to enable robust generalization. Extensive experiments on multiple real-world
datasets show that G-ZAP achieves state-of-the-art results under PAN-scale fusion in both visual quality and quantitative metrics. Notably, G-ZAP supports weight reuse across image pairs while maintaining competitiveness with per-pair retraining, demonstrating strong potential for efficient real-world deployment.
Read the full abstract ▾
All authors Zhiqi Yang, Shan Yin, Jingze Liang, Liang-Jian Deng
Computer Vision
PGcGAN: Pathological Gait-Conditioned GAN for Human Gait Synthesis
Mritula Chandrasekaran, Sanket Kachole, Jarek Francik et al. (+1 more)
Pathological gait analysis is constrained by limited and variable clinical
datasets, which restrict the modeling of diverse gait impairments. To address this challenge, we propose a Pathological Gait-conditioned Generative
Adversarial Network (PGcGAN) that synthesises pathology-specific gait sequences directly from observed 3D pose keypoint trajectories data. The framework incorporates one-hot encoded pathology labels within both the generator and discriminator, enabling controlled synthesis across six gait categories. The generator adopts a conditional autoencoder architecture trained with
adversarial and reconstruction objectives to preserve structural and temporal gait characteristics. Experiments on the Pathological Gait
Dataset demonstrate strong
alignment between real and synthetic sequences through PCA and t-SNE analyses, visual kinematic inspection, and downstream classification tasks. Augmenting real data with synthetic sequences improved pathological gait recognition across GRU, LSTM, and CNN models, indicating that pathology-conditioned gait synthesis can effectively support data augmentation in pathological gait analysis.
Read the full abstract ▾
All authors Mritula Chandrasekaran, Sanket Kachole, Jarek Francik, Dimitrios Makris
Computer Vision · Cross-listed from cs.RO
OCRA: Object-Centric Learning with 3D and Tactile Priors for Human-to-Robot Action Transfer
Kuanning Wang, Ke Fan, Yuqian Fu et al. (+6 more)
We present OCRA, an Object-Centric framework for
video-based human-to-
Robot Action transfer that learns directly from human demonstration
videos to enable robust manipulation. Object-centric learning emphasizes task-relevant objects and their interactions while filtering out irrelevant background, providing a natural and scalable way to teach
robots. OCRA leverages multi-view RGB
videos, the state-of-the-art 3D foundation model VGGT, and advanced detection and
segmentation models to reconstruct object-centric 3D point clouds, capturing rich interactions between objects. To handle properties not easily perceived by
vision alone, we incorporate tactile priors via a large-scale
dataset of over one million tactile images. These 3D and tactile priors are fused through a
multimodal module (ResFiLM) and fed into a
Diffusion Policy to generate robust manipulation actions. Extensive experiments on both
vision-only and visuo-tactile tasks show that OCRA significantly outperforms existing baselines and ablations, demonstrating its effectiveness for learning from human demonstration
videos.
Read the full abstract ▾
All authors Kuanning Wang, Ke Fan, Yuqian Fu, Siyu Lin, Hu Luo, Daniel Seita, Yanwei Fu, Yu-Gang Jiang, Xiangyang Xue