Computer Vision
AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs
Aahana Basappa, Pranay Goel, Anusri Karra et al. (+3 more)
We investigate visual
reasoning limitations of both
multimodal large
language models (MLLMs) and image generation models (IGMs) by creating a novel
benchmark to systematically compare failure modes across image-to-text and text-to-image tasks, enabling cross-modal evaluation of visual understanding. Despite rapid growth in machine learning,
vision language models (VLMs) still fail to understand basic visual concepts such as object orientation, quantity, and spatial relationships, which highlights gaps in elementary visual
reasoning. By adapting MMVP
benchmark questions into explicit and implicit
prompts, we create AMVICC, a novel
benchmark for profiling failure modes across various modalities. After testing 11 MLLMs and 3 IGMs in 9 categories of visual
reasoning, our results show that failure modes are often shared between models and modalities. However, certain failures are model-specific and modality-specific, and this can potentially be attributed to various factors. IGMs consistently struggle to manipulate specific visual components in response to
prompts, especially in explicit
prompts, suggesting poor control over fine-grained visual attributes. Our findings apply most directly to the evaluation of existing state-of-the-art models on structured visual
reasoning tasks. This work lays the foundation for future cross-modal
alignment studies, offering a framework to probe whether image generation and visual interpretation failures stem from shared limitations. These insights can guide future improvements in unified
vision-
language modeling.
Read the full abstract ▾
All authors Aahana Basappa, Pranay Goel, Anusri Karra, Anish Karra, Asa Gilmore, Kevin Zhu
Computer Vision
PAS-Mamba: Phase-Amplitude-Spatial State Space Model for MRI Reconstruction
Xiaoyan Kui, Zijie Fan, Zexin Ji et al. (+5 more)
Joint feature modeling in both the spatial and frequency domains has become a mainstream approach in MRI reconstruction. However, existing methods generally treat the frequency domain as a whole, neglecting the differences in the information carried by its internal components. According to
Fourier transform theory, phase and amplitude represent different types of information in the image. Our spectrum swapping experiments show that magnitude mainly reflects pixel-level intensity, while phase predominantly governs image structure. To prevent interference between phase and magnitude feature learning caused by unified frequency-domain modeling, we propose the Phase-Amplitude-Spatial State Space Model (PAS-Mamba) for MRI Reconstruction, a framework that decouples phase and magnitude modeling in the frequency domain and combines it with image-domain features for better reconstruction. In the image domain, LocalMamba preserves spatial locality to sharpen fine anatomical details. In frequency domain, we disentangle amplitude and phase into two specialized branches to avoid representational coupling. To respect the concentric geometry of frequency information, we propose Circular Frequency Domain Scanning (CFDS) to serialize features from low to high frequencies. Finally, a Dual-Domain Complementary Fusion Module (DDCFM) adaptively fuses amplitude phase representations and enables bidirectional exchange between frequency and image domains, delivering superior reconstruction. Extensive experiments on the IXI and fastMRI knee
datasets show that PAS-Mamba consistently outperforms state of the art reconstruction methods.
Read the full abstract ▾
All authors Xiaoyan Kui, Zijie Fan, Zexin Ji, Qinsong Li, Hao Xu, Weixin Si, Haodong Xu, Beiji Zou
Computer Vision
GutenOCR: A Grounded Vision-Language Front-End for Documents
Hunter Heidenreich, Ben Elliott, Olivia Dinica et al. (+1 more)
GutenOCR is a family of grounded OCR front-ends obtained by fine-tuning Qwen2.5-VL-3B and Qwen2.5-VL-7B. The resulting single-checkpoint
vision-
language models expose reading, detection, and grounding through a unified,
prompt-based interface. Trained on business documents, scientific articles, and synthetic grounding data, the models support full-page and localized reading with line- and paragraph-level bounding boxes and conditional “where is x?” queries. We introduce a grounded OCR evaluation protocol and show that GutenOCR-7B more than doubles the composite grounded OCR score of its Qwen2.5-VL-7B backbone on 10.5K held-out business and scientific pages (0.40 to 0.82). On Fox and OmniDocBench v1.5, our approach substantially improves region- and line-level OCR as well as text-detection recall, but reveals trade-offs in page-level linearization, color-guided OCR, and formula-heavy layouts.
Read the full abstract ▾
All authors Hunter Heidenreich, Ben Elliott, Olivia Dinica, Yosheb Getachew
Computer Vision
XD-MAP: Cross-Modal Domain Adaptation via Semantic Parametric Maps for Scalable Training Data Generation
Frank Bieder, Hendrik Königshof, Haohao Hu et al. (+4 more)
Until open-world foundation models match the performance of specialized approaches, deep learning systems remain dependent on task- and sensor-specific data availability. To bridge the gap between available
datasets and deployment domains, domain adaptation strategies are widely used. In this work, we propose XD-MAP, a novel approach to transfer sensor-specific knowledge from an image
dataset to
LiDAR, an entirely different sensing domain. Our method leverages detections on camera images to create a semantic parametric map. The map elements are modeled to produce pseudo labels in the target domain without any manual annotation effort. Unlike previous domain transfer approaches, our method does not require direct overlap between sensors and enables extending the angular perception range from a front-view camera to a full 360{\deg} view. On our large-scale road feature
dataset, XD-MAP outperforms single shot baseline approaches by +19.5 mIoU for 2D semantic
segmentation, +19.5 PQth for 2D panoptic
segmentation, and +32.3 mIoU in 3D semantic
segmentation. The results demonstrate the effectiveness of our approach achieving strong performance on
LiDAR data without any manual labeling.
Read the full abstract ▾
All authors Frank Bieder, Hendrik Königshof, Haohao Hu, Fabian Immel, Yinzhe Shen, Jan-Hendrik Pauls, Christoph Stiller
Computer Vision
Real-Time Wildfire Localization on the NASA Autonomous Modular Sensor using Deep Learning
Yajvan Ravan, Aref Malek, Chester Dolph et al. (+1 more)
High-altitude, multi-spectral, aerial imagery is scarce and expensive to acquire, yet it is necessary for algorithmic advances and application of machine learning models to high-impact problems such as wildfire detection. We introduce a human-annotated
dataset from the NASA Autonomous Modular Sensor (AMS) using 12-channel, medium to high altitude (3 - 50 km) aerial wildfire images similar to those used in current US wildfire missions. Our
dataset combines spectral data from 12 different channels, including infrared (IR), short-wave IR (SWIR), and thermal. We take imagery from 20 wildfire missions and randomly sample small patches to generate over 4000 images with high variability, including occlusions by smoke/clouds, easily-confused false positives, and nighttime imagery. We demonstrate results from a deep-learning model to automate the human-intensive process of fire perimeter determination. We train two deep neural networks, one for image classification and the other for pixel-level
segmentation. The networks are combined into a unique real-time
segmentation model to efficiently localize active wildfire on an incoming image feed. Our model achieves 96% classification accuracy, 74% Intersection-over-Union(IoU), and 84% recall surpassing past methods, including models trained on satellite data and classical color-rule
algorithms. By leveraging a multi-spectral
dataset, our model is able to detect active wildfire at nighttime and behind clouds, while distinguishing between false positives. We find that data from the SWIR, IR, and thermal bands is the most important to distinguish fire perimeters. Our code and
dataset can be found here: https://github.com/nasa/Autonomous-Modular-Sensor-Wildfire-
Segmentation/tree/main and https://drive.google.com/drive/folders/1-u4vs9rqwkwgdeeeoUhftCxrfe_4QPTn?=usp=drive_link
Read the full abstract ▾
All authors Yajvan Ravan, Aref Malek, Chester Dolph, Nikhil Behari
Computer Vision
Gaussian Based Adaptive Multi-Modal 3D Semantic Occupancy Prediction
A. Enes Doruk
The sparse object detection paradigm shift towards dense 3D semantic occupancy prediction is necessary for dealing with long-tail safety challenges for autonomous vehicles. Nonetheless, the current voxelization methods commonly suffer from excessive computation complexity demands, where the fusion process is brittle, static, and breaks down under dynamic environmental settings. To this end, this research work enhances a novel Gaussian-based adaptive camera-
LiDAR multimodal 3D occupancy prediction model that seamlessly bridges the semantic strengths of camera modality with the
geometric strengths of
LiDAR modality through a
memory-efficient 3D Gaussian model. The proposed solution has four key components: (1)
LiDAR Depth Feature Aggregation (LDFA), where depth-wise deformable sampling is employed for dealing with
geometric sparsity, (2)
Entropy-Based Feature Smoothing, where cross-
entropy is employed for handling domain-specific noise, (3) Adaptive Camera-
LiDAR Fusion, where dynamic recalibration of sensor outputs is performed based on model outputs, and (4) Gauss-Mamba Head that uses Selective State Space Models for global context decoding that enjoys linear computation complexity.
Read the full abstract ▾
Computer Vision
Vision-Based Natural Language Scene Understanding for Autonomous Driving: An Extended Dataset and a New Model for Traffic Scene Description Generation
Danial Sadrian Zadeh, Otman A. Basir and Behzad Moshiri
Traffic scene understanding is essential for enabling autonomous vehicles to accurately perceive and interpret their environment, thereby ensuring safe
navigation. This paper presents a novel framework that transforms a single frontal-view camera image into a concise natural
language description, effectively capturing spatial layouts, semantic relationships, and driving-relevant cues. The proposed model leverages a hybrid
attention mechanism to enhance spatial and semantic feature extraction and integrates these features to generate contextually rich and detailed scene descriptions. To address the limited availability of specialized
datasets in this domain, a new
dataset derived from the BDD100K
dataset has been developed, with comprehensive guidelines provided for its construction. Furthermore, the study offers an in-depth discussion of relevant evaluation metrics, identifying the most appropriate measures for this task. Extensive quantitative evaluations using metrics such as CIDEr and SPICE, complemented by human judgment assessments, demonstrate that the proposed model achieves strong performance and effectively fulfills its intended objectives on the newly developed
dataset.
Read the full abstract ▾
Computer Vision
Large-Scale Label Quality Assessment for Medical Segmentation via a Vision-Language Judge and Synthetic Data
Yixiong Chen, Zongwei Zhou, Wenxuan Li et al. (+1 more)
Large-scale medical
segmentation datasets often combine manual and pseudo-labels of uneven quality, which can compromise training and evaluation. Low-quality labels may hamper performance and make the model training less robust. To address this issue, we propose SegAE (
Segmentation Assessment Engine), a lightweight
vision-
language model (VLM) that automatically predicts label quality across 142 anatomical structures. Trained on over four million image-label pairs with quality scores, SegAE achieves a high correlation coefficient of 0.902 with ground-truth Dice similarity and evaluates a 3D mask in 0.06s. SegAE shows several practical benefits: (I) Our analysis reveals widespread low-quality labeling across public
datasets; (II) SegAE improves data efficiency and training performance in active and semi-supervised learning, reducing
dataset annotation cost by one-third and quality-checking time by 70% per label. This tool provides a simple and effective solution for quality control in large-scale medical
segmentation datasets. The
dataset, model weights, and codes are released at https://github.com/Schuture/SegAE.
Read the full abstract ▾
All authors Yixiong Chen, Zongwei Zhou, Wenxuan Li, Alan Yuille
Computer Vision
CityCube: Benchmarking Cross-view Spatial Reasoning on Vision-Language Models in Urban Environments
Haotian Xu, Yue Hu, Zhengqiu Zhu et al. (+7 more)
Cross-view spatial
reasoning is essential for
embodied AI, underpinning spatial understanding, mental simulation and planning in complex environments. Existing
benchmarks primarily emphasize indoor or street settings, overlooking the unique challenges of open-ended urban spaces characterized by rich semantics, complex geometries, and view variations. To address this, we introduce CityCube, a systematic
benchmark designed to probe cross-view
reasoning capabilities of current VLMs in urban settings. CityCube integrates four viewpoint dynamics to mimic camera movements and spans a wide spectrum of perspectives from multiple platforms, e.g., vehicles, drones and satellites. For a comprehensive assessment, it features 5,022 meticulously annotated multi-view QA pairs categorized into five cognitive dimensions and three spatial relation expressions. A comprehensive evaluation of 33 VLMs reveals a significant performance disparity with humans: even large-scale models struggle to exceed 54.1% accuracy, remaining 34.2% below human performance. By contrast, small-scale fine-tuned VLMs achieve over 60.0% accuracy, highlighting the necessity of our
benchmark. Further analyses indicate the task correlations and fundamental cognitive disparity between VLMs and human-like
reasoning.
Read the full abstract ▾
All authors Haotian Xu, Yue Hu, Zhengqiu Zhu, Chen Gao, Ziyou Wang, Junreng Rao, Wenhao Lu, Weishi Li, Quanjun Yin, Yong Li
Computer Vision · Cross-listed from eess.IV
Partial Decoder Attention Network with Contour-weighted Loss Function for Data-Imbalance Medical Image Segmentation
Zhengyong Huang, Ning Jiang, Xingwen Sun et al. (+4 more)
Image
segmentation is pivotal in medical image analysis, facilitating clinical diagnosis, treatment planning, and disease evaluation. Deep learning has significantly advanced automatic
segmentation methodologies by providing superior modeling capability for complex structures and fine-grained anatomical regions. However, medical images often suffer from data imbalance issues, such as large volume disparities among organs or tissues, and uneven sample distributions across different anatomical structures. This imbalance tends to bias the model toward larger organs or more frequently represented structures, while overlooking smaller or less represented structures, thereby affecting the
segmentation accuracy and robustness. To address these challenges, we proposed a novel contour-weighted
segmentation approach, which improves the model's capability to represent small and underrepresented structures. We developed PDANet, a lightweight and efficient
segmentation network based on a partial decoder mechanism. We evaluated our method using three prominent public
datasets. The experimental results show that our methodology excelled in three distinct tasks: segmenting multiple abdominal organs,
brain tumors, and pelvic bone fragments with injuries. It consistently outperformed nine state-of-the-art methods. Moreover, the proposed contour-weighted strategy improved
segmentation for other comparison methods across the three
datasets, yielding average enhancements in Dice scores of 2.32%, 1.67%, and 3.60%, respectively. These results demonstrate that our contour-weighted
segmentation method surpassed current leading approaches in both accuracy and robustness. As a model-independent strategy, it can seamlessly fit various
segmentation frameworks, enhancing their performance. This flexibility highlighted its practical importance and potential for broad use in medical image analysis.
Read the full abstract ▾
All authors Zhengyong Huang, Ning Jiang, Xingwen Sun, Lihua Zhang, Peng Chen, Jens Domke, Yao Sui
Computer Vision · Cross-listed from eess.IV
Unsupervised Deformable Image Registration with Local-Global Attention and Image Decomposition
Zhengyong Huang, Xingwen Sun, Xuting Chang et al. (+5 more)
Deformable image registration is a critical technology in medical image analysis, with broad applications in clinical practice such as disease diagnosis, multi-modal fusion, and
surgical navigation. Traditional methods often rely on iterative optimization, which is computationally intensive and lacks generalizability. Recent advances in deep learning have introduced
attention-based mechanisms that improve feature
alignment, yet accurately registering regions with high anatomical variability remains challenging. In this study, we proposed a novel unsupervised deformable image registration framework, LGANet++, which employs a novel local-global
attention mechanism integrated with a unique technique for feature interaction and fusion to enhance registration accuracy, robustness, and generalizability. We evaluated our approach using five publicly available
datasets, representing three distinct registration scenarios: cross-patient, cross-time, and cross-modal CT-MR registration. The results demonstrated that our approach consistently outperforms several state-of-the-art registration methods, improving registration accuracy by 1.39% in cross-patient registration, 0.71% in cross-time registration, and 6.12% in cross-modal CT-MR registration tasks. These results underscore the potential of LGANet++ to support clinical workflows requiring reliable and efficient image registration. The source code is available at https://github.com/huangzyong/LGANet-Registration.
Read the full abstract ▾
All authors Zhengyong Huang, Xingwen Sun, Xuting Chang, Ning Jiang, Yao Wang, Jianfei Sun, Hongbin Han, Yao Sui
Computer Vision · Cross-listed from eess.IV
Self-Supervised Score-Based Despeckling for SAR Imagery via Log-Domain Transformation
Junhyuk Heo
The speckle noise inherent in Synthetic Aperture Radar (SAR) imagery significantly degrades image quality and complicates subsequent analysis. Given that SAR speckle is multiplicative and Gamma-distributed, effectively despeckling SAR imagery remains challenging. This paper introduces a novel self-supervised framework for SAR image despeckling based on score-based generative models operating in the transformed log domain. We first transform the data into the log-domain and then convert the speckle noise residuals into an approximately additive Gaussian distribution. This step enables the application of score-based models, which are trained in the transformed domain using a self-supervised objective. This objective allows our model to learn the clean underlying signal by training on further corrupted versions of the input data itself. Consequently, our method exhibits significantly shorter inference times compared to many existing self-supervised techniques, offering a robust and practical solution for SAR image restoration.
Read the full abstract ▾
Computer Vision
LURE: Latent Space Unblocking for Multi-Concept Reawakening in Diffusion Models
Mengyu Sun, Ziyuan Yang, Andrew Beng Jin Teoh et al. (+3 more)
Concept erasure aims to suppress sensitive content in
diffusion models, but recent studies show that erased concepts can still be reawakened, revealing vulnerabilities in erasure methods. Existing reawakening methods mainly rely on
prompt-level optimization to manipulate sampling trajectories, neglecting other generative factors, which limits a comprehensive understanding of the underlying dynamics. In this paper, we model the generation process as an implicit function to enable a comprehensive theoretical analysis of multiple factors, including text conditions, model parameters, and latent states. We theoretically show that perturbing each factor can reawaken erased concepts. Building on this insight, we propose a novel concept reawakening method: Latent space Unblocking for concept REawakening (LURE), which reawakens erased concepts by reconstructing the latent space and guiding the sampling trajectory. Specifically, our semantic re-binding mechanism reconstructs the latent space by aligning denoising predictions with target distributions to reestablish severed text-visual associations. However, in multi-concept scenarios, naive reconstruction can cause gradient conflicts and feature
entanglement. To address this, we introduce Gradient Field Orthogonalization, which enforces feature orthogonality to prevent mutual interference. Additionally, our Latent Semantic Identification-Guided Sampling (LSIS) ensures stability of the reawakening process via posterior density verification. Extensive experiments demonstrate that LURE enables simultaneous, high-fidelity reawakening of multiple erased concepts across diverse erasure tasks and methods.
Read the full abstract ▾
All authors Mengyu Sun, Ziyuan Yang, Andrew Beng Jin Teoh, Junxu Liu, Haibo Hu, Yi Zhang
Computer Vision
Implicit Neural Representation Facilitates Unified Universal Vision Encoding
Matthew Gwilliam, Xiao Wang, Xuefeng Hu et al. (+1 more)
Models for image representation learning are typically designed for either recognition or generation. Various forms of contrastive learning help models learn to convert images to
embeddings that are useful for classification, detection, and
segmentation. On the other hand, models can be trained to reconstruct images with pixel-wise, perceptual, and
adversarial losses in order to learn a latent space that is useful for image generation. We seek to unify these two directions with a first-of-its-kind model that learns representations which are simultaneously useful for recognition and generation. We train our model as a hyper-network for implicit neural representation, which learns to map images to model weights for fast, accurate reconstruction. We further integrate our INR hyper-network with knowledge
distillation to improve its generalization and performance. Beyond the novel training design, the model also learns an unprecedented compressed
embedding space with outstanding performance for various visual tasks. The complete model competes with state-of-the-art results for image representation learning, while also enabling generative capabilities with its high-quality tiny
embeddings. The code is available at https://github.com/tiktok/huvr.
Read the full abstract ▾
All authors Matthew Gwilliam, Xiao Wang, Xuefeng Hu, Zhenheng Yang
Computer Vision
VideoMaMa: Mask-Guided Video Matting via Generative Prior
Sangbeom Lim, Seoung Wug Oh, Jiahui Huang et al. (+3 more)
Generalizing
video matting models to real-world
videos remains a significant challenge due to the scarcity of labeled data. To address this, we present
Video Mask-to-Matte Model (VideoMaMa) that converts coarse
segmentation masks into pixel accurate alpha mattes, by leveraging pretrained
video diffusion models. VideoMaMa demonstrates strong zero-shot generalization to real-world footage, even though it is trained solely on synthetic data. Building on this capability, we develop a scalable pseudo-labeling pipeline for large-scale
video matting and construct the Matting Anything in
Video (MA-V)
dataset, which offers high-quality matting annotations for more than 50K real-world
videos spanning diverse scenes and motions. To validate the effectiveness of this
dataset, we fine-tune the SAM2 model on MA-V to obtain SAM2-Matte, which outperforms the same model trained on existing matting
datasets in terms of robustness on in-the-wild
videos. These findings emphasize the importance of large-scale pseudo-labeled
video matting and showcase how generative priors and accessible
segmentation cues can drive scalable progress in
video matting research.
Read the full abstract ▾
All authors Sangbeom Lim, Seoung Wug Oh, Jiahui Huang, Heeji Yoon, Seungryong Kim, Joon-Young Lee
Computer Vision
Motion 3-to-4: 3D Motion Reconstruction for 4D Synthesis
Hongyuan Chen, Xingyu Chen, Youjia Zhang et al. (+2 more)
We present Motion 3-to-4, a feed-forward framework for synthesising high-quality 4D dynamic objects from a single
monocular video and an optional 3D reference mesh. While recent advances have significantly improved 2D,
video, and 3D content generation, 4D synthesis remains difficult due to limited training data and the inherent ambiguity of recovering geometry and motion from a
monocular viewpoint. Motion 3-to-4 addresses these challenges by decomposing 4D synthesis into static 3D shape generation and motion reconstruction. Using a canonical reference mesh, our model learns a compact motion latent representation and predicts per-frame vertex trajectories to recover complete, temporally coherent geometry. A scalable frame-wise
transformer further enables robustness to varying sequence lengths. Evaluations on both standard
benchmarks and a new
dataset with accurate ground-truth geometry show that Motion 3-to-4 delivers superior fidelity and spatial consistency compared to prior work. Project page is available at https://motion3-to-4.github.io/.
Read the full abstract ▾
All authors Hongyuan Chen, Xingyu Chen, Youjia Zhang, Zexiang Xu, Anpei Chen
Computer Vision
LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR
Said Taghadouini, Adrien Cavaillès and Baptiste Aubertin
We present LightOnOCR-2-1B, a 1B-parameter end-to-end
multilingual vision–
language model that converts document images (e.g., PDFs) into clean, naturally ordered text without brittle OCR pipelines. Trained on a large-scale, high-quality
distillation mix with strong coverage of scans, French documents, and scientific PDFs, LightOnOCR-2 achieves state-of-the-art results on OlmOCR-Bench while being 9$\times$ smaller and substantially faster than prior best-performing models. We further extend the output format to predict normalized bounding boxes for embedded images, introducing localization during pretraining via a resume strategy and refining it with RLVR using IoU-based rewards. Finally, we improve robustness with checkpoint averaging and task-arithmetic merging. We release model checkpoints under Apache 2.0, and publicly release the
dataset and LightOnOCR-bbox-bench evaluation under their respective licenses.
Read the full abstract ▾
Computer Vision
OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer
Pengze Zhang, Yanze Wu, Mengtian Li et al. (+8 more)
Videos convey richer information than images or text, capturing both spatial and temporal dynamics. However, most existing
video customization methods rely on reference images or task-specific temporal priors, failing to fully exploit the rich spatio-temporal information inherent in
videos, thereby limiting flexibility and generalization in
video generation. To address these limitations, we propose OmniTransfer, a unified framework for spatio-temporal
video transfer. It leverages multi-view information across frames to enhance appearance consistency and exploits temporal cues to enable fine-grained temporal control. To unify various
video transfer tasks, OmniTransfer incorporates three key designs: Task-aware Positional Bias that adaptively leverages reference
video information to improve temporal
alignment or appearance consistency; Reference-decoupled Causal Learning separating reference and target branches to enable precise reference transfer while improving efficiency; and Task-adaptive
Multimodal Alignment using
multimodal semantic guidance to dynamically distinguish and tackle different tasks. Extensive experiments show that OmniTransfer outperforms existing methods in appearance (ID and style) and temporal transfer (camera movement and
video effects), while matching pose-guided methods in motion transfer without using pose, establishing a new paradigm for flexible, high-fidelity
video generation.
Read the full abstract ▾
All authors Pengze Zhang, Yanze Wu, Mengtian Li, Xu Bai, Songtao Zhao, Fulong Ye, Chong Mou, Xinghui Li, Zhuowei Chen, Qian He, Mingyuan Gao
Computer Vision
Soft Tail-dropping for Adaptive Visual Tokenization
Zeyuan Chen, Kai Zhang, Zhuowen Tu et al. (+1 more)
We present Soft Tail-dropping Adaptive Tokenizer (STAT), a 1D discrete visual tokenizer that adaptively chooses the number of output
tokens per image according to its structural complexity and level of detail. STAT encodes an image into a sequence of discrete codes together with per-
token keep probabilities. Beyond standard autoencoder objectives, we regularize these keep probabilities to be monotonically decreasing along the sequence and explicitly align their distribution with an image-level complexity measure. As a result, STAT produces length-adaptive 1D visual
tokens that are naturally compatible with causal 1D autoregressive (AR) visual generative models. On ImageNet-1k, equipping vanilla causal AR models with STAT yields competitive or superior visual generation quality compared to other probabilistic model families, while also exhibiting favorable scaling behavior that has been elusive in prior vanilla AR visual generation attempts.
Read the full abstract ▾
All authors Zeyuan Chen, Kai Zhang, Zhuowen Tu, Yuanjun Xiong
Computer Vision · Cross-listed from cs.LG
KAGE-Bench: Fast Known-Axis Visual Generalization Evaluation for Reinforcement Learning
Egor Cherepanov, Daniil Zelezetsky, Alexey K. Kovalev et al. (+1 more)
Pixel-based reinforcement learning
agents often fail under purely visual distribution shift even when latent dynamics and rewards are unchanged, but existing
benchmarks entangle multiple sources of shift and hinder systematic analysis. We introduce KAGE-Env, a JAX-native 2D platformer that factorizes the observation process into independently controllable visual axes while keeping the underlying control problem fixed. By construction, varying a visual axis affects performance only through the induced state-conditional action distribution of a pixel policy, providing a clean abstraction for visual generalization. Building on this environment, we define KAGE-Bench, a
benchmark of six known-axis suites comprising 34 train-evaluation configuration pairs that isolate individual visual shifts. Using a standard PPO-CNN baseline, we observe strong axis-dependent failures, with background and photometric shifts often collapsing success, while
agent-appearance shifts are comparatively benign. Several shifts preserve forward motion while breaking task completion, showing that return alone can obscure generalization failures. Finally, the fully vectorized JAX implementation enables up to 33M environment steps per second on a single GPU, enabling fast and reproducible sweeps over visual factors. Code: https://avanturist322.github.io/KAGEBench/.
Read the full abstract ▾
All authors Egor Cherepanov, Daniil Zelezetsky, Alexey K. Kovalev, Aleksandr I. Panov
Computer Vision
Rig-Aware 3D Reconstruction of Vehicle Undercarriages using Gaussian Splatting
Nitin Kulkarni, Akhil Devarashetti, Charlie Cluss et al. (+5 more)
Inspecting the undercarriage of used vehicles is a labor-intensive task that requires inspectors to crouch or crawl underneath each vehicle to thoroughly examine it. Additionally, online buyers rarely see undercarriage photos. We present an end-to-end pipeline that utilizes a three-camera rig to capture
videos of the undercarriage as the vehicle drives over it, and produces an interactive 3D model of the undercarriage. The 3D model enables inspectors and customers to rotate, zoom, and slice through the undercarriage, allowing them to detect rust, leaks, or impact damage in seconds, thereby improving both workplace safety and buyer confidence. Our primary contribution is a rig-aware Structure-from-Motion (SfM) pipeline specifically designed to overcome the challenges of wide-angle lens distortion and low-parallax scenes. Our method overcomes the challenges of wide-angle lens distortion and low-parallax scenes by integrating precise camera calibration, synchronized
video streams, and strong
geometric priors from the camera rig. We use a constrained matching strategy with learned components, the DISK feature extractor, and the
attention-based LightGlue matcher to generate high-quality sparse point clouds that are often unattainable with standard SfM pipelines. These point clouds seed the Gaussian splatting process to generate photorealistic undercarriage models that render in real-time. Our experiments and ablation studies demonstrate that our design choices are essential to achieve state-of-the-art quality.
Read the full abstract ▾
All authors Nitin Kulkarni, Akhil Devarashetti, Charlie Cluss, Livio Forte, Dan Buckmaster, Philip Schneider, Chunming Qiao, Alina Vereshchaka
Computer Vision · Cross-listed from cs.GR
Copy-Trasform-Paste: Zero-Shot Object-Object Alignment Guided by Vision-Language and Geometric Constraints
Rotem Gatenyo and Ohad Fried
We study zero-shot 3D
alignment of two given meshes, using a text
prompt describing their spatial relation – an essential capability for content creation and scene assembly. Earlier approaches primarily rely on
geometric alignment procedures, while recent work leverages pretrained 2D
diffusion models to model
language-conditioned object-object spatial relationships. In contrast, we directly optimize the relative pose at test time, updating translation, rotation, and isotropic scale with CLIP-driven gradients via a differentiable renderer, without training a new model. Our framework augments
language supervision with geometry-aware objectives: a variant of soft-Iterative Closest Point (ICP) term to encourage surface attachment and a penetration loss to discourage interpenetration. A phased schedule strengthens contact constraints over time, and camera control concentrates the optimization on the interaction region. To enable evaluation, we curate a
benchmark containing diverse categories and relations, and compare against baselines. Our method outperforms all alternatives, yielding semantically faithful and physically plausible
alignments.
Read the full abstract ▾
Computer Vision
IIR-VLM: In-Context Instance-level Recognition for Large Vision-Language Models
Liang Shi, Wei Li, Kevin M Beussman et al. (+2 more)
Instance-level recognition (ILR) concerns distinguishing individual instances from one another, with person re-identification as a prominent example. Despite the impressive visual perception capabilities of modern VLMs, we find their performance on ILR unsatisfactory, often dramatically underperforming domain-specific ILR models. This limitation hinders many practical application of VLMs, e.g. where recognizing familiar people and objects is crucial for effective visual understanding. Existing solutions typically learn to recognize instances one at a time using instance-specific
datasets, which not only incur substantial data collection and training costs but also struggle with fine-grained discrimination. In this work, we propose IIR-VLM, a VLM enhanced for In-context Instance-level Recognition. We integrate pre-trained ILR expert models as auxiliary visual encoders to provide specialized features for learning diverse instances, which enables VLMs to learn new instances in-context in a one-shot manner. Further, IIR-VLM leverages this knowledge for instance-aware visual understanding. We validate IIR-VLM's efficacy on existing instance personalization
benchmarks. Finally, we demonstrate its superior ILR performance on a challenging new
benchmark, which assesses ILR capabilities across varying difficulty and diverse categories, with person, face, pet and general objects as the instances at task.
Read the full abstract ▾
All authors Liang Shi, Wei Li, Kevin M Beussman, Lin Chen, Yun Fu
Computer Vision
Progressive Masked Refinement Self-supervised Learning for Low-Dose CT Denoising
Yichao Liu, Zongru Shao, Rui Wen et al. (+2 more)
Self-supervised learning has been increasingly investigated for low-dose computed
tomography (LDCT) image denoising, as it alleviates the dependence on paired normal-dose CT (NDCT) data, which are often difficult to obtain. However, many existing self-supervised blind-spot denoising methods may under-utilize pixel-wise supervisory information loss due to evaluating the training loss only at masked locations. To mitigate this issue, we propose a novel Progressive Masked Refinement Learning framework that progressively refines denoising results while preserving and exploiting available LDCT information. Specifically, we explicitly inject a combination of controlled Gaussian and Poisson noise during training to regularize the denoising process and mitigate trivial identity mapping. Furthermore, we introduce a step-wise mask denoising mechanism that gradually reduces the discrepancy between synthetic corruption and the noise characteristics of LDCT images, enabling more fine-grained learning for denoising. Extensive experiments on the Mayo LDCT
dataset demonstrate that the proposed method outperforms existing self-supervised approaches and achieves performance comparable to, or better than, several representative supervised denoising methods.
Read the full abstract ▾
All authors Yichao Liu, Zongru Shao, Rui Wen, Yueyang Teng, Junwen Guo
Computer Vision
ASBA: A-line State Space Model and B-line Attention for Sparse Optical Doppler Tomography Reconstruction
Zhenghong Li, Wensheng Cheng, Congwu Du et al. (+3 more)
Optical Doppler
Tomography (ODT) is an emerging blood flow analysis technique. A 2D ODT image (B-scan) is generated by sequentially acquiring 1D depth-resolved raw A-scans (A-line) along the lateral axis (B-line), followed by Doppler phase-subtraction analysis. To ensure high-fidelity B-scan images, current practices rely on dense sampling, which prolongs scanning time, increases storage demands, and limits the capture of rapid blood flow dynamics. Recent studies have explored sparse sampling of raw A-scans to alleviate these limitations, but their effectiveness is hindered by the conservative sampling rates and the uniform modeling of flow and background signals. In this study, we introduce a novel blood flow-aware network, named ASBA (A-line ROI State space model and B-line phase
Attention), to reconstruct ODT images from highly sparsely sampled raw A-scans. Specifically, we propose an A-line ROI state space model to extract sparsely distributed flow features along the A-line, and a B-line phase
attention to capture long-range flow signals along each B-line based on phase difference. Moreover, we introduce a flow-aware weighted loss function that encourages the network to prioritize the accurate reconstruction of flow signals. Extensive experiments on real animal data demonstrate that the proposed approach clearly outperforms existing state-of-the-art reconstruction methods.
Read the full abstract ▾
All authors Zhenghong Li, Wensheng Cheng, Congwu Du, Yingtian Pan, Zhaozheng Yin, Haibin Ling
Computer Vision
One-Shot Refiner: Boosting Feed-forward Novel View Synthesis via One-Step Diffusion
Yitong Dong, Qi Zhang, Minchao Jiang et al. (+6 more)
We present a novel framework for high-fidelity novel view synthesis (NVS) from sparse images, addressing key limitations in recent feed-forward 3D Gaussian Splatting (3DGS) methods built on
Vision Transformer (ViT) backbones. While ViT-based pipelines offer strong
geometric priors, they are often constrained by low-resolution inputs due to computational costs. Moreover, existing generative enhancement methods tend to be 3D-agnostic, resulting in inconsistent structures across views, especially in unseen regions. To overcome these challenges, we design a Dual-Domain Detail Perception Module, which enables handling high-resolution images without being limited by the ViT backbone, and endows Gaussians with additional features to store high-frequency details. We develop a feature-guided
diffusion network, which can preserve high-frequency details during the restoration process. We introduce a unified training strategy that enables joint optimization of the ViT-based
geometric backbone and the
diffusion-based refinement module. Experiments demonstrate that our method can maintain superior generation quality across multiple
datasets.
Read the full abstract ▾
All authors Yitong Dong, Qi Zhang, Minchao Jiang, Zhiqiang Wu, Qingnan Fan, Ying Feng, Huaqi Zhang, Hujun Bao, Guofeng Zhang
Computer Vision
LLM Augmented Intervenable Multimodal Adaptor for Post-operative Complication Prediction in Lung Cancer Surgery
Shubham Pandey, Bhavin Jawade, Srirangaraj Setlur et al. (+2 more)
Postoperative complications remain a critical concern in clinical practice, adversely affecting patient outcomes and contributing to rising healthcare costs. We present MIRACLE, a deep learning architecture for prediction of risk of postoperative complications in lung cancer surgery by integrating preoperative clinical and radiological data. MIRACLE employs a hyperspherical
embedding space fusion of heterogeneous inputs, enabling the extraction of robust, discriminative features from both structured clinical records and high-dimensional radiological images. To enhance transparency of prediction and clinical utility, we incorporate an interventional deep learning module in MIRACLE, that not only refines predictions but also provides interpretable and actionable insights, allowing domain experts to interactively adjust recommendations based on clinical expertise. We validate our approach on POC-L, a real-world
dataset comprising 3,094 lung cancer patients who underwent surgery at Roswell Park Comprehensive Cancer Center. Our results demonstrate that MIRACLE outperforms various traditional machine learning models and contemporary large
language models (LLM) variants alone, for personalized and explainable postoperative risk management.
Read the full abstract ▾
All authors Shubham Pandey, Bhavin Jawade, Srirangaraj Setlur, Venu Govindaraju, Kenneth Seastedt
Computer Vision · Cross-listed from cs.RO
TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformers
Bin Yu, Shijie Lian, Xiaopeng Lin et al. (+8 more)
The fundamental premise of
Vision-
Language-Action (VLA) models is to harness the extensive general capabilities of pre-trained
Vision-
Language Models (VLMs) for generalized
embodied intelligence. However, standard robotic fine-tuning inevitably disrupts the pre-trained feature space, leading to "catastrophic forgetting" that compromises the general visual understanding we aim to leverage. To effectively utilize the uncorrupted general capabilities of VLMs for robotic tasks, we propose TwinBrainVLA, which coordinates two isomorphic VLM pathways: a frozen generalist (also called "Left
Brain") and a trainable specialist (also called "Right
Brain"). Our architecture utilizes a Asymmetric Mixture-of-
Transformers (AsyMoT) mechanism, enabling the Right
Brain to dynamically query and fuse intact semantic knowledge from the Left
Brain with proprioceptive states. This fused representation conditions a flow-matching action expert for precise continuous control. Empirical results on SimplerEnv and RoboCasa
benchmarks demonstrate that by explicitly retaining general capabilities, TwinBrainVLA achieves substantial performance gains over baseline models in complex manipulation tasks.
Read the full abstract ▾
All authors Bin Yu, Shijie Lian, Xiaopeng Lin, Yuliang Wei, Zhaolong Shen, Changti Wu, Yuzhuo Miao, Xinming Wang, Bailing Wang, Cong Huang, Kai Chen
Computer Vision
GIC-DLC: Differentiable Logic Circuits for Hardware-Friendly Grayscale Image Compression
Till Aczel, David F. Jenny, Simon Bührer et al. (+3 more)
Neural image codecs achieve higher compression ratios than traditional hand-crafted methods such as PNG or JPEG-XL, but often incur substantial computational overhead, limiting their deployment on energy-constrained devices such as smartphones, cameras, and drones. We propose Grayscale Image Compression with Differentiable Logic
Circuits (GIC-DLC), a hardware-aware codec where we train lookup tables to combine the flexibility of neural networks with the efficiency of Boolean operations. Experiments on grayscale
benchmark datasets show that GIC-DLC outperforms traditional codecs in compression efficiency while allowing substantial reductions in energy consumption and latency. These results demonstrate that learned compression can be hardware-friendly, offering a promising direction for low-power image compression on edge devices.
Read the full abstract ▾
All authors Till Aczel, David F. Jenny, Simon Bührer, Andreas Plesner, Antonio Di Maio, Roger Wattenhofer
Computer Vision
The Side Effects of Being Smart: Safety Risks in MLLMs' Multi-Image Reasoning
Renmiao Chen, Yida Lu, Shiyao Cui et al. (+6 more)
As
Multimodal Large
Language Models (MLLMs) acquire stronger
reasoning capabilities to handle complex, multi-image instructions, this advancement may pose new safety risks. We study this problem by introducing MIR-SafetyBench, the first
benchmark focused on multi-image
reasoning safety, which consists of 2,676 instances across a taxonomy of 9 multi-image relations. Our extensive evaluations on 19 MLLMs reveal a troubling trend: models with more advanced multi-image
reasoning can be more vulnerable on MIR-SafetyBench. Beyond attack success rates, we find that many responses labeled as safe are superficial, often driven by misunderstanding or evasive, non-committal replies. We further observe that unsafe generations exhibit lower
attention entropy than safe ones on average. This internal signature suggests a possible risk that models may over-focus on task solving while neglecting safety constraints. Our code and data are available at https://github.com/thu-coai/MIR-SafetyBench.
Read the full abstract ▾
All authors Renmiao Chen, Yida Lu, Shiyao Cui, Xuan Ouyang, Victor Shea-Jay Huang, Shumin Zhang, Chengwei Pan, Han Qiu, Minlie Huang