Computer Vision
Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding
Christopher Clark, Jieyu Zhang, Zixian Ma et al. (+18 more)
Today's strongest
video-
language models (VLMs) remain proprietary. The strongest open-weight models either rely on synthetic data from proprietary VLMs, effectively distilling from them, or do not disclose their training data or recipe. As a result, the open-source community lacks the foundations needed to improve on the state-of-the-art
video (and image)
language models. Crucially, many downstream applications require more than just high-level
video understanding; they require grounding – either by pointing or by tracking in pixels. Even proprietary models lack this capability. We present Molmo2, a new family of VLMs that are state-of-the-art among open-source models and demonstrate exceptional new capabilities in point-driven grounding in single image, multi-image, and
video tasks. Our key contribution is a collection of 7 new
video datasets and 2 multi-image
datasets, including a
dataset of highly detailed
video captions for pre-training, a free-form
video Q&A
dataset for fine-tuning, a new object tracking
dataset with complex queries, and an innovative new
video pointing
dataset, all collected without the use of closed VLMs. We also present a training recipe for this data utilizing an efficient packing and message-tree encoding scheme, and show bi-directional
attention on
vision tokens and a novel
token-weight strategy improves performance. Our best-in-class 8B model outperforms others in the class of open weight and data models on short
videos, counting, and captioning, and is competitive on long-
videos. On
video-grounding Molmo2 significantly outperforms existing open-weight models like Qwen3-VL (35.5 vs 29.6 accuracy on
video counting) and surpasses proprietary models like Gemini 3 Pro on some tasks (38.4 vs 20.0 F1 on
video pointing and 56.2 vs 41.1 J&F on
video tracking).
Read the full abstract ▾
All authors Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, Vincent Shao, Yue Yang, Weikai Huang, Ziqi Gao, Taira Anderson, Jianrui Zhang, Jitesh Jain, George Stoica, Winson Han, Ali Farhadi, Ranjay Krishna
Computer Vision · Cross-listed from eess.IV
Multi-Objective Pareto-Front Optimization for Efficient Adaptive VVC Streaming
Angeliki Katsenou, Vignesh V. Menon, Guoda Laurinaviciute et al. (+2 more)
Adaptive
video streaming has facilitated improved
video streaming over the past years. A balance among coding performance objectives such as bitrate,
video quality, and decoding complexity is required to achieve efficient, content- and codec-dependent, adaptive
video streaming. This paper proposes a multi-objective Pareto-front (PF) optimization framework to construct quality-monotonic, content-adaptive bitrate ladders Versatile
Video Coding (VVC) streaming that jointly optimize
video quality, bitrate, and decoding time, which is used as a practical proxy for decoding energy. Two strategies are introduced: the Joint Rate-Quality-Time Pareto Front (JRQT-PF) and the Joint Quality-Time Pareto Front (JQT-PF), each exploring different tradeoff formulations and objective prioritizations. The ladders are constructed under quality monotonicity constraints during adaptive streaming to ensure a consistent Quality of Experience (QoE). Experiments are conducted on a large-scale UHD
dataset (Inter-4K), with quality assessed using PSNR, VMAF, and XPSNR, and complexity measured via decoding time and energy consumption. The JQT-PF method achieves 11.76% average bitrate savings while reducing average decoding time by 0.29% to maintain the same XPSNR, compared to a widely-used fixed ladder. More aggressive configurations yield up to 27.88% bitrate savings at the cost of increased complexity. The JRQT-PF strategy, on the other hand, offers more controlled tradeoffs, achieving 6.38 % bitrate savings and 6.17 % decoding time reduction. This framework outperforms existing methods, including fixed ladders, VMAF- and XPSNR-based dynamic resolution selection, and complexity-aware
benchmarks. The results confirm that PF optimization with decoding time constraints enables sustainable, high-quality streaming tailored to network and device capabilities.
Read the full abstract ▾
All authors Angeliki Katsenou, Vignesh V. Menon, Guoda Laurinaviciute, Benjamin Bross, Detlev Marpe
Computer Vision
RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation
Peng Chen, Xiaobao Wei, Yi Yang et al. (+3 more)
Talking head generation is increasingly important in virtual reality (VR), especially for social scenarios involving multi-turn conversation. Existing approaches face notable limitations: mesh-based 3D methods can model dual-person dialogue but lack realistic textures, while large-model-based 2D methods produce natural appearances but incur prohibitive computational costs. Recently, 3D Gaussian Splatting (3DGS) based methods achieve efficient and realistic rendering but remain speaker-only and ignore social relationships. We introduce RSATalker, the first framework that leverages 3DGS for realistic and socially-aware talking head generation with support for multi-turn conversation. Our method first drives mesh-based 3D facial motion from
speech, then binds 3D Gaussians to mesh facets to render high-fidelity 2D avatar
videos. To capture interpersonal dynamics, we propose a socially-aware module that encodes social relationships, including blood and non-blood as well as equal and unequal, into high-level
embeddings through a learnable query mechanism. We design a three-stage training paradigm and construct the RSATalker
dataset with
speech-mesh-image triplets annotated with social relationships. Extensive experiments demonstrate that RSATalker achieves state-of-the-art performance in both realism and social awareness. The code and
dataset will be released.
Read the full abstract ▾
All authors Peng Chen, Xiaobao Wei, Yi Yang, Naiming Yao, Hui Chen, Feng Tian
Computer Vision
Action100M: A Large-scale Video Action Dataset
Delong Chen, Tejaswi Kasarla, Yejin Bang et al. (+6 more)
Inferring physical actions from visual observations is a fundamental capability for advancing machine intelligence in the physical world. Achieving this requires large-scale, open-vocabulary
video action
datasets that span broad domains. We introduce Action100M, a large-scale
dataset constructed from 1.2M Internet instructional
videos (14.6 years of duration), yielding O(100 million) temporally localized segments with open-vocabulary action supervision and rich captions. Action100M is generated by a fully automated pipeline that (i) performs hierarchical temporal
segmentation using V-JEPA 2
embeddings, (ii) produces multi-level frame and segment captions organized as a Tree-of-Captions, and (iii) aggregates evidence with a
reasoning model (GPT-OSS-120B) under a multi-round Self-Refine procedure to output structured annotations (brief/detailed action, actor, brief/detailed caption). Training VL-JEPA on Action100M demonstrates consistent data-scaling improvements and strong zero-shot performance across diverse action recognition
benchmarks, establishing Action100M as a new foundation for scalable research in
video understanding and world modeling.
Read the full abstract ▾
All authors Delong Chen, Tejaswi Kasarla, Yejin Bang, Mustafa Shukor, Willy Chung, Jade Yu, Allen Bolourchi, Theo Moutakanni, Pascale Fung
Computer Vision
Adversarial Evasion Attacks on Computer Vision using SHAP Values
Frank Mollard, Marcus Becker and Florian Roehrbein
The paper introduces a white-box attack on computer
vision models using SHAP values. It demonstrates how
adversarial evasion attacks can compromise the performance of deep learning models by reducing output confidence or inducing misclassifications. Such attacks are particularly insidious as they can deceive the perception of an
algorithm while eluding human perception due to their imperceptibility to the human eye. The proposed attack leverages SHAP values to quantify the significance of individual inputs to the output at the inference stage. A comparison is drawn between the SHAP attack and the well-known Fast Gradient Sign Method. We find evidence that SHAP attacks are more robust in generating misclassifications particularly in gradient hiding scenarios.
Read the full abstract ▾
Computer Vision
Jordan-Segmentable Masks: A Topology-Aware definition for characterizing Binary Image Segmentation
Serena Grazia De Benedictis, Amedeo Altavilla and Nicoletta Del Buono
Image
segmentation plays a central role in computer
vision. However, widely used evaluation metrics, whether pixel-wise, region-based, or boundary-focused, often struggle to capture the structural and topological coherence of a
segmentation. In many practical scenarios, such as medical imaging or object delineation, small inaccuracies in boundary, holes, or fragmented predictions can result in high metric scores, despite the fact that the resulting masks fail to preserve the object global shape or connectivity. This highlights a limitation of conventional metrics: they are unable to assess whether a predicted
segmentation partitions the image into meaningful interior and exterior regions. In this work, we introduce a
topology-aware notion of
segmentation based on the Jordan Curve
Theorem, and adapted for use in digital planes. We define the concept of a Jordan-segmentatable mask, which is a binary
segmentation whose structure ensures a topological separation of the image domain into two connected components. We analyze
segmentation masks through the lens of digital
topology and homology theory, extracting a $4$-curve candidate from the mask, verifying its topological validity using Betti numbers. A mask is considered Jordan-segmentatable when this candidate forms a digital 4-curve with $\beta_0 = \beta_1 = 1$, or equivalently when its complement splits into exactly two $8$-connected components. This framework provides a mathematically rigorous, unsupervised criterion with which to assess the structural coherence of
segmentation masks. By combining digital Jordan theory and homological invariants, our approach provides a valuable alternative to standard evaluation metrics, especially in applications where topological correctness must be preserved.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Process-Guided Concept Bottleneck Model
Reza M. Asiyabi, SEOSAW Partnership and Steven Hancock Casey Ryan
Concept Bottleneck Models (CBMs) improve the explainability of black-box Deep Learning (DL) by introducing intermediate semantic concepts. However, standard CBMs often overlook domain-specific relationships and causal mechanisms, and their dependence on complete concept labels limits applicability in scientific domains where supervision is sparse but processes are well defined. To address this, we propose the Process-Guided Concept Bottleneck Model (PG-CBM), an extension of CBMs which constrains learning to follow domain-defined causal mechanisms through biophysically meaningful intermediate concepts. Using above ground biomass density estimation from Earth Observation data as a case study, we show that PG-CBM reduces error and bias compared to multiple
benchmarks, whilst leveraging multi-source heterogeneous training data and producing interpretable intermediate outputs. Beyond improved accuracy, PG-CBM enhances transparency, enables detection of spurious learning, and provides scientific insights, representing a step toward more trustworthy AI systems in scientific applications.
Read the full abstract ▾
Computer Vision
DeepUrban: Interaction-Aware Trajectory Prediction and Planning for Automated Driving by Aerial Imagery
Constantin Selzer and Fabian B. Flohr
The efficacy of autonomous driving systems hinges critically on robust prediction and planning capabilities. However, current
benchmarks are impeded by a notable scarcity of scenarios featuring dense traffic, which is essential for understanding and modeling complex interactions among road users. To address this gap, we collaborated with our industrial partner, DeepScenario, to develop DeepUrban-a new drone
dataset designed to enhance trajectory prediction and planning
benchmarks focusing on dense urban settings. DeepUrban provides a rich collection of 3D traffic objects, extracted from high-resolution images captured over urban intersections at approximately 100 meters altitude. The
dataset is further enriched with comprehensive map and scene information to support advanced modeling and simulation tasks. We evaluate state-of-the-art (SOTA) prediction and planning methods, and conducted experiments on generalization capabilities. Our findings demonstrate that adding DeepUrban to nuScenes can boost the accuracy of vehicle predictions and planning, achieving improvements up to 44.1 % / 44.3% on the ADE / FDE metrics. Website: https://iv.ee.hm.edu/deepurban
Read the full abstract ▾
Computer Vision
Inference-time Physics Alignment of Video Generative Models with Latent World Models
Jianhao Yuan, Xiaofeng Zhang, Felix Friedrich et al. (+7 more)
State-of-the-art
video generative models produce promising visual content yet often violate basic physics principles, limiting their utility. While some attribute this deficiency to insufficient physics understanding from pre-training, we find that the shortfall in physics plausibility also stems from suboptimal inference strategies. We therefore introduce WMReward and treat improving physics plausibility of
video generation as an inference-time
alignment problem. In particular, we leverage the strong physics prior of a latent world model (here, VJEPA-2) as a reward to search and steer multiple candidate denoising trajectories, enabling scaling test-time compute for better generation performance. Empirically, our approach substantially improves physics plausibility across image-conditioned, multiframe-conditioned, and text-conditioned generation settings, with validation from human preference study. Notably, in the ICCV 2025 Perception Test PhysicsIQ Challenge, we achieve a final score of 62.64%, winning first place and outperforming the previous state of the art by 7.42%. Our work demonstrates the viability of using latent world models to improve physics plausibility of
video generation, beyond this specific instantiation or parameterization.
Read the full abstract ▾
All authors Jianhao Yuan, Xiaofeng Zhang, Felix Friedrich, Nicolas Beltran-Velez, Melissa Hall, Reyhane Askari-Hemmat, Xiaochuang Han, Nicolas Ballas, Michal Drozdzal, Adriana Romero-Soriano
Computer Vision
Unleashing the Capabilities of Large Vision-Language Models for Intelligent Perception of Roadside Infrastructure
Luxuan Fu, Chong Liu, Bisheng Yang et al. (+1 more)
Automated perception of urban roadside infrastructure is crucial for smart city management, yet general-purpose models often struggle to capture the necessary fine-grained attributes and domain rules. While Large
Vision Language Models (VLMs) excel at open-world recognition, they often struggle to accurately interpret complex facility states in compliance with engineering standards, leading to unreliable performance in real-world applications. To address this, we propose a domain-adapted framework that transforms VLMs into specialized
agents for intelligent infrastructure analysis. Our approach integrates a data-efficient fine-tuning strategy with a knowledge-grounded
reasoning mechanism. Specifically, we leverage open-vocabulary fine-tuning on Grounding DINO to robustly localize diverse assets with minimal supervision, followed by LoRA-based adaptation on Qwen-VL for deep semantic attribute
reasoning. To mitigate
hallucinations and enforce professional compliance, we introduce a dual-modality
Retrieval-Augmented Generation (RAG) module that dynamically retrieves authoritative industry standards and visual exemplars during inference. Evaluated on a comprehensive new
dataset of urban roadside scenes, our framework achieves a detection performance of 58.9 mAP and an attribute recognition accuracy of 95.5%, demonstrating a robust solution for intelligent infrastructure monitoring.
Read the full abstract ▾
All authors Luxuan Fu, Chong Liu, Bisheng Yang, Zhen Dong
Computer Vision
Enhancing the quality of gauge images captured in smoke and haze scenes through deep learning
Oscar H. Ramírez-Agudelo, Akshay N. Shewatkar, Edoardo Milana et al. (+2 more)
Images captured in hazy and smoky environments suffer from reduced visibility, posing a challenge when monitoring infrastructures and hindering emergency services during critical situations. The proposed work investigates the use of the deep learning models to enhance the automatic, machine-based readability of gauge in smoky environments, with accurate gauge data interpretation serving as a valuable tool for first responders. The study utilizes two deep learning architectures, FFA-Net and AECR-Net, to improve the visibility of gauge images, corrupted with light up to dense haze and smoke. Since
benchmark datasets of analog gauge images are unavailable, a new synthetic
dataset, containing over 14,000 images, was generated using the Unreal Engine. The models were trained with an 80% train, 10% validation, and 10% test split for the haze and smoke
dataset, respectively. For the synthetic haze
dataset, the SSIM and PSNR metrics are about 0.98 and 43\,dB, respectively, comparing well to state-of-the art results. Additionally, more robust results are retrieved from the AECR-Net, when compared to the FFA-Net. Although the results from the synthetic smoke
dataset are poorer, the trained models achieve interesting results. In general, imaging in the presence of smoke are more difficult to enhance given the inhomogeneity and high density. Secondly, FFA-Net and AECR-Net are implemented to dehaze and not to desmoke images. This work shows that use of deep learning architectures can improve the quality of analog gauge images captured in smoke and haze scenes immensely. Finally, the enhanced output images can be successfully post-processed for automatic autonomous reading of gauges
Read the full abstract ▾
All authors Oscar H. Ramírez-Agudelo, Akshay N. Shewatkar, Edoardo Milana, Roland C. Aydin, Kai Franke
Computer Vision
SVII-3D: Advancing Roadside Infrastructure Inventory with Decimeter-level 3D Localization and Comprehension from Sparse Street Imagery
Chong Liu, Luxuan Fu, Yang Jia et al. (+2 more)
The automated creation of digital twins and precise asset inventories is a critical task in smart city construction and facility lifecycle management. However, utilizing cost-effective sparse imagery remains challenging due to limited robustness, inaccurate localization, and a lack of fine-grained state understanding. To address these limitations, SVII-3D, a unified framework for holistic asset digitization, is proposed. First, LoRA fine-tuned open-set detection is fused with a spatial-
attention matching network to robustly associate observations across sparse views. Second, a geometry-guided refinement mechanism is introduced to resolve structural errors, achieving precise decimeter-level 3D localization. Third, transcending static
geometric mapping, a
Vision-
Language Model
agent leveraging multi-modal prompting is incorporated to automatically diagnose fine-grained operational states. Experiments demonstrate that SVII-3D significantly improves identification accuracy and minimizes localization errors. Consequently, this framework offers a scalable, cost-effective solution for high-fidelity infrastructure digitization, effectively bridging the gap between sparse perception and automated intelligent maintenance.
Read the full abstract ▾
All authors Chong Liu, Luxuan Fu, Yang Jia, Zhen Dong, Bisheng Yang
Computer Vision · Cross-listed from cs.AI
A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5
Xingjun Ma, Yixu Wang, Hengyuan Xu et al. (+18 more)
The rapid
evolution of Large
Language Models (LLMs) and
Multimodal Large
Language Models (MLLMs) has driven major gains in
reasoning, perception, and generation across
language and
vision, yet whether these advances translate into comparable improvements in safety remains unclear, partly due to fragmented evaluations that focus on isolated modalities or threat models. In this report, we present an integrated safety evaluation of six frontier models–GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5–assessing each across
language,
vision-
language, and image generation using a unified protocol that combines
benchmark,
adversarial,
multilingual, and compliance evaluations. By aggregating results into safety leaderboards and model profiles, we reveal a highly uneven safety landscape: while GPT-5.2 demonstrates consistently strong and balanced performance, other models exhibit clear trade-offs across
benchmark safety,
adversarial robustness,
multilingual generalization, and regulatory compliance. Despite strong results under standard
benchmarks, all models remain highly vulnerable under
adversarial testing, with worst-case safety rates dropping below 6%. Text-to-image models show slightly stronger
alignment in regulated visual risk categories, yet remain fragile when faced with
adversarial or semantically ambiguous
prompts. Overall, these findings highlight that safety in frontier models is inherently multidimensional–shaped by modality,
language, and evaluation design–underscoring the need for standardized, holistic safety assessments to better reflect real-world risk and guide responsible deployment.
Read the full abstract ▾
All authors Xingjun Ma, Yixu Wang, Hengyuan Xu, Yutao Wu, Yifan Ding, Yunhan Zhao, Zilong Wang, Jiabin Hua, Ming Wen, Jianan Liu, Ranjie Duan, Yifeng Gao, Yingshui Tan, Yunhao Chen, Hui Xue, Xin Wang, Wei Cheng, Jingjing Chen, Zuxuan Wu, Bo Li, Yu-Gang Jiang
Computer Vision
BikeActions: An Open Platform and Benchmark for Cyclist-Centric VRU Action Recognition
Max A. Buettner, Kanak Mazumder, Luca Koecher et al. (+3 more)
Anticipating the intentions of Vulnerable Road Users (VRUs) is a critical challenge for safe autonomous driving (AD) and mobile robotics. While current research predominantly focuses on pedestrian crossing behaviors from a vehicle's perspective, interactions within dense shared spaces remain underexplored. To bridge this gap, we introduce FUSE-Bike, the first fully open perception platform of its kind. Equipped with two
LiDARs, a camera, and GNSS, it facilitates high-fidelity, close-range data capture directly from a cyclist's viewpoint. Leveraging this platform, we present BikeActions, a novel multi-modal
dataset comprising 852 annotated samples across 5 distinct action classes, specifically tailored to improve VRU behavior modeling. We establish a rigorous
benchmark by evaluating state-of-the-art graph convolution and
transformer-based models on our publicly released data splits, establishing the first performance baselines for this challenging task. We release the full
dataset together with data curation tools, the open hardware design, and the
benchmark code to foster future research in VRU action understanding under https://iv.ee.hm.edu/bikeactions/.
Read the full abstract ▾
All authors Max A. Buettner, Kanak Mazumder, Luca Koecher, Mario Finkbeiner, Sebastian Niebler, Fabian B. Flohr
Computer Vision
SatMap: Revisiting Satellite Maps as Prior for Online HD Map Construction
Kanak Mazumder and Fabian B. Flohr
Online high-definition (HD) map construction is an essential part of a safe and robust end-to-end autonomous driving (AD) pipeline. Onboard camera-based approaches suffer from limited depth perception and degraded accuracy due to occlusion. In this work, we propose SatMap, an online vectorized HD map estimation method that integrates satellite maps with multi-view camera observations and directly predicts a vectorized HD map for downstream prediction and planning modules. Our method leverages lane-level semantics and texture from satellite imagery captured from a Bird's Eye View (BEV) perspective as a global prior, effectively mitigating depth ambiguity and occlusion. In our experiments on the nuScenes
dataset, SatMap achieves 34.8% mAP performance improvement over the camera-only baseline and 8.5% mAP improvement over the camera-
LiDAR fusion baseline. Moreover, we evaluate our model in long-range and adverse weather conditions to demonstrate the advantages of using a satellite prior map. Source code will be available at https://iv.ee.hm.edu/satmap/.
Read the full abstract ▾
Computer Vision
MERGETUNE: Continued Fine-Tuning of Vision-Language Models
Wenqing Wang, Da Li, Xiatian Zhu et al. (+1 more)
Fine-tuning
vision-
language models (VLMs) such as CLIP often leads to catastrophic forgetting of pretrained knowledge. Prior work primarily aims to mitigate forgetting during adaptation; however, forgetting often remains inevitable during this process. We introduce a novel paradigm, continued fine-tuning (CFT), which seeks to recover pretrained knowledge after a zero-shot model has already been adapted. We propose a simple, model-agnostic CFT strategy (named MERGETUNE) guided by linear mode connectivity (LMC), which can be applied post hoc to existing fine-tuned models without requiring architectural changes. Given a fine-tuned model, we continue fine-tuning its trainable parameters (e.g., soft
prompts or linear heads) to search for a continued model which has two low-loss paths to the zero-shot (e.g., CLIP) and the fine-tuned (e.g., CoOp) solutions. By exploiting the geometry of the loss landscape, the continued model implicitly merges the two solutions, restoring pretrained knowledge lost in the fine-tuned counterpart. A challenge is that the vanilla LMC constraint requires data replay from the pretraining task. We approximate this constraint for the zero-shot model via a second-order surrogate, eliminating the need for large-scale data replay. Experiments show that MERGETUNE improves the harmonic mean of CoOp by +5.6% on base-novel generalisation without adding parameters. On robust fine-tuning evaluations, the LMC-merged model from MERGETUNE surpasses ensemble baselines with lower inference cost, achieving further gains and state-of-the-art results when ensembled with the zero-shot model. Our code is available at https://github.com/Surrey-UP-Lab/MERGETUNE.
Read the full abstract ▾
All authors Wenqing Wang, Da Li, Xiatian Zhu, Josef Kittler
Computer Vision
Yu Wang, Yi Wang, Rui Dai et al. (+4 more)
As hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various entities from satellite imagery is crucial for a range of downstream applications. Current advanced
segmentation models can reliably segment entities defined by physical attributes (e.g., buildings, water bodies) but still struggle with socially defined categories (e.g., schools, parks). In this work, we achieve socio-semantic
segmentation by
vision-
language model
reasoning. To facilitate this, we introduce the Urban Socio-Semantic
Segmentation dataset named SocioSeg, a new resource comprising satellite imagery, digital maps, and pixel-level labels of social semantic entities organized in a hierarchical structure. Additionally, we propose a novel
vision-
language reasoning framework called SocioReasoner that simulates the human process of identifying and annotating social semantic entities via cross-modal recognition and multi-stage
reasoning. We employ reinforcement learning to optimize this non-differentiable process and elicit the
reasoning capabilities of the
vision-
language model. Experiments demonstrate our approach's gains over state-of-the-art models and strong zero-shot generalization. The
dataset and code are open-sourced under the Apache License 2.0 at https://github.com/AMAP-ML/SocioReasoner.
Read the full abstract ▾
All authors Yu Wang, Yi Wang, Rui Dai, Yujie Wang, Kaikui Liu, Xiangxiang Chu, Yansheng Li
Computer Vision · Cross-listed from cs.AI
ChartComplete: A Taxonomy-based Inclusive Chart Dataset
Ahmad Mustapha, Charbel Toumieh and Mariette Awad
With advancements in deep learning (DL) and computer
vision techniques, the field of chart understanding is evolving rapidly. In particular,
multimodal large
language models (MLLMs) are proving to be efficient and accurate in understanding charts. To accurately measure the performance of MLLMs, the research community has developed multiple
datasets to serve as
benchmarks. By examining these
datasets, we found that they are all limited to a small set of chart types. To bridge this gap, we propose the ChartComplete
dataset. The
dataset is based on a chart taxonomy borrowed from the visualization community, and it covers thirty different chart types. The
dataset is a collection of classified chart images and does not include a learning signal. We present the ChartComplete
dataset as is to the community to build upon it.
Read the full abstract ▾
Computer Vision
Lunar-G2R: Geometry-to-Reflectance Learning for High-Fidelity Lunar BRDF Estimation
Clementine Grethen, Nicolas Menga, Roland Brochard et al. (+4 more)
We address the problem of estimating realistic, spatially varying reflectance for complex planetary surfaces such as the lunar regolith, which is critical for high-fidelity rendering and
vision-based
navigation. Existing lunar rendering pipelines rely on simplified or spatially uniform BRDF models whose parameters are difficult to estimate and fail to capture local reflectance variations, limiting photometric realism. We propose Lunar-G2R, a geometry-to-reflectance learning framework that predicts spatially varying BRDF parameters directly from a lunar digital elevation model (DEM), without requiring multi-view imagery, controlled illumination, or dedicated reflectance-capture hardware at inference time. The method leverages a U-Net trained with differentiable rendering to minimize photometric discrepancies between real orbital images and physically based renderings under known viewing and illumination geometry. Experiments on a geographically held-out region of the Tycho crater show that our approach reduces photometric error by 38 % compared to a state-of-the-art baseline, while achieving higher PSNR and SSIM and improved perceptual similarity, capturing fine-scale reflectance variations absent from spatially uniform models. To our knowledge, this is the first method to infer a spatially varying reflectance model directly from terrain geometry.
Read the full abstract ▾
All authors Clementine Grethen, Nicolas Menga, Roland Brochard, Geraldine Morin, Simone Gasparini, Jeremy Lebreton, Manuel Sanchez Gestido
Computer Vision · Cross-listed from cs.MM
Subjective evaluation of UHD video coded using VVC with LCEVC and ML-VVC
Naeem Ramzan and Muhammad Tufail Khan
This paper presents the results of a subjective quality assessment of a multilayer
video coding configuration in which Low Complexity Enhancement
Video Coding (LCEVC) is applied as an enhancement layer on top of a Versatile
Video Coding (VVC) base layer. The evaluation follows the same test methodology and conditions previously defined for MPEG multilayer
video coding assessments, with the LCEVC enhancement layer encoded using version 8.1 of the LCEVC Test Model (LTM). The test compares reconstructed UHD output generated from an HD VVC base layer with LCEVC enhancement against two reference cases: upsampled VVC base layer decoding and multilayer VVC (ML-VVC). Two operating points are considered, corresponding to enhancement layers representing approximately 10% and 50% of the total bitrate. Subjective assessment was conducted using the Degradation Category Rating (DCR) methodology with twenty five participants, across a
dataset comprising fifteen SDR and HDR sequences. The reported results include Mean Opinion Scores (MOS) with associated 95% confidence intervals, enabling comparison of perceptual quality across coding approaches and operating points within the defined test scope.
Read the full abstract ▾
Computer Vision
Multi-Temporal Frames Projection for Dynamic Processes Fusion in Fluorescence Microscopy
Hassan Eshkiki, Sarah Costa, Mostafa Mohammadpour et al. (+3 more)
Fluorescence microscopy is widely employed for the analysis of living biological samples; however, the utility of the resulting recordings is frequently constrained by noise, temporal variability, and inconsistent visualisation of signals that oscillate over time. We present a unique computational framework that integrates information from multiple time-resolved frames into a single high-quality image, while preserving the underlying biological content of the original
video. We evaluate the proposed method through an extensive number of configurations (n = 111) and on a challenging
dataset comprising dynamic, heterogeneous, and morphologically complex 2D monolayers of cardiac
cells. Results show that our framework, which consists of a combination of explainable techniques from different computer
vision application fields, is capable of generating composite images that preserve and enhance the quality and information of individual microscopy frames, yielding 44% average increase in
cell count compared to previous methods. The proposed pipeline is applicable to other imaging domains that require the fusion of multi-temporal image stacks into high-quality 2D images, thereby facilitating annotation and downstream
segmentation.
Read the full abstract ▾
All authors Hassan Eshkiki, Sarah Costa, Mostafa Mohammadpour, Farinaz Tanhaei, Christopher H. George, Fabio Caraffini
Computer Vision
Handling Missing Modalities in Multimodal Survival Prediction for Non-Small Cell Lung Cancer
Filippo Ruffini, Camillo Maria Caruso, Claudia Tacconi et al. (+16 more)
Accurate survival prediction in Non-Small
Cell Lung Cancer (NSCLC) requires integrating clinical, radiological, and histopathological data.
Multimodal Deep Learning (MDL) can improve precision prognosis, but small cohorts and missing modalities limit its clinical applicability, as conventional approaches enforce complete case filtering or imputation. We present a missing-aware
multimodal survival framework that combines Computed
Tomography (CT), Whole-Slide Histopathology Images (WSI), and structured clinical variables for overall survival modeling in unresectable stage II-III NSCLC. The framework uses Foundation Models (FMs) for modality-specific feature extraction and a missing-aware encoding strategy that enables intermediate
multimodal fusion under naturally incomplete modality profiles. By design, the architecture processes all available data without dropping patients during training or inference. Intermediate fusion outperforms unimodal baselines and both early and late fusion strategies, with the trimodal configuration reaching a C-index of 74.42. Modality-importance analyses show that the fusion model adapts its reliance on each data stream according to representation informativeness, shaped by the
alignment between FM pretraining objectives and the survival task. The learned risk scores produce clinically meaningful stratification of disease progression and metastatic risk, with statistically significant log-rank tests across all modality combinations, supporting the translational relevance of the proposed framework.
Read the full abstract ▾
All authors Filippo Ruffini, Camillo Maria Caruso, Claudia Tacconi, Lorenzo Nibid, Francesca Miccolis, Marta Lovino, Carlo Greco, Edy Ippolito, Michele Fiore, Alessio Cortellini, Bruno Beomonte Zobel, Giuseppe Perrone, Bruno Vincenzi, Claudio Marrocco, Alessandro Bria, Elisa Ficarra, Sara Ramella, Valerio Guarrasi, Paolo Soda
Computer Vision
Global Context Compression with Interleaved Vision-Text Transformation
Dian Jiao, Jiaxin Duan, Shuai Zhao et al. (+3 more)
Recent achievements of
vision-
language models in end-to-end OCR point to a new avenue for low-loss compression of textual information. This motivates earlier works that render the
Transformer's input into images for prefilling, which effectively reduces the number of
tokens through visual encoding, thereby alleviating the quadratically increased
Attention computations. However, this partial compression fails to save computational or
memory costs at
token-by-
token inference. In this paper, we investigate global context compression, which saves
tokens at both prefilling and inference stages. Consequently, we propose VIST2, a novel
Transformer that interleaves input text chunks alongside their visual encoding, while depending exclusively on visual
tokens in the pre-context to predict the next text
token distribution. Around this idea, we render text chunks into sketch images and train VIST2 in multiple stages, starting from curriculum-scheduled pretraining for optical
language modeling, followed by modal-interleaved instruction tuning. We conduct extensive experiments using VIST2 families scaled from 0.6B to 8B to explore the training recipe and hyperparameters. With a 4$\times$ compression ratio, the resulting models demonstrate significant superiority over baselines on long writing tasks, achieving, on average, a 3$\times$ speedup in first-
token generation, 77% reduction in
memory usage, and 74% reduction in FLOPS. Our codes and
datasets will be public to support further studies.
Read the full abstract ▾
All authors Dian Jiao, Jiaxin Duan, Shuai Zhao, Jiabing Leng, Yiran Zhang, Feng Huang
Computer Vision
Towards Efficient Low-rate Image Compression with Frequency-aware Diffusion Prior Refinement
Yichong Xia, Yimin Zhou, Jinpeng Wang et al. (+1 more)
Recent advancements in
diffusion-based generative priors have enabled visually plausible image compression at extremely low bit rates. However, existing approaches suffer from slow sampling processes and suboptimal bit allocation due to fragmented training paradigms. In this work, we propose Accelerate
Diffusion-based Image Compression via Consistency Prior Refinement (DiffCR), a novel compression framework for efficient and high-fidelity image reconstruction. At the heart of DiffCR is a Frequency-aware Skip Estimation (FaSE) module that refines the $\epsilon$-prediction prior from a pre-trained latent
diffusion model and aligns it with compressed latents at different timesteps via Frequency Decoupling
Attention (FDA). Furthermore, a lightweight consistency
estimator enables fast two-step decoding by preserving the semantic trajectory of
diffusion sampling. Without updating the backbone
diffusion model, DiffCR achieves substantial bitrate savings (27.2% BD-rate (LPIPS) and 65.1% BD-rate (PSNR)) and over $10\times$ speed-up compared to SOTA
diffusion-based compression baselines.
Read the full abstract ▾
All authors Yichong Xia, Yimin Zhou, Jinpeng Wang, Bin Chen
Computer Vision
Fine-Grained Human Pose Editing Assessment via Layer-Selective MLLMs
Ningyu Sun, Zhaolin Cai, Zitong Xu et al. (+5 more)
Text-guided human pose editing has gained significant traction in AIGC applications. However,it remains plagued by structural anomalies and generative artifacts. Existing evaluation metrics often isolate authenticity detection from quality assessment, failing to provide fine-grained insights into pose-specific inconsistencies. To address these limitations, we introduce HPE-Bench, a specialized
benchmark comprising 1,700 standardized samples from 17 state-of-the-art editing models, offering both authenticity labels and multi-dimensional quality scores. Furthermore, we propose a unified framework based on layer-selective
multimodal large
language models (MLLMs). By employing contrastive LoRA tuning and a novel layer sensitivity analysis (LSA) mechanism, we identify the optimal feature layer for pose evaluation. Our framework achieves superior performance in both authenticity detection and multi-dimensional quality regression, effectively bridging the gap between forensic detection and quality assessment.
Read the full abstract ▾
All authors Ningyu Sun, Zhaolin Cai, Zitong Xu, Peihang Chen, Huiyu Duan, Yichao Yan, Xiongkuo Min, Xiaokang Yang
Computer Vision
An analytic theory of convolutional neural network inverse problems solvers
Minh Hai Nguyen, Quoc Bao Do, Edouard Pauwels et al. (+1 more)
Supervised convolutional neural networks (CNNs) are widely used to solve imaging inverse problems, achieving state-of-the-art performance in numerous applications. However, despite their empirical success, these methods are poorly understood from a theoretical perspective and often treated as black boxes. To bridge this gap, we analyze trained neural networks through the lens of the Minimum Mean Square Error (MMSE)
estimator, incorporating functional constraints that capture two fundamental inductive biases of CNNs: translation equivariance and locality via finite receptive fields. Under the empirical training distribution, we derive an analytic, interpretable, and tractable formula for this constrained variant, termed Local-Equivariant MMSE (LE-MMSE). Through extensive numerical experiments across various inverse problems (denoising, inpainting, deconvolution),
datasets (FFHQ, CIFAR-10, FashionMNIST), and architectures (U-Net, ResNet, PatchMLP), we demonstrate that our theory matches the neural networks outputs (PSNR $\gtrsim25$dB). Furthermore, we provide insights into the differences between physics-aware and physics-agnostic
estimators, the impact of high-density regions in the training (patch) distribution, and the influence of other factors (
dataset size, patch size, etc).
Read the full abstract ▾
All authors Minh Hai Nguyen, Quoc Bao Do, Edouard Pauwels, Pierre Weiss
Computer Vision
Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoders
Siqi Kou, Jiachun Jin, Zetong Zhou et al. (+8 more)
Recent progress in text-to-image (T2I)
diffusion models (DMs) has enabled high-quality visual synthesis from diverse textual
prompts. Yet, most existing T2I DMs, even those equipped with large
language model (LLM)-based text encoders, remain text-pixel mappers – they employ LLMs merely as text encoders, without leveraging their inherent
reasoning capabilities to infer what should be visually depicted given the textual
prompt. To move beyond such literal generation, we propose the think-then-generate (T2G) paradigm, where the LLM-based text encoder is encouraged to reason about and rewrite raw user
prompts; the states of the rewritten
prompts then serve as
diffusion conditioning. To achieve this, we first activate the think-then-rewrite pattern of the LLM encoder with a lightweight supervised fine-tuning process. Subsequently, the LLM encoder and
diffusion backbone are co-optimized to ensure faithful
reasoning about the context and accurate rendering of the semantics via Dual-GRPO. In particular, the text encoder is reinforced using image-grounded rewards to infer and recall world knowledge, while the
diffusion backbone is pushed to produce semantically consistent and visually coherent images. Experiments show substantial improvements in factual consistency, semantic
alignment, and visual realism across
reasoning-based image generation and editing
benchmarks, achieving 0.79 on WISE score, nearly on par with GPT-4. Our results constitute a promising step toward next-generation unified models with
reasoning, expression, and demonstration capacities.
Read the full abstract ▾
All authors Siqi Kou, Jiachun Jin, Zetong Zhou, Ye Ma, Yugang Wang, Quan Chen, Peng Jiang, Xiao Yang, Jun Zhu, Kai Yu, Zhijie Deng
Computer Vision
SRAW-Attack: Space-Reweighted Adversarial Warping Attack for SAR Target Recognition
Yiming Zhang, Weibo Qin, Yuntian Liu et al. (+1 more)
Synthetic aperture radar (SAR) imagery exhibits intrinsic information sparsity due to its unique electromagnetic scattering mechanism. Despite the widespread adoption of deep neural network (DNN)-based SAR automatic target recognition (SAR-ATR) systems, they remain vulnerable to
adversarial examples and tend to over-rely on background regions, leading to degraded
adversarial robustness. Existing
adversarial attacks for SAR-ATR often require visually perceptible distortions to achieve effective performance, thereby necessitating an attack method that balances effectiveness and stealthiness. In this paper, a novel attack method termed Space-Reweighted
Adversarial Warping (SRAW) is proposed, which generates
adversarial examples through optimized spatial deformation with reweighted budgets across foreground and background regions. Extensive experiments demonstrate that SRAW significantly degrades the performance of state-of-the-art SAR-ATR models and consistently outperforms existing methods in terms of imperceptibility and
adversarial transferability. Code is made available at https://github.com/boremycin/SAR-ATR-TransAttack.
Read the full abstract ▾
All authors Yiming Zhang, Weibo Qin, Yuntian Liu, Feng Wang
Computer Vision
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
Xueyun Tian, Wei Li, Bingbing Xu et al. (+3 more)
Recent Omni-
multimodal Large
Language Models show promise in unified audio,
vision, and text modeling. However, streaming audio-
video understanding remains challenging, as existing approaches suffer from disjointed capabilities: they typically exhibit incomplete modality support or lack autonomous proactive monitoring. To address this, we present ROMA, a real-time omni-
multimodal assistant for unified reactive and proactive interaction. ROMA processes continuous inputs as synchronized
multimodal units, aligning dense audio with discrete
video frames to handle granularity mismatches. For online decision-making, we introduce a lightweight speak head that decouples response initiation from generation to ensure precise triggering without task conflict. We train ROMA with a curated streaming
dataset and a two-stage curriculum that progressively optimizes for streaming format adaptation and proactive responsiveness. To standardize the fragmented evaluation landscape, we reorganize diverse
benchmarks into a unified suite covering both proactive (alert, narration) and reactive (QA) settings. Extensive experiments across 12
benchmarks demonstrate ROMA achieves state-of-the-art performance on proactive tasks while competitive in reactive settings, validating its robustness in unified real-time omni-
multimodal understanding.
Read the full abstract ▾
All authors Xueyun Tian, Wei Li, Bingbing Xu, Heng Dong, Yuanzhuo Wang, Huawei Shen
Computer Vision
Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Models
Peng-Fei Zhang and Zi Huang
Existing
adversarial attacks for VLP models are mostly sample-specific, resulting in substantial computational overhead when scaled to large
datasets or new scenarios. To overcome this limitation, we propose Hierarchical Refinement Attack (HRA), a
multimodal universal attack framework for VLP models. For the image modality, we refine the optimization path by leveraging a temporal hierarchy of historical and estimated future gradients to avoid local minima and stabilize universal perturbation learning. For the text modality, it hierarchically models textual importance by considering both intra- and inter-sentence contributions to identify globally influential words, which are then used as universal text perturbations. Extensive experiments across various downstream tasks, VLP models, and
datasets, demonstrate the superior transferability of the proposed universal
multimodal attacks.
Read the full abstract ▾