Computer Vision
BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation
Yan Li, Zezi Zeng, Ziwei Zhou et al. (+13 more)
Recent advances in image generation models have expanded their applications beyond aesthetic imagery toward practical visual content creation. However, existing
benchmarks mainly focus on natural image synthesis and fail to systematically evaluate models under the structured and multi-constraint requirements of real-world commercial design tasks. In this work, we introduce BizGenEval, a systematic
benchmark for commercial visual content generation. The
benchmark spans five representative document types: slides, charts, webpages, posters, and scientific figures, and evaluates four key capability dimensions: text rendering, layout control, attribute binding, and knowledge-based
reasoning, forming 20 diverse evaluation tasks. BizGenEval contains 400 carefully curated
prompts and 8000 human-verified checklist questions to rigorously assess whether generated images satisfy complex visual and semantic constraints. We conduct large-scale benchmarking on 26 popular image generation systems, including state-of-the-art commercial APIs and leading open-source models. The results reveal substantial capability gaps between current generative models and the requirements of professional visual content creation. We hope BizGenEval serves as a standardized
benchmark for real-world commercial visual content generation.
Read the full abstract ▾
All authors Yan Li, Zezi Zeng, Ziwei Zhou, Xin Gao, Muzhao Tian, Yifan Yang, Mingxi Cheng, Qi Dai, Yuqing Yang, Lili Qiu, Zhendong Wang, Zhengyuan Yang, Xue Yang, Lijuan Wang, Ji Li, Chong Luo
Computer Vision
PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference
Xiaofeng Mao, Shaohao Rui, Kaining Ying et al. (+4 more)
Autoregressive
video diffusion models have demonstrated remarkable progress, yet they remain bottlenecked by intractable linear KV-cache growth, temporal repetition, and compounding errors during long-
video generation. To address these challenges, we present PackForcing, a unified framework that efficiently manages the generation history through a novel three-partition KV-cache strategy. Specifically, we categorize the historical context into three distinct types: (1) Sink
tokens, which preserve early anchor frames at full resolution to maintain global semantics; (2) Mid
tokens, which achieve a massive spatiotemporal compression (32x
token reduction) via a dual-branch network fusing progressive 3D convolutions with low-resolution VAE re-encoding; and (3) Recent
tokens, kept at full resolution to ensure local temporal coherence. To strictly bound the
memory footprint without sacrificing quality, we introduce a dynamic top-$k$ context selection mechanism for the mid
tokens, coupled with a continuous Temporal RoPE Adjustment that seamlessly re-aligns position gaps caused by dropped
tokens with negligible overhead. Empowered by this principled hierarchical context compression, PackForcing can generate coherent 2-minute, 832x480
videos at 16 FPS on a single H200 GPU. It achieves a bounded KV cache of just 4 GB and enables a remarkable 24x temporal extrapolation (5s to 120s), operating effectively either zero-shot or trained on merely 5-second clips. Extensive results on VBench demonstrate state-of-the-art temporal consistency (26.07) and dynamic degree (56.25), proving that short-
video supervision is sufficient for high-quality, long-
video synthesis. https://github.com/ShandaAI/PackForcing
Read the full abstract ▾
All authors Xiaofeng Mao, Shaohao Rui, Kaining Ying, Bo Zheng, Chuanhao Li, Mingmin Chi, Kaipeng Zhang
Computer Vision
PixelSmile: Toward Fine-Grained Facial Expression Editing
Jiabin Hua, Hengyuan Xu, Aojie Li et al. (+4 more)
Fine-grained facial expression editing has long been limited by intrinsic semantic overlap. To address this, we construct the Flex Facial Expression (FFE)
dataset with continuous affective annotations and establish FFE-Bench to evaluate structural confusion, editing accuracy, linear controllability, and the trade-off between expression editing and identity preservation. We propose PixelSmile, a
diffusion framework that disentangles expression semantics via fully symmetric joint training. PixelSmile combines intensity supervision with contrastive learning to produce stronger and more distinguishable expressions, achieving precise and stable linear expression control through textual latent interpolation. Extensive experiments demonstrate that PixelSmile achieves superior disentanglement and robust identity preservation, confirming its effectiveness for continuous, controllable, and fine-grained expression editing, while naturally supporting smooth expression blending.
Read the full abstract ▾
All authors Jiabin Hua, Hengyuan Xu, Aojie Li, Wei Cheng, Gang Yu, Xingjun Ma, Yu-Gang Jiang
Computer Vision
AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation
Chen Si, Yulin Liu, Bo Ai et al. (+4 more)
We present AnyHand, a large-scale synthetic
dataset designed to advance the state of the art in 3D hand pose estimation. While recent works with foundation approaches have shown that scaling training data markedly improves hand pose estimation, existing real-world
datasets are limited in coverage, and prior synthetic
datasets rarely provide occlusions, arm details, and aligned depth together at scale. To address this bottleneck, our proposed AnyHand contains 2.5M single-hand and 4.1M hand-object interaction RGB-D images, with rich
geometric annotations. We show that extending the original training data recipes of existing RGB baselines with AnyHand yields significant gains on multiple
benchmarks (FreiHAND and HO-3D), even when keeping the architectures and training schemes fixed. Together with extensive ablations on the scale and composition of the training data setups, these results suggest that training data diversity and quality are as critical as scale for advancing hand pose estimation. We further examine the utility of AnyHand's aligned depth maps in the appendix, showing that scaling RGB-D supervision with AnyHand allows a lightweight depth-fusion variant of existing RGB baselines to outperform prior RGB-D methods.
Read the full abstract ▾
All authors Chen Si, Yulin Liu, Bo Ai, Jianwen Xie, Rolandos Alexandros Potamias, Chuanxia Zheng, Hao Su
Computer Vision
No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models
Hai X. Pham, David T. Hoffmann, Ricardo Guerrero et al. (+1 more)
Contrastive
vision-
language (V&L) models remain a popular choice for various applications. However, several limitations have emerged, most notably the limited ability of V&L models to learn compositional representations. Prior methods often addressed this limitation by generating custom training data to obtain hard negative samples. Hard negatives have been shown to improve performance on compositionality tasks, but are often specific to a single
benchmark, do not generalize, and can cause substantial degradation of basic V&L capabilities such as zero-shot or
retrieval performance, rendering them impractical. In this work we follow a different approach. We identify two root causes that limit compositionality performance of V&Ls: 1) Long training captions do not require a compositional representation; and 2) The final global pooling in the text and image encoders lead to a complete loss of the necessary information to learn binding in the first place. As a remedy, we propose two simple solutions: 1) We obtain short concept centric caption parts using standard NLP software and align those with the image; and 2) We introduce a parameter-free cross-modal
attention-pooling to obtain concept centric visual
embeddings from the image encoder. With these two changes and simple auxiliary contrastive losses, we obtain SOTA performance on standard compositionality
benchmarks, while maintaining or improving strong zero-shot and
retrieval capabilities. This is achieved without increasing inference cost. We release the code for this work at https://github.com/saic-fi/concept_centric_clip.
Read the full abstract ▾
All authors Hai X. Pham, David T. Hoffmann, Ricardo Guerrero, Brais Martinez
Computer Vision · Cross-listed from cs.AI
R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoning
Zirui Zhang, Haoyu Dong, Kexin Pei et al. (+1 more)
Robust perception and
reasoning require consistency across sensory modalities. Yet current
multimodal models often violate this principle, yielding contradictory predictions for visual and textual representations of the same concept. Rather than masking these failures with standard voting mechanisms, which can amplify systematic biases, we show that cross-modal inconsistency provides a rich and natural signal for learning. We introduce RC2, a reinforcement learning framework that resolves internal conflicts by enforcing cross-modal cycle consistency. By requiring a model to perform backward inference, switch modalities, and reliably reconstruct the answer through forward inference, we obtain a dense, label-free reward. This cyclic constraint encourages the model to align its internal representations autonomously. Optimizing for this structure mitigates modality-specific errors and improves
reasoning accuracy by up to 7.6 points. Our results suggest that advanced
reasoning emerges not only from scaling data, but also from enforcing a structurally consistent understanding of the world.
Read the full abstract ▾
All authors Zirui Zhang, Haoyu Dong, Kexin Pei, Chengzhi Mao
Computer Vision
Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models
Kaijin Chen, Dingkang Liang, Xin Zhou et al. (+4 more)
Video world models have shown immense potential in simulating the physical world, yet existing
memory mechanisms primarily treat environments as static canvases. When dynamic subjects hide out of sight and later re-emerge, current methods often struggle, leading to frozen, distorted, or vanishing subjects. To address this, we introduce Hybrid
Memory, a novel paradigm requiring models to simultaneously act as precise archivists for static backgrounds and vigilant trackers for dynamic subjects, ensuring motion continuity during out-of-view intervals. To facilitate research in this direction, we construct HM-World, the first large-scale
video dataset dedicated to hybrid
memory. It features 59K high-fidelity clips with decoupled camera and subject trajectories, encompassing 17 diverse scenes, 49 distinct subjects, and meticulously designed exit-entry events to rigorously evaluate hybrid coherence. Furthermore, we propose HyDRA, a specialized
memory architecture that compresses
memory into
tokens and utilizes a spatiotemporal relevance-driven
retrieval mechanism. By selectively attending to relevant motion cues, HyDRA effectively preserves the identity and motion of hidden subjects. Extensive experiments on HM-World demonstrate that our method significantly outperforms state-of-the-art approaches in both dynamic subject consistency and overall generation quality. Code is publicly available at https://github.com/H-EmbodVis/HyDRA.
Read the full abstract ▾
All authors Kaijin Chen, Dingkang Liang, Xin Zhou, Yikang Ding, Xiaoqiang Liu, Pengfei Wan, Xiang Bai
Computer Vision
Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs
Vishal Narnaware, Animesh Gupta, Kevin Zhai et al. (+2 more)
Multimodal Diffusion Large
Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to
multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate
tokens based on textual likelihood without verifying localized visual support. We establish that this
language-only ranking induces an objective mismatch, where
language probability mass acts as a misspecified proxy for the intended
multimodal task. Consequently, we reinterpret
hallucination as a localized optimization error, a phenomenon where the decoder exploits
language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial
entropy of cross-
attention distributions. By enforcing a localization consensus across
attention heads, the method penalizes spatially uniform distributions and re-ranks
token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across
hallucination-sensitive and general-purpose
benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.
Read the full abstract ▾
All authors Vishal Narnaware, Animesh Gupta, Kevin Zhai, Zhenyi Wang, Mubarak Shah
Computer Vision
TRACE: Object Motion Editing in Videos with First-Frame Trajectory Guidance
Quynh Phung, Long Mai, Cusuh Ham et al. (+3 more)
We study object motion path editing in
videos, where the goal is to alter a target object's trajectory while preserving the original scene content. Unlike prior
video editing methods that primarily manipulate appearance or rely on point-track-based trajectory control, which is often challenging for users to provide during inference, especially in
videos with camera motion, we offer a practical, easy-to-use approach to controllable object-centric motion editing. We present Trace, a framework that enables users to design the desired trajectory in a single anchor frame and then synthesizes a temporally consistent edited
video. Our approach addresses this task with a two-stage pipeline: a cross-view motion transformation module that maps first-frame path design to frame-aligned box trajectories under camera motion, and a motion-conditioned
video re-synthesis module that follows these trajectories to regenerate the object while preserving the remaining content of the input
video. Experiments on diverse real-world
videos show that our method produces more coherent, realistic, and controllable motion edits than recent image-to-
video and
video-to-
video methods.
Read the full abstract ▾
All authors Quynh Phung, Long Mai, Cusuh Ham, Feng Liu, Jia-Bin Huang, Aniruddha Mahapatra
Computer Vision
Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training
Jinbo Xing, Zeyinzi Jiang, Yuxiang Tuo et al. (+15 more)
Recent unified models have made unprecedented progress in both understanding and generation. However, while most of them accept multi-modal inputs, they typically produce only single-modality outputs. This challenge of producing interleaved content is mainly due to training data scarcity and the difficulty of modeling long-range cross-modal context. To address this issue, we decompose interleaved generation into textual planning and visual consistency modeling, and introduce a framework consisting of a planner and a visualizer. The planner produces dense textual descriptions for visual content, while the visualizer synthesizes images accordingly. Under this guidance, we construct large-scale textual-proxy interleaved data (where visual content is represented in text) to train the planner, and curate reference-guided image data to train the visualizer. These designs give rise to Wan-Weaver, which exhibits emergent interleaved generation ability with long-range textual coherence and visual consistency. Meanwhile, the integration of diverse understanding and generation data into planner training enables Wan-Weaver to achieve robust task
reasoning and generation proficiency. To assess the model's capability in interleaved generation, we further construct a
benchmark that spans a wide range of use cases across multiple dimensions. Extensive experiments demonstrate that, even without access to any real interleaved data, Wan-Weaver achieves superior performance over existing methods.
Read the full abstract ▾
All authors Jinbo Xing, Zeyinzi Jiang, Yuxiang Tuo, Chaojie Mao, Xiaotang Gai, Xi Chen, Jingfeng Zhang, Yulin Pan, Zhen Han, Jie Xiao, Keyu Yan, Chenwei Xie, Chongyang Zhong, Kai Zhu, Tong Shen, Lianghua Huang, Yu Liu, Yujiu Yang
Computer Vision
LEMMA: Laplacian pyramids for Efficient Marine SeMAntic Segmentation
Ishaan Gakhar, Laven Srivastava, Sankarshanaa Sagaram et al. (+2 more)
Semantic
segmentation in marine environments is crucial for the autonomous
navigation of unmanned surface vessels (USVs) and coastal Earth Observation events such as oil spills. However, existing methods, often relying on deep CNNs and
transformer-based architectures, face challenges in deployment due to their high computational costs and resource-intensive nature. These limitations hinder the practicality of real-time, low-cost applications in real-world marine settings. To address this, we propose LEMMA, a lightweight semantic
segmentation model designed specifically for accurate remote sensing
segmentation under resource constraints. The proposed architecture leverages
Laplacian Pyramids to enhance edge recognition, a critical component for effective feature extraction in complex marine environments for disaster response, environmental surveillance, and coastal monitoring. By integrating edge information early in the feature extraction process, LEMMA eliminates the need for computationally expensive feature map computations in deeper network layers, drastically reducing model size, complexity and inference time. LEMMA demonstrates state-of-the-art performance across
datasets captured from diverse platforms while reducing trainable parameters and computational requirements by up to 71x, GFLOPs by up to 88.5%, and inference time by up to 84.65%, as compared to existing models. Experimental results highlight its effectiveness and real-world applicability, including 93.42% IoU on the Oil Spill
dataset and 98.97% mIoU on Mastr1325.
Read the full abstract ▾
All authors Ishaan Gakhar, Laven Srivastava, Sankarshanaa Sagaram, Aditya Kasliwal, Ujjwal Verma
Computer Vision
Just Zoom In: Cross-View Geo-Localization via Autoregressive Zooming
Yunus Talha Erzurumlu, Jiyong Kwag and Alper Yilmaz
Cross-view geo-localization (CVGL) estimates a camera's location by matching a street-view image to geo-referenced overhead imagery, enabling GPS-denied localization and
navigation. Existing methods almost universally formulate CVGL as an image-
retrieval problem in a contrastively trained
embedding space. This ties performance to large batches and hard negative mining, and it ignores both the
geometric structure of maps and the coverage mismatch between street-view and overhead imagery. In particular, salient landmarks visible from the street view can fall outside a fixed satellite crop, making
retrieval targets ambiguous and limiting explicit spatial inference over the map. We propose Just Zoom In, an alternative formulation that performs CVGL via autoregressive zooming over a city-scale overhead map. Starting from a coarse satellite view, the model takes a short sequence of zoom-in decisions to select a terminal satellite
cell at a target resolution, without contrastive losses or hard negative mining. We further introduce a realistic
benchmark with crowd-sourced street views and high-resolution satellite imagery that reflects real capture conditions. On this
benchmark, Just Zoom In achieves state-of-the-art performance, improving Recall@1 within 50 m by 5.5% and Recall@1 within 100 m by 9.6% over the strongest contrastive-
retrieval baseline. These results demonstrate the effectiveness of sequential coarse-to-fine spatial
reasoning for cross-view geo-localization.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning
Jai Bardhan, Patrik Drozdik, Josef Sivic et al. (+1 more)
Action-conditioned
robot world models generate future
video frames of the manipulated scene given a
robot action sequence, offering a promising alternative for simulating tasks that are difficult to model with traditional physics engines. However, these models are optimized for short-term prediction and break down when deployed autoregressively: each predicted clip feeds back as context for the next, causing errors to compound and visual quality to rapidly degrade. We address this through the following contributions. First, we introduce a reinforcement learning (RL) post-training scheme that trains the world model on its own autoregressive rollouts rather than on ground-truth histories. We achieve this by adapting a recent contrastive RL objective for
diffusion models to our setting and show that its convergence guarantees carry over exactly. Second, we design a training protocol that generates and compares multiple candidate variable-length futures from the same rollout state, reinforcing higher-fidelity predictions over lower-fidelity ones. Third, we develop efficient, multi-view visual fidelity rewards that combine complementary perceptual metrics across camera views and are aggregated at the clip level for dense, low-variance training signal. Fourth, we show that our approach establishes a new state-of-the-art for rollout fidelity on the DROID
dataset, outperforming the strongest baseline on all metrics (e.g., LPIPS reduced by 14% on external cameras, SSIM improved by 9.1% on the wrist camera), winning 98% of paired comparisons, and achieving an 80% preference rate in a blind human study.
Read the full abstract ▾
All authors Jai Bardhan, Patrik Drozdik, Josef Sivic, Vladimir Petrik
Computer Vision · Cross-listed from cs.RO
Can Users Specify Driving Speed? Bench2Drive-Speed: Benchmark and Baselines for Desired-Speed Conditioned Autonomous Driving
Yuqian Shao, Xiaosong Jia, Langechuan Liu et al. (+1 more)
End-to-end autonomous driving (E2E-AD) has achieved remarkable progress. However, one practical and useful function has been long overlooked: users may wish to customize the desired speed of the policy or specify whether to allow the autonomous vehicle to overtake. To bridge this gap, we present Bench2Drive-Speed, a
benchmark with metrics,
dataset, and baselines for desired-speed conditioned autonomous driving. We introduce explicit inputs of users' desired target-speed and overtake/follow instructions to driving policy models. We design quantitative metrics, including Speed-Adherence Score and Overtake Score, to measure how faithfully policies follow user specifications, while remaining compatible with standard autonomous driving metrics. To enable training of speed-conditioned policies, one approach is to collect expert demonstrations that strictly follow speed requirements, an expensive and unscalable process in the real world. An alternative is to adapt existing regular driving data by treating the speed observed in future frames as the target speed for training. To investigate this, we construct CustomizedSpeedDataset, composed of 2,100 clips annotated with experts demonstrations, enabling systematic investigation of supervision strategies. Our experiments show that, under proper re-annotation, models trained on regular driving data perform comparably to on expert demonstrations, suggesting that speed supervision can be introduced without additional complex real-world data collection. Furthermore, we find that while target-speed following can be achieved without degrading regular driving performance, executing overtaking commands remains challenging due to the inherent difficulty of interactive behaviors. All code,
datasets and baselines are available at https://github.com/Thinklab-SJTU/Bench2Drive-Speed
Read the full abstract ▾
All authors Yuqian Shao, Xiaosong Jia, Langechuan Liu, Junchi Yan
Computer Vision · Cross-listed from cs.RO
Fast-dVLA: Accelerating Discrete Diffusion VLA to Real-Time Performance
Wenxuan Song, Jiayi Chen, Shuai Chen et al. (+8 more)
This paper proposes a novel approach to address the challenge that pretrained VLA models often fail to effectively improve performance and reduce adaptation costs during standard supervised finetuning (SFT). Some advanced finetuning methods with auxiliary training objectives can improve performance and reduce the number of convergence steps. However, they typically incur significant computational overhead due to the additional losses from auxiliary tasks. To simultaneously achieve the enhanced capabilities of auxiliary training with the simplicity of standard SFT, we decouple the two objectives of auxiliary task training within the parameter space, namely, enhancing general capabilities and fitting task-specific action distributions. To deliver this goal, we only need to train the model to converge on a small-scale task set using two distinct training strategies. The difference between the resulting model parameters can then be interpreted as capability vectors provided by auxiliary tasks. These vectors are then merged with pretrained parameters to form a capability-enhanced meta model. Moreover, when standard SFT is augmented with a lightweight orthogonal regularization loss, the merged model attains performance comparable to auxiliary finetuned baselines with reduced computational overhead. Experimental results demonstrate that this approach is highly effective across diverse
robot tasks. Project page: https://chris1220313648.github.io/Fast-dVLA/
Read the full abstract ▾
All authors Wenxuan Song, Jiayi Chen, Shuai Chen, Jingbo Wang, Pengxiang Ding, Han Zhao, Yikai Qin, Xinhu Zheng, Donglin Wang, Yan Wang, Haoang Li
Computer Vision · Cross-listed from eess.IV
Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos
Abdullah Hamdi, Changchun Yang and Xin Gao
Early screening via colonoscopy is critical for colon cancer prevention, yet developing robust AI systems for this domain is hindered by the lack of densely annotated, long-sequence
video datasets. Existing
datasets predominantly focus on single-class polyp detection and lack the rich spatial, temporal, and linguistic annotations required to evaluate modern
Multimodal Large
Language Models (MLLMs). To address this critical gap, we introduce Colon-Bench, generated via a novel multi-stage agentic workflow. Our pipeline seamlessly integrates temporal proposals, bounding-box tracking, AI-driven visual confirmation, and human-in-the-loop review to scalably annotate full-procedure
videos. The resulting verified
benchmark is unprecedented in scope, encompassing 528
videos, 14 distinct lesion categories (including polyps, ulcers, and bleeding), over 300,000 bounding boxes, 213,000
segmentation masks, and 133,000 words of clinical descriptions. We utilize Colon-Bench to rigorously evaluate state-of-the-art MLLMs across lesion classification, Open-Vocabulary
Video Object
Segmentation (OV-VOS), and
video Visual Question Answering (VQA). The MLLM results demonstrate surprisingly high localization performance in medical domains compared to SAM-3. Finally, we analyze common VQA errors from MLLMs to introduce a novel "colon-skill" prompting strategy, improving zero-shot MLLM performance by up to 9.7% across most MLLMs. The
dataset and the code are available at https://abdullahamdi.com/colon-bench .
Read the full abstract ▾
Computer Vision
Designing Any Imaging System from Natural Language: Agent-Constrained Composition over a Finite Primitive Basis
Chengshuai Yang
Designing a computational imaging system – selecting operators, setting parameters, validating consistency – requires weeks of specialist effort per modality, creating an expertise bottleneck that excludes the broader scientific community from prototyping imaging instruments. We introduce spec.md, a structured specification format, and three autonomous
agents – Plan, Judge, and Execute – that translate a one-sentence natural-
language description into a validated forward model with bounded reconstruction error. A design-to-real error
theorem decomposes total reconstruction error into five independently bounded terms, each linked to a corrective action. On 6 real-data modalities spanning all 5 carrier families, the automated pipeline matches expert-library quality (98.1 +/- 4.2%). Ten novel designs – composing primitives into chains from 3D to 5D – demonstrate compositional reach beyond any single-modality tool.
Read the full abstract ▾
Computer Vision
LanteRn: Latent Visual Structured Reasoning
André G. Viveiros, Nuno Gonçalves, Matthias Lindemann et al. (+1 more)
While
language reasoning models excel in many tasks, visual
reasoning remains challenging for current large
multimodal models (LMMs). As a result, most LMMs default to verbalizing perceptual content into text, a strong limitation for tasks requiring fine-grained spatial and visual understanding. While recent approaches take steps toward thinking with images by invoking tools or generating intermediate images, they either rely on external modules, or incur unnecessary computation by
reasoning directly in pixel space. In this paper, we introduce LanteRn, a framework that enables LMMs to interleave
language with compact latent visual representations, allowing visual
reasoning to occur directly in latent space. LanteRn augments a
vision-
language transformer with the ability to generate and attend to continuous visual thought
embeddings during inference. We train the model in two stages: supervised fine-tuning to ground visual features in latent states, followed by reinforcement learning to align latent
reasoning with task-level utility. We evaluate LanteRn on three perception-centric
benchmarks (VisCoT, V*, and Blink), observing consistent improvements in visual grounding and fine-grained
reasoning. These results suggest that internal latent representations provide a promising direction for more efficient
multimodal reasoning.
Read the full abstract ▾
All authors André G. Viveiros, Nuno Gonçalves, Matthias Lindemann, André Martins
Computer Vision
Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification
Ünsal Öztürk, Hatef Otroshi Shahreza and Sébastien Marcel
Multimodal Large
Language Models (MLLMs) have recently been explored as face verification systems that determine whether two face images are of the same person. Unlike dedicated face recognition systems, MLLMs approach this task through visual prompting and rely on general visual and
reasoning abilities. However, the demographic
fairness of these models remains largely unexplored. In this paper, we present a benchmarking study that evaluates nine open-source MLLMs from six model families, ranging from 2B to 8B parameters, on the IJB-C and RFW face verification protocols across four ethnicity groups and two gender groups. We measure verification accuracy with the Equal Error Rate and True Match Rate at multiple operating points per demographic group, and we quantify demographic disparity with four FMR-based
fairness metrics. Our results show that FaceLLM-8B, the only face-specialised model in our study, substantially outperforms general-purpose MLLMs on both
benchmarks. The bias patterns we observe differ from those commonly reported for traditional face recognition, with different groups being most affected depending on the
benchmark and the model. We also note that the most accurate models are not necessarily the fairest and that models with poor overall accuracy can appear fair simply because they produce uniformly high error rates across all demographic groups.
Read the full abstract ▾
Computer Vision
DeepFAN, a transformer-based deep learning model for human-artificial intelligence collaborative assessment of incidental pulmonary nodules in CT scans: a multi-reader, multi-case trial
Zhenchen Zhu, Ge Hu, Weixiong Tan et al. (+16 more)
The widespread adoption of CT has notably increased the number of detected lung nodules. However, current deep learning methods for classifying benign and malignant nodules often fail to comprehensively integrate global and local features, and most of them have not been validated through clinical trials. To address this, we developed DeepFAN, a
transformer-based model trained on over 10K pathology-confirmed nodules and further conducted a multi-reader, multi-case clinical trial to evaluate its efficacy in assisting junior radiologists. DeepFAN achieved diagnostic area under the curve (AUC) of 0.939 (95% CI 0.930-0.948) on an internal test set and 0.954 (95% CI 0.934-0.973) on the clinical trial
dataset involving 400 cases across three independent medical institutions. Explainability analysis indicated higher contributions from global than local features. Twelve readers' average performance significantly improved by 10.9% (95% CI 8.3%-13.5%) in AUC, 10.0% (95% CI 8.9%-11.1%) in accuracy, 7.6% (95% CI 6.1%-9.2%) in sensitivity, and 12.6% (95% CI 10.9%-14.3%) in specificity (P<0.001 for all). Nodule-level inter-reader diagnostic consistency improved from fair to moderate (overall k: 0.313 vs. 0.421; P=0.019). In conclusion, DeepFAN effectively assisted junior radiologists and may help homogenize diagnostic quality and reduce unnecessary follow-up of indeterminate pulmonary nodules. Chinese Clinical Trial Registry: ChiCTR2400084624.
Read the full abstract ▾
All authors Zhenchen Zhu, Ge Hu, Weixiong Tan, Kai Gao, Chao Sun, Zhen Zhou, Kepei Xu, Wei Han, Meixia Shang, Xiaoming Qiu, Yiqing Tan, Jinhua Wang, Zhoumeng Ying, Li Peng, Wei Song, Lan Song, Zhengyu Jin, Nan Hong, Yizhou Yu
Computer Vision
UNIC: Neural Garment Deformation Field for Real-time Clothed Character Animation
Chengfeng Zhao, Junbo Qi, Yulou Liu et al. (+6 more)
Simulating physically realistic garment deformations is an essential task for virtual immersive experience, which is often achieved by physics simulation methods. However, these methods are typically time-consuming, computationally demanding, and require costly hardware, which is not suitable for real-time applications. Recent learning-based methods tried to resolve this problem by training graph neural networks to learn the garment deformation on vertices, which, however, fail to capture the intricate deformation of complex garment meshes with complex
topologies. In this paper, we introduce a novel neural deformation field-based method, named UNIC, to animate the garments of an avatar in real time, given the motion sequences. Our key idea is to learn the instance-specific neural deformation field to animate the garment meshes. Such an instance-specific learning scheme does not require UNIC to generalize to new garments but only to new motion sequences, which greatly reduces the difficulty in training and improves the deformation quality. Moreover, neural deformation fields map the 3D points to their deformation offsets, which not only avoids handling
topologies of the complex garments but also injects a natural smoothness constraint in the deformation learning. Extensive experiments have been conducted on various kinds of garment meshes to demonstrate the effectiveness and efficiency of UNIC over baseline methods, making it potentially practical and useful in real-world interactive applications like
video games.
Read the full abstract ▾
All authors Chengfeng Zhao, Junbo Qi, Yulou Liu, Zhiyang Dou, Minchen Li, Taku Komura, Ziwei Liu, Wenping Wang, Yuan Liu
Computer Vision
Hierarchy-Guided Multimodal Representation Learning for Taxonomic Inference
Sk Miraj Ahmed, Xi Yu, Yunqi Li et al. (+2 more)
Accurate biodiversity identification from large-scale field data is a foundational problem with direct impact on ecology, conservation, and environmental monitoring. In practice, the core task is taxonomic prediction - inferring order, family, genus, or species from imperfect inputs such as specimen images, DNA barcodes, or both. Existing
multimodal methods often treat taxonomy as a flat label space and therefore fail to encode the hierarchical structure of biological classification, which is critical for robustness under noise and missing modalities. We present two end-to-end variants for hierarchy-aware
multimodal learning: CLiBD-HiR, which introduces Hierarchical Information Regularization (HiR) to shape
embedding geometry across taxonomic levels, yielding structured and noise-robust representations; and CLiBD-HiR-Fuse, which additionally trains a lightweight fusion predictor that supports image-only, DNA-only, or joint inference and is resilient to modality corruption. Across large-scale biodiversity
benchmarks, our approach improves taxonomic classification accuracy by over 14 percent compared to strong
multimodal baselines, with particularly large gains under partial and corrupted DNA conditions. These results highlight that explicitly encoding biological hierarchy, together with flexible fusion, is key for practical biodiversity foundation models.
Read the full abstract ▾
All authors Sk Miraj Ahmed, Xi Yu, Yunqi Li, Yuewei Lin, Wei Xu
Computer Vision
GeoHeight-Bench: Towards Height-Aware Multimodal Reasoning in Remote Sensing
Xuran Hu, Zhitong Xiong, Zhongcheng Hong et al. (+3 more)
Current Large
Multimodal Models (LMMs) in Earth observation are predominantly evaluated on planar optical tasks and often neglect the vertical dimension, although vertical
geometric structure can be critical in applications such as disaster response and urban-morphology analysis. Progress on height-aware
reasoning is also hindered by the absence of systematic evaluation: few
benchmarks pair optical imagery with height products across pixel-, object-, and scene-level
reasoning. To address this gap, we introduce GeoHeight-Bench, a large-scale
benchmark for height-aware remote sensing understanding, together with a more challenging terrain-oriented extension, GeoHeight-Bench+. The
benchmark is constructed through a scalable, VLM-driven generation pipeline that combines metadata extraction with
prompt engineering, and its quality is assessed through a human-in-the-loop verification protocol. To examine whether height-aware
reasoning can be learned from optical imagery, we further provide GeoHeightChat, a height-aware baseline that transfers implicit height-related
geometric representations into an optical LMM. Evaluations of a broad range of closed- and open-source LMMs show that current models remain limited in their ability to reason about height information, while aligning implicit height priors improves most height-dependent tasks. However, several tasks, particularly slope
reasoning and terrain-based flood-susceptibility mapping, remain largely unsolved, highlighting concrete open problems for height-aware GeoAI.
Dataset and Code will be released \href{https://teriri1999.github.io/GeoHeight/}{here}.
Read the full abstract ▾
All authors Xuran Hu, Zhitong Xiong, Zhongcheng Hong, Yifang Ban, Xiaoxiang Zhu, Wufan Zhao
Computer Vision
Towards Comprehensive Real-Time Scene Understanding in Ophthalmic Surgery through Multimodal Image Fusion
Nikolo Rohrmoser, Ghazal Ghazaei, Michael Sommersperger et al. (+1 more)
Purpose: The integration of
multimodal imaging into operating rooms paves the way for comprehensive
surgical scene understanding. In ophthalmic surgery, by now, two complementary imaging modalities are available: operating microscope (OPMI) imaging and real-time intraoperative optical coherence
tomography (iOCT). This first work toward temporal OPMI and iOCT feature fusion demonstrates the potential of
multimodal image processing for multi-head prediction through the example of precise instrument tracking in vitreoretinal surgery. Methods: We propose a
multimodal, temporal, real-time capable network architecture to perform joint instrument detection, keypoint localization, and tool-tissue distance estimation. Our network design integrates a cross-
attention fusion module to merge OPMI and iOCT image features, which are efficiently extracted via a YoloNAS and a CNN encoder, respectively. Furthermore, a region-based recurrent module leverages temporal coherence. Results: Our experiments demonstrate reliable instrument localization and keypoint detection (95.79% mAP50) and show that the incorporation of iOCT significantly improves tool-tissue distance estimation, while achieving real-time processing rates of 22.5 ms per frame. Especially for close distances to the retina (below 1 mm), the distance estimation accuracy improved from 284 $\mu m$ (OPMI only) to 33 $\mu m$ (
multimodal). Conclusion: Feature fusion of
multimodal imaging can enhance multi-task prediction accuracy compared to single-modality processing and real-time processing performance can be achieved through tailored network design. While our results demonstrate the potential of multi-modal processing for image-guided vitreoretinal surgery, they also underline key challenges that motivate future research toward more reliable, consistent, and comprehensive
surgical scene understanding.
Read the full abstract ▾
All authors Nikolo Rohrmoser, Ghazal Ghazaei, Michael Sommersperger, Nassir Navab
Computer Vision
PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos
Yihao Wang, Yang Miao, Wenshuai Zhao et al. (+8 more)
Articulation perception aims to recover the motion and structure of articulated objects (e.g., drawers and cupboards), and is fundamental to 3D scene understanding in robotics, simulation, and animation. Existing learning-based methods rely heavily on supervised training with high-quality 3D data and manual annotations, limiting scalability and diversity. To address this limitation, we propose PAWS, a method that directly extracts object articulations from hand-object interactions in large-scale in-the-wild
egocentric videos. We evaluate our method on the public data sets, including HD-EPIC and Arti4D data sets, achieving significant improvements over baselines. We further demonstrate that the extracted articulations benefit downstream tasks, including fine-tuning 3D articulation prediction models and enabling
robot manipulation. See the project website at https://aaltoml.github.io/PAWS/.
Read the full abstract ▾
All authors Yihao Wang, Yang Miao, Wenshuai Zhao, Wenyan Yang, Zihan Wang, Joni Pajarinen, Luc Van Gool, Danda Pani Paudel, Juho Kannala, Xi Wang, Arno Solin
Computer Vision
Insights on back marking for the automated identification of animals
David Brunner, Marie Bordes, Elisabeth Mayrhuber et al. (+3 more)
To date, there is little research on how to design back marks to best support individual-level monitoring of uniform looking species like pigs. With the recent surge of machine learning-based monitoring solutions, there is a particular need for guidelines on the design of marks that can be effectively recognised by such
algorithms. This study provides valuable insights on effective back mark design, based on the analysis of a machine learning model, trained to distinguish pigs via their back marks. Specifically, a neural network of type ResNet-50 was trained to classify ten pigs with unique back marks. The analysis of the model's predictions highlights the significance of certain design choices. Most importantly, the set of back marks must be designed such that each mark remains unambiguous under conditions of motion blur, diverse view angles and occlusions, caused by animal behaviour. The generated insights can support individual-level monitoring in future studies and real-world applications by optimizing back mark design.
Read the full abstract ▾
All authors David Brunner, Marie Bordes, Elisabeth Mayrhuber, Stephan M. Winkler, Viktoria Dorfer, Maciej Oczak
Computer Vision
BFMD: A Full-Match Badminton Dense Dataset for Dense Shot Captioning
Ning Ding, Keisuke Fujii and Toru Tamaki
Understanding tactical dynamics in badminton requires analyzing entire matches rather than isolated clips. However, existing badminton
datasets mainly focus on short clips or task-specific annotations and rarely provide full-match data with dense
multimodal annotations. This limitation makes it difficult to generate accurate shot captions and perform match-level analysis. To address this limitation, we introduce the first Badminton Full Match Dense (BFMD)
dataset, with 19 broadcast matches (including both singles and doubles) covering over 20 hours of play, comprising 1,687 rallies and 16,751 hit events, each annotated with a shot caption. The
dataset provides hierarchical annotations including match segments, rally events, and dense rally-level
multimodal annotations such as shot types, shuttle trajectories, player pose keypoints, and shot captions. We develop a VideoMAE-based
multimodal captioning framework with a Semantic Feedback mechanism that leverages shot semantics to guide caption generation and improve semantic consistency. Experimental results demonstrate that
multimodal modeling and semantic feedback improve shot caption quality over RGB-only baselines. We further showcase the potential of BFMD by analyzing the temporal
evolution of tactical patterns across full matches.
Read the full abstract ▾
Computer Vision
Beyond the Golden Data: Resolving the Motion-Vision Quality Dilemma via Timestep Selective Training
Xiangyang Luo, Qingyu Li, Yuming Li et al. (+6 more)
Recent advances in
video generation models have achieved impressive results. However, these models heavily rely on the use of high-quality data that combines both high visual quality and high motion quality. In this paper, we identify a key challenge in
video data curation: the Motion-
Vision Quality Dilemma. We discovered that visual quality and motion intensity inherently exhibit a negative correlation, making it hard to obtain golden data that excels in both aspects. To address this challenge, we first examine the hierarchical learning dynamics of
video diffusion models and conduct gradient-based analysis on quality-degraded samples. We discover that quality-imbalanced data can produce gradients similar to golden data at appropriate timesteps. Based on this, we introduce the novel concept of Timestep selection in Training Process. We propose Timestep-aware Quality Decoupling (TQD), which modifies the data sampling distribution to better match the model's learning process. For certain types of data, the sampling distribution is skewed toward higher timesteps for motion-rich data, while high visual quality data is more likely to be sampled during lower timesteps. Through extensive experiments, we demonstrate that TQD enables training exclusively on separated imbalanced data to achieve performance surpassing conventional training with better data, challenging the necessity of perfect data in
video generation. Moreover, our method also boosts model performance when trained on high-quality data, showcasing its effectiveness across different data scenarios.
Read the full abstract ▾
All authors Xiangyang Luo, Qingyu Li, Yuming Li, Guanbo Huang, Yongjie Zhu, Wenyu Qin, Meng Wang, Pengfei Wan, Shao-Lun Huang
Computer Vision
CHIRP dataset: towards long-term, individual-level, behavioral monitoring of bird populations in the wild
Alex Hoi Hang Chan, Neha Singhal, Onur Kocahan et al. (+6 more)
Long-term behavioral monitoring of individual animals is crucial for studying behavioral changes that occur over different time scales, especially for conservation and evolutionary biology. Computer
vision methods have proven to benefit biodiversity monitoring, but automated behavior monitoring in wild populations remains challenging. This stems from the lack of
datasets that cover a range of computer
vision tasks necessary to extract biologically meaningful measurements of individual animals. Here, we introduce such a
dataset (CHIRP) with a new method (CORVID) for individual re-identification of wild birds. The CHIRP (Combining beHaviour, Individual Re-identification and Postures)
dataset is curated from a long-term population of wild Siberian jays studied in Swedish Lapland, supporting re-identification (re-id), action recognition, 2D keypoint estimation, object detection, and instance
segmentation. In addition to traditional task-specific benchmarking, we introduce application-specific benchmarking with biologically relevant metrics (feeding rates, co-occurrence rates) to evaluate the performance of models in real-world use cases. Finally, we present CORVID (COlouR-based
Video re-ID), a novel pipeline for individual identification of birds based on the
segmentation and classification of colored leg rings, a widespread approach for visual identification of individual birds. CORVID offers a probability-based id tracking method by matching the detected combination of color rings with a database. We use application-specific benchmarking to show that CORVID outperforms state-of-the-art re-id methods. We hope this work offers the community a blueprint for curating real-world
datasets from ethically approved biological studies to bridge the gap between computer
vision research and biological applications.
Read the full abstract ▾
All authors Alex Hoi Hang Chan, Neha Singhal, Onur Kocahan, Andrea Meltzer, Saverio Lubrano, Miyako H. Warrington, Michel Griesser, Fumihiro Kano, Hemal Naik
Computer Vision
Challenges in Hyperspectral Imaging for Autonomous Driving: The HSI-Drive Case
Koldo Basterretxea, Jon Gutiérrez-Zaballa and Javier Echanobe
The use of hyperspectral imaging (HSI) in autonomous driving (AD), while promising, faces many challenges related to the specifics and requirements of this application domain. On the one hand, non-controlled and variable lighting conditions, the wide depth-of-field ranges, and dynamic scenes with fast-moving objects. On the other hand, the requirements for real-time operation and the limited computational resources of embedded platforms. The combination of these factors determines both the criteria for selecting appropriate HSI technologies and the development of custom
vision algorithms that leverage the spectral and spatial information obtained from the sensors. In this article, we analyse several techniques explored in the research of HSI-based
vision systems with application to AD, using as an example results obtained from experiments using data from the most recent version of the HSI-Drive
dataset.
Read the full abstract ▾