Computer Vision · Cross-listed from cs.LG
Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learning
Alberto Tamajo, Srinandan Dasmahapatra and Rahman Attar
Neural networks suffer from catastrophic forgetting in class-incremental learning (CIL) settings. Rehearsal$\unicode{x2013}$replaying a subset of past samples$\unicode{x2013}$is a well-established mitigation strategy. However, recent results suggest that, despite balanced rehearsal allocation, some classes are forgotten substantially more than others. Despite its relevance, this imbalanced forgetting phenomenon remains underexplored. This work shows that imbalanced forgetting arises systematically and severely in rehearsal-based CIL and investigates it extensively. Specifically, we construct, from a principled analysis, three last-layer coefficients that capture different gradient-level sources of interference affecting each past class during an incremental step. We then demonstrate that, together, they reliably predict how past classes will rank in terms of forgetting at the end of that step. While predictive performance alone does not establish causality, these results support the interpretation of the coefficients as a plausible mechanistic account linking last-layer gradient-level interactions during training to class-level forgetting outcomes. Notably, one coefficient$\unicode{x2013}$capturing self-induced interference$\unicode{x2013}$emerges as the strongest predictor, with controlled experiments providing evidence consistent with this coefficient being influenced by the new-class interference coefficient. Overall, our findings provide valuable insights and suggest promising directions for mitigating imbalanced forgetting by reducing class-wise disparities in the identified sources of interference.
Read the full abstract ▾
Computer Vision
MonoPRIO: Adaptive Prior Conditioning for Unified Monocular 3D Object Detection
Leon Davies, Qinggang Meng, Mohamad Saada et al. (+2 more)
Monocular 3D object detection remains challenging because size and depth are underdetermined by single-view evidence, particularly under occlusion, truncation, and scale-depth ambiguity. Although recent methods improve depth and
geometric reasoning, metric size remains unstable in unified multi-class settings, where class variability and limited visibility broaden plausible size modes. We propose MonoPRIO, a unified
monocular detector that targets this bottleneck through adaptive prior conditioning in the size pathway. MonoPRIO constructs class-aware size prototypes offline, routes each decoder query to a soft mixture prior, applies uncertainty-aware log-space conditioning, and uses Cluster-Aligned Prior (CAP) regularisation on matched positives. On the KITTI test server, among the compared methods reporting all three categories, MonoPRIO achieves the highest Moderate AP for Car, Pedestrian, and Cyclist. In the car-only setting, it obtains the best 3D bounding-box AP across Easy, Moderate, and Hard among the compared methods without additional depth or
LiDAR supervision, while requiring lower latency than MonoCLUE under matched measurement conditions. The frozen CLIP RN50 encoder is executed only during offline prior-bank construction and not during detector training or inference. Ablations show that simple prior baselines do not recover the full-model gain and that learned soft routing outperforms uniform and hard routing over the same visual-size bank. Routed injection provides the main improvement; CAP is complementary. MonoPRIO improves over MonoDGP on the nuScenes frontal protocol. The clearest benefits occur in low-data and partially occluded regimes, while atypical geometry or severe visibility loss can cause prior mismatch. Code, trained models, result logs, and reproducibility material are available at https://github.com/leon-davies/MonoPRIO.
Read the full abstract ▾
All authors Leon Davies, Qinggang Meng, Mohamad Saada, Baihua Li, Simon Sølvsten
Computer Vision
BioHuman: Learning Biomechanical Human Representations from Video
Yujun Huo, He Zhang, Chentao Song et al. (+3 more)
Understanding human motion beyond surface kinematics is crucial for motion analysis, rehabilitation, and injury risk assessment. However, progress in this domain is limited by the lack of large-scale
datasets with biomechanical annotations, and by existing approaches that cannot directly infer internal biomechanical states from visual observations. In this paper, we introduce a simulation-based framework for estimating muscle activations from existing motion capture
datasets, resulting in BioHuman10M, a large-scale
dataset with synchronized
video, motion, and activations. Building on BioHuman10M, we propose BioHuman, an end-to-end model that takes
monocular video as input and jointly predicts human motion and muscle activations, effectively bridging visual observations and internal biomechanical states. Extensive experiments demonstrate that BioHuman enables accurate reconstruction of both kinematic motion and muscle activity, and generalizes across diverse subjects and motions. We believe our approach establishes a new
benchmark for
video-based biomechanical understanding and opens up new possibilities for physically grounded human modeling.
Read the full abstract ▾
All authors Yujun Huo, He Zhang, Chentao Song, Honglin Song, Zongyu Zuo, Tao Yu
Computer Vision · Cross-listed from cs.CL
Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining
Weimin Xiong, Shuhao Gu, Bowen Ye et al. (+5 more)
Recent advances in
multimodal large
language models have driven growing interest in graphical user interface (GUI)
agents, yet their generalization remains constrained by the scarcity of large-scale training data spanning diverse real-world applications. Existing
datasets rely heavily on costly manual annotations and are typically confined to narrow domains. To address this challenge, we propose
Video2GUI, a fully automated framework that extracts grounded GUI interaction trajectories directly from unlabeled Internet
videos.
Video2GUI employs a coarse-to-fine filtering strategy to identify high-quality GUI tutorial
videos and convert them into structured
agent trajectories. Applying this pipeline to 500 million
video metadata entries, we construct WildGUI, a large-scale
dataset containing 12 million interaction trajectories spanning over 1,500 applications and websites. Pre-training Qwen2.5-VL and
Mimo-VL on WildGUI yields consistent improvements of 5-20% across multiple GUI grounding and action
benchmarks, matching or surpassing state-of-the-art performance. We will release both the WildGUI
dataset and the
Video2GUI pipeline to support future research of GUI
agents.
Read the full abstract ▾
All authors Weimin Xiong, Shuhao Gu, Bowen Ye, Zihao Yue, Lei Li, Feifan Song, Sujian Li, Hao Tian
Computer Vision
EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding
Yuejiao Su, Xinshen Zhang, Zhen Ye et al. (+3 more)
Understanding human–environment interactions from
egocentric vision is essential for assistive robotics and
embodied intelligent
agents, yet existing
multimodal large
language models (MLLMs) still struggle with accurate interaction
reasoning and fine-grained pixel grounding. To this end, this paper introduces EARL, an
Egocentric Analysis-guided Reinforcement Learning framework that explicitly transfers coarse interaction semantics to query-oriented answering and grounding. Specifically, EARL adopts a two-stage parsing framework including coarse-grained interpretation and fine-grained response. The first stage holistically interprets
egocentric interactions and generates a structured textual description. The second stage produces the textual answer and pixel-level mask in response to the user query. To bridge the two stages, we extract a global interaction descriptor as a semantic prior, which is integrated via a novel Analysis-guided Feature Synthesizer (AFS) for query-oriented
reasoning. To optimize heterogeneous outputs, including textual answers, bounding boxes, and grounding masks, we design a multi-faceted reward function and train the response stage with GRPO. Experiments on Ego-IRGBench show that EARL achieves 65.48% cIoU for pixel grounding, outperforming previous RL-based methods by 8.37%, while OOD grounding results on EgoHOS indicate strong transferability to unseen
egocentric grounding scenarios.
Read the full abstract ▾
All authors Yuejiao Su, Xinshen Zhang, Zhen Ye, Lei Yao, Lap-Pui Chau, Yi Wang
Computer Vision
Ruixu Zhang, Deyi Ji, Lanyun Zhu et al. (+4 more)
Self-
evolution offers a promising path for improving
reasoning models without relying on intensive human annotation. However, extending this paradigm to
video understanding remains underexplored and challenging:
videos are long, dynamic, and redundant, while the evidence needed for
reasoning is often sparse and temporally localized. Naively generating difficult question-answer pairs from full
videos can therefore produce supervision that appears challenging but is weakly grounded, relying on static cues or
language priors rather than temporal evidence. In this work, we argue that the key bottleneck of
video self-
evolution is not difficulty alone, but grounding. We propose
Video-Zero, an annotation-free Questioner–Solver co-
evolution framework that centers self-
evolution on temporally localized evidence. The Questioner discovers informative evidence segments and generates evidence-grounded questions, while the Solver learns to answer and align its predictions with the supporting evidence. This closes an iterative loop of evidence discovery, grounded supervision, and evidence-aligned learning. Across 13
benchmarks spanning temporal grounding, long-
video understanding, and
video reasoning,
Video-Zero consistently improves multiple
video VLM backbones, demonstrating the effectiveness and transferability of evidence-centered self-
evolution.
Read the full abstract ▾
All authors Ruixu Zhang, Deyi Ji, Lanyun Zhu, Xuanyi Liu, Yuxin Meng, Ruihang Chu, Yujiu Yang
Computer Vision · Cross-listed from cs.GR
UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars
Xiaoyu Zhan, Xinyu Fu, Chenghao Yang et al. (+9 more)
Speech-driven gestures and facial animations are fundamental to expressive digital avatars in games, virtual production, and interactive media. However, existing methods are either limited to a single modality for audio motion
alignment, failing to fully utilize the potential of massive human motion data, or are constrained by the representation ability and throughput of
multimodal models, which makes it difficult to achieve high-quality motion generation or real-time performance. We present UMo, a unified sparse motion modeling architecture for real-time co-
speech avatars, which processes text, audio, and motion
tokens within a unified formulation. Leveraging a spatially sparse Mixture-of-Experts framework and a temporally sparse, keyframe-centric design, UMo efficiently performs real-time dense reconstruction, enabling temporally coherent and high-fidelity animation generation for both facial expressions and gestures. Furthermore, we implement a multi-stage training strategy with targeted audio augmentation to enhance acoustic diversity and semantic consistency. Consequently, UMo preserves fine-grained
speech-motion
alignment even under strict latency constraints. Extensive quantitative and qualitative evaluations show that UMo achieves better output quality under low latency and real-time performance constraints, offering a practical solution for high-fidelity real-time co-
speech avatars.
Read the full abstract ▾
All authors Xiaoyu Zhan, Xinyu Fu, Chenghao Yang, Xiaohong Zhang, Dongjie Fu, Pengcheng Fang, Tengjiao Sun, Xiaohao Cai, Hansung Kim, Yuanqi Li, Jie Guo, Yanwen Guo
Computer Vision
CHASM: Cross-frequency Harmonized Axis-Separable Mixing for Spectral Token Operators
Pengcheng Fang, Hongli Chen, Yuxia Chen et al. (+3 more)
Spectral
token mixers based on
Fourier transforms provide an efficient way to model global interactions in visual feature maps. Existing designs often either apply filter-wise spectral responses along fixed channel axes, or learn adaptive frequency-indexed channel mixing without explicitly aligning the channel directions used across frequencies. We propose CHASM, a Cross-frequency Harmonized Axis-Separable Mixer, as a structured middle ground. CHASM separates what should be shared from what should remain frequency-specific: all frequencies share a learned channel eigenbasis, while each frequency retains its own positive spectral gains. The shared basis makes channel directions comparable across the spectrum, whereas the positive gains preserve local spectral adaptivity. CHASM applies this structured operator separably along the height and width axes and is used as a drop-in replacement mixer inside existing backbones. We provide a structural characterization of the shared-basis operator family and evaluate CHASM through controlled same-backbone comparisons. Across accelerated MRI reconstruction, undersampled MRI
segmentation, and natural-image reconstruction, CHASM consistently improves over same-backbone spectral-mixer baselines. Ablations show that removing the shared-basis constraint weakens performance, and randomizing coherent sampling geometry substantially reduces the gain, supporting cross-frequency harmonization as a useful inductive bias for spectral
token operators.
Read the full abstract ▾
All authors Pengcheng Fang, Hongli Chen, Yuxia Chen, Tengjiao Sun, Jiaxin Liu, Xiaohao Cai
Computer Vision
Towards Label-Free Single-Cell Phenotyping Using Multi-Task Learning
Saqib Nazir and Ardhendu Behera
Label-free single-
cell imaging offers a scalable, non-invasive alternative to fluorescence-based cytometry, yet inferring molecular phenotypes directly from bright-field morphology remains challenging. We present a unified Deep Learning (DL) framework that jointly performs White Blood
Cell (WBC) classification and continuous
protein-expression regression from label-free Differential Phase Contrast (DPC) images. Our model employs a Hybrid architecture that fuses convolutional fine-grained texture features with
transformer-based global representations through a learnable cross-branch gating module, enabling robust morpho-molecular inference from DPC images. To support downstream interpretability, we further incorporate a Large
Language Model (LLM) that generates concise, biologically grounded summaries of the predicted
cell states. Experiments on the Berkeley Single
Cell Computational Microscopy (BSCCM) and Blood
Cells Image
benchmarks demonstrate strong performance, achieving a 91.3% WBC classification accuracy and a 0.72 Pearson correlation for CD16 expression regression on BSCCM. These results underscore the promise of label-free single-
cell imaging for cost-effective hematological profiling, enabling simultaneous phenotype identification and quantitative biomarker estimation without fluorescent staining. The source code is available at https://github.com/saqibnaziir/Single-
Cell-Phenotyping.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.GR
SCAMP: Sparse-anchor Control is One Small Projection
Pengcheng Fang, Tengjiao Sun, Xiaoyu Zhan et al. (+4 more)
Authoring with a text-to-motion generator needs sparse anchors: chosen joints, at chosen frames, at given positions. Meeting them currently costs a conditioning branch trained for the task, or hundreds of per-clip optimisation steps in the architecture's native variables. In any generator that decodes a continuous state through a frozen differentiable decoder, the anchors ask for little and leave most of the state free: a few hundred numbers against a state of tens of thousands. Every control method is a choice among the states that satisfy them, and the choices differ along the directions the anchors cannot see and the motion can. SCAMP makes the choice that moves none of them: damped Gauss-Newton in the space of the anchors, through the frozen decoder alone, training-free, with one dimensionless damping constant. Every increment is a combination of the rows of the anchors' Jacobian, so the correction is orthogonal to everything the anchors never see, and the system solved is the size of the request rather than of the state. Applied unchanged to seven published generators spanning
diffusion,
token and latent designs, it matches or exceeds in anchor error every released control method it is measured against, and closes the anchors on hosts that ship none. Confined to those rows, a correction can only take the shapes the decoder admits, so what it costs belongs to the decoder, and holding the solver fixed makes that cost measurable: it divides by decoder family, windowed decoders staying within a small multiple of the unconstrained generator's foot skating where analytic recoveries multiply it several times over. Built to that criterion, our own generator reaches 0.083 m anchor error at FID 0.102 in 0.50 s per clip. The decoder's temporal support is a design criterion for controllable motion generation.
Read the full abstract ▾
All authors Pengcheng Fang, Tengjiao Sun, Xiaoyu Zhan, Yanwen Guo, Hansung Kim, Xiaohao Cai, Dongjie Fu
Computer Vision · Cross-listed from cs.RO
IntentVLA: Short-Horizon Intent Modeling for Aliased Robot Manipulation
Shijie Lian, Bin Yu, Xiaopeng Lin et al. (+8 more)
Robot imitation data are often
multimodal: similar visual-
language observations may be followed by different action chunks because human demonstrators act with different short-horizon intents, task phases, or recent context. Existing frame-conditioned VLA policies infer each chunk from the current observation and instruction alone, so under partial observability they may resample different intents across adjacent replanning steps, leading to inter-chunk conflict and unstable execution. We introduce IntentVLA, a history-conditioned VLA framework that encodes recent visual observations into a compact short-horizon intent representation and uses it to condition chunk generation. We further introduce AliasBench, a 12-task ambiguity-aware
benchmark on RoboTwin2 with matched training data and evaluation environments that isolate short-horizon observation aliasing. Across AliasBench, SimplerEnv, LIBERO, and RoboCasa, IntentVLA improves rollout stability and outperforms strong VLA baselines
Read the full abstract ▾
All authors Shijie Lian, Bin Yu, Xiaopeng Lin, Zhaolong Shen, Laurence Tianruo Yang, Yurun Jin, Haishan Liu, Changti Wu, Hang Yuan, Cong Huang, Kai Chen
Computer Vision
Vision-Core Guided Contrastive Learning for Balanced Multi-modal Prognosis Prediction of Stroke
Liren Chen, Lidong Sun, Mingyan Huang et al. (+5 more)
Deep learning and multi-modal fusion have demonstrated transformative potential in medical diagnosis by integrating diverse data sources. However, accurate prognosis for ischemic stroke remains challenging due to limitations in existing multi-modal approaches. First, current methods are predominantly confined to dual-modal fusion, lacking a framework that effectively integrates the trifecta of medical images, structured clinical data, and unstructured text. Second, they often fail to establish deep bidirectional interactions between modalities; To address these critical gaps, this paper proposes a novel tri-modal fusion model for ischemic stroke prognosis. Our approach first enriches the data representation by employing a Large
Language Model (LLM) to automatically generate semi-structured diagnostic text from
brain MRIs. This process not only addresses the scarcity of expert annotations but also serves as a regularized semantic enhancement, improving
multimodal fusion robustness. Furthermore, we design a core component termed the
Vision-Conditioned Dual
Alignment Fusion Module (VDAFM), which strategically uses visual features as a conditional prior to guide fine-grained interaction with the generated text. This module achieves a dynamic and profound fusion through a dual semantic
alignment loss, effectively mitigating modal heterogeneity. Extensive experiments on a real-world clinical
dataset demonstrate that our model achieves state-of-the-art performance.
Read the full abstract ▾
All authors Liren Chen, Lidong Sun, Mingyan Huang, Junzhe Tang, Yinghui Zhu, Guanjie Wang, Yiqing Xia, Ting Xiao
Computer Vision
Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners
Qingyang Liu, Bingjie Gao, Canmiao Fu et al. (+9 more)
Recent unified models integrate
multimodal understanding and generation within a single framework. However, an "understanding-generation gap" persists, where models can capture user intent but often fail to translate this semantic knowledge into precise pixel-level manipulation. This gap results in two bottlenecks in anything-to-image task (X2I): the
attention entanglement bottleneck, where blind planning struggles with complex
prompts, and the visual refinement bottleneck, where unstructured feedback fails to correct imperfections efficiently. In this paper, we propose a novel framework that empowers unified models to autonomously switch between generation strategies based on instruction complexity and model capability. To achieve this, we construct a hierarchical data pipeline that constructs execution paths across three adaptive modes: direct generation for simple cases, self-reflection for quality refinement, and multi-step planning for decomposing complex scenarios. Building on this pipeline, we contribute a high-quality
dataset with over 50,000 samples and implement a two-stage training strategy comprising SFT and RL. Specifically, we design step-wise
reasoning rewards to ensure logical consistency and intra-group complexity penalty to prevent redundant computational overhead. Extensive experiments demonstrate that our method outperforms existing baselines on X2I, achieving superior generation fidelity among simple-to-complex instructions. The code is released at https://github.com/WeChatCV/Interleaved_Visual_Reasoner.
Read the full abstract ▾
All authors Qingyang Liu, Bingjie Gao, Canmiao Fu, Zhipeng Huang, Chen Li, Feng Wang, Shuochen Chang, Shaobo Wang, Yali Wang, Keming Ye, Jiangtong Li, Li Niu
Computer Vision
StyleTextGen: Style-Conditioned Multilingual Scene Text Generation
Zeyu Chen, Fangmin Zhao, Yan Shu et al. (+3 more)
Style-conditioned scene text generation faces unique challenges in extracting precise text styles from complex backgrounds and maintaining fine-grained style consistency across characters, especially for
multilingual scripts. We propose StyleTextGen, a novel framework that learns to perceive and replicate visual text styles across different
languages and writing systems. Our approach features three key contributions: First, we introduce a dual-branch style encoder dedicated to style modeling, yielding robust
multilingual text style representations in complex real-world scenes. Second, we design a text style consistency loss that enhances style coherence and improves overall visual quality. Third, we develop a mask-guided inference strategy that ensures precise style
alignment between generated and reference text. To facilitate systematic evaluation, we construct StyleText-CE, a bilingual scene text style
benchmark covering both monolingual and cross-lingual settings. Extensive experiments demonstrate that StyleTextGen significantly outperforms existing methods in style consistency and cross-lingual generalization, establishing new state-of-the-art performance in
multilingual style-conditioned text generation.
Read the full abstract ▾
All authors Zeyu Chen, Fangmin Zhao, Yan Shu, Yichao Liu, Liu Yu, Yu Zhou
Computer Vision
Towards Continuous Sign Language Conversation from Isolated Signs
Youngmin Kim, Kyobin Choo, Jiwoo Park et al. (+4 more)
Sign
language is the primary
language for many Deaf and Hard-of-Hearing (DHH) signers, yet most conversational AI systems still mediate interaction through spoken or written
language. This spoken-
language-centered interface can limit access for signers for whom spoken or written
language is not the most accessible medium, motivating direct sign-to-sign conversational modeling. However, sentence-level sign
video data are expensive to collect and annotate, leaving existing sign translation and production models with limited vocabulary coverage and weak open-domain generalization. We address this bottleneck by constructing continuous sign conversations from isolated signs: large-scale labeled isolated clips are collected as lexically grounded motion primitives and recomposed into sign-
language-ordered utterances derived from existing dialogue corpora. We introduce SignaVox-W, which provides, to our knowledge, the largest labeled isolated-sign vocabulary to date, and SignaVox-U, a continuous 3D sign conversation
dataset built from SignaVox-W. To bridge structural mismatch between spoken and signed
languages, we use a
retrieval-guided spoken-to-gloss translator; to bridge independently collected isolated clips, we propose BRAID, a
diffusion Transformer that performs duration
alignment and co-articulatory boundary inpainting. With the resulting data, we train SignaVox, a direct sign-to-sign conversational model that generates 3D body, hand, and facial motion responses from prior signing context without spoken-
language text or externally provided glosses at inference time. Quantitative and qualitative evaluations show improved isolated-to-continuous motion quality, stronger response-level semantic
alignment, and scalable signer-centered interaction that better supports visual-spatial articulation.
Read the full abstract ▾
All authors Youngmin Kim, Kyobin Choo, Jiwoo Park, Minseo Kim, Chanyoung Kim, Junhyeok Kim, Seong Jae Hwang
Computer Vision
SceneFunRI: Reasoning the Invisible for Task-Driven Functional Object Localization
Posheng Chen, Powen Cheng, Gueter Josmy Faure et al. (+2 more)
In real-world scenes, target objects may reside in regions that are not visible. While humans can often infer the locations of occluded objects from context and commonsense knowledge, this capability remains a major challenge for
vision-
language models (VLMs). To address this gap, we introduce SceneFunRI, a
benchmark for
Reasoning the Invisible. Based on the SceneFun3D
dataset, SceneFunRI formulates the task as a 2D spatial
reasoning problem via a semi-automatic pipeline and comprises 855 instances. It requires models to infer the locations of invisible functional objects from task instructions and commonsense
reasoning. The strongest baseline model (Gemini 3 Flash) only achieves an CAcc@75 of 15.20, an mIoU of 0.74, and a Dist of 28.65. We group our prompting analysis into three categories: Strong Instruction Prompting,
Reasoning-based Prompting, and Spatial Process of Elimination (SPoE). These findings indicate that invisible-region
reasoning remains an unstable capability in current VLMs, motivating future work on models that more tightly integrate task intent, commonsense priors, spatial grounding, and uncertainty-aware search.
Read the full abstract ▾
All authors Posheng Chen, Powen Cheng, Gueter Josmy Faure, Hung-Ting Su, Winston H. Hsu
Computer Vision
Generating HDR Video from SDR Video
SaiKiran Tedla, Francesco Banterle, Trevor Canham et al. (+6 more)
The high dynamic range (HDR)
video ecosystem is approaching maturity, but the problem of upconverting legacy standard dynamic range (SDR)
videos persists without a convincing solution. We propose a framework for HDR
video synthesis from casual SDR footage by leveraging large-scale generative
video models. We introduce a Multi-Exposure
Video Model (MEVM) that can predict exposure-bracketed linear SDR
video sequences from a single nonlinear SDR
video input. We further propose a learnable
Video Merging Model (VMM) that merges the predicted exposure-bracketed
video into a high-quality HDR sequence while preserving detail in both shadows and highlights. Extensive experiments, quantitative and qualitative evaluation, and a user study demonstrate that our approach enables robust HDR conversion for in-the-wild examples from casual consumer
videos and even iconic films. Finally, our model can support HDR synthesis pipelines built upon existing SDR generative
video models. Output HDR
videos can be viewed on our supplementary webpage: sdr2hdrvideo.github.io
Read the full abstract ▾
All authors SaiKiran Tedla, Francesco Banterle, Trevor Canham, Karanpreet Raja, David B. Lindell, Kiriakos N. Kutulakos, Jiacheng Li, Feiran Li, Daisuke Iso
Computer Vision
EponaV2: Driving World Model with Comprehensive Future Reasoning
Jiawei Xu, Zhizhou Zhong, Zhijian Shu et al. (+8 more)
Data scaling plays a pivotal role in the pursuit of general intelligence. However, the prevailing perception-planning paradigm in autonomous driving relies heavily on expensive manual annotations to supervise trajectory planning, which severely limits its scalability. Conversely, although existing perception-free driving world models achieve impressive driving performance, their real-world
reasoning ability for planning is solely built on next frame image
forecasting. Due to the lack of enough supervision, these models often struggle with comprehensive scene understanding, resulting in unsatisfactory trajectory planning. In this paper, we propose EponaV2, a novel paradigm of driving world models, which achieves high-quality planning with comprehensive future
reasoning. Inspired by how human drivers anticipate 3D geometry and semantics, we train our model to forecast more comprehensive future representations, which can be additionally decoded to future geometry and semantic maps. Extracting the 3D and semantic modalities enables our model to deeply understand the surrounding environment, and the future prediction task significantly enhances the real-world
reasoning capabilities of EponaV2, ultimately leading to improved trajectory planning. Moreover, inspired by the training recipe of Large
Language Models (LLMs), we introduce a flow matching group relative policy optimization mechanism to further improve planning accuracy. The state-of-the-art (SOTA) performances of EponaV2 among perception-free models on three NAVSIM
benchmarks (+1.3PDMS, +5.5EPDMS) demonstrate the effectiveness of our methods.
Read the full abstract ▾
All authors Jiawei Xu, Zhizhou Zhong, Zhijian Shu, Mingkai Jia, Mingxiao Li, Jia-Wang Bian, Qian Zhang, Kaicheng Zhang, Jin Xie, Jian Yang, Wei Yin
Computer Vision
Are Candidate Models Really Needed for Active Learning?
Harshini Mridula Mohan, Maanya Manjunath, Vipul Arya et al. (+2 more)
Deep learning has profoundly impacted domains such as computer
vision and natural
language processing by uncovering complex patterns in vast
datasets. However, the reliance on extensive labeled data poses significant challenges, including resource constraints and annotation errors, particularly in training Convolutional Neural Networks (CNNs) and
transformers due to a larger number of parameters. Active learning offers a promising solution to reduce labeling burdens by strategically selecting the most informative samples for annotation. However, the current active learning frameworks are time-intensive which select the samples iteratively with the help of initial candidate models. This study investigates the feasibility of using CNNs and
transformers with randomly initialized weights, eliminating the need for initial candidate models while achieving results comparable to active learning frameworks that depend on such candidate models. We evaluate three confidence-based sampling strategies: high confidence (HC), low confidence (LC), and a combination of high confidence in the early stages of training and low confidence at later stages of training (HCLC). Among these, mostly LC demonstrated the best performance in our experiments, showcasing its effectiveness as an active learning strategy without the need for candidate models. Further, extensive experiments verify the robustness of the proposed active learning methods. By challenging traditional frameworks, the proposed work introduces a streamlined approach to active learning, advancing efficiency and flexibility across diverse
datasets and domains.
Read the full abstract ▾
All authors Harshini Mridula Mohan, Maanya Manjunath, Vipul Arya, S.H. Shabbeer Basha, Nitin Cheekatla
Computer Vision
MiVE: Multiscale Vision-language features for reference-guided video Editing
Tong Wang, Meng Zou, Chengjing Wu et al. (+4 more)
Reference-guided
video editing takes a source
video, a text instruction, and a reference image as inputs, requiring the model to faithfully apply the instructed edits while preserving original motion and unedited content. Existing methods fall into two paradigms, each with inherent limitations: decoupled encoders suffer from modality gaps when processing instructions and visual content independently, while unified
vision-
language encoders lose fine-grained spatial details by relying solely on final-layer representations. We observe that VLM layers encode complementary information hierarchically – early layers capture localized spatial details essential for precise editing, while deeper layers encode global semantics for instruction comprehension. Building on this insight, we present MiVE (Multiscale
Vision-
language features for reference-guided
video Editing), a framework that repurposes VLMs as multiscale feature extractors. MiVE extracts hierarchical features from Qwen3-VL and integrates them into a unified self-
attention Diffusion Transformer, eliminating the modality mismatch inherent in cross-
attention designs. Experiments demonstrate that MiVE achieves state-of-the-art performance by ranking highest in human preference, outperforming both academic methods and commercial systems.
Read the full abstract ▾
All authors Tong Wang, Meng Zou, Chengjing Wu, Xiaochao Qu, Luoqi Liu, Xiaolin Hu, Ting Liu
Computer Vision
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Tan Pan, Shuhao Mei, Yixuan Sun et al. (+8 more)
Self-supervised pre-training methods in medical imaging typically treat each individual as an isolated instance, learning representations through augmentation-based objectives or masked reconstruction. They often do not adequately capitalize on a key characteristic of physiological features: anatomical structures maintain consistent spatial relationships across individuals (instances), such as the thalamus being medial to the basal ganglia, regardless of variations in
brain size, shape, or pathology. We propose leveraging this cross-instance topological consistency as a supervisory signal. The challenge arises from the inherent variability in medical imaging, which can differ significantly across instances and modalities. To tackle this, we focus on two
alignment regimes. (i) Intra-instance: with pixel-level correspondences available, a cross-modal triplet objective explicitly preserves local neighborhood
topology. (ii) Inter-instance: without such supervision, we derive pseudo-correspondences to control partial neighborhood
alignment and prevent
topology collapse across modalities. We validate our approach across 7 downstream multi-modal tasks, achieving average improvements of 1.1% and 5.94% in
segmentation and classification tasks, respectively, and demonstrating significantly better robustness when modalities are missing at test time.
Read the full abstract ▾
All authors Tan Pan, Shuhao Mei, Yixuan Sun, Kaiyu Guo, Chen Jiang, Zhaorui Tan, Mengzhu Li, Limei Han, Xiang Zou, Yuan Cheng, Mahsa Baktashmotlagh
Computer Vision
TERRA-CD: Multi-Temporal Framework for Multi-class and Semantic Change Detection
Omkar Oak, Rukmini Nazre, Rujuta Budke et al. (+1 more)
Urban vegetation monitoring plays a vital role in understanding environmental changes, yet comprehensive
datasets for this purpose remain limited. To address this gap, we present the Temporal Remote-sensing Repository for Analyzing Change Detection (TERRA-CD), a
benchmark dataset comprising 5,221 Sentinel-2 image pairs from 2019 and 2024, covering 232 cities across the USA and Europe. The
dataset features three distinct annotation schemes: 4-class land cover mapping masks, 3-class vegetation change masks, and 13-class semantic change masks capturing all possible land cover transitions. Using various deep learning approaches including Siamese networks, STANet variants, Bi-SRNet, Changemask, Post-Classification Comparison, and HRSCD strategies, we evaluated the
dataset's effectiveness for both vegetation Multi-class Change Detection as well as Semantic Change Detection. The proposed
dataset and methods are available at https://github.com/omkarsoak/TERRA-CD.
Read the full abstract ▾
All authors Omkar Oak, Rukmini Nazre, Rujuta Budke, Suraj Sawant
Computer Vision
Vision-Based Water Level and Flow Estimation
ZhiXin Sun
With the rapid
evolution of computer
vision,
vision-based methodologies for water level and river surface velocity estimation have reached significant maturity. Compared to traditional sensing, these techniques offer superior interpretability, automated data archiving, and enhanced system robustness. However, challenges such as environmental sensitivity, limited precision, and complex site calibration persist. This work proposes an integrated framework that synergizes state-of-the-art (SOTA)
vision models with statistical modeling. By leveraging physical priors and robust filtering strategies, we improve the accuracy of water level detection and flow estimation. Code will be available at https://github.com/sunzx97/
Vision_Based_Water_Level_and_Flow_Estimation.git
Read the full abstract ▾
Computer Vision
How to Evaluate and Refine your CAM
Luca Domeniconi, Alessandra Stramiglio, Michele Lombardi et al. (+1 more)
Class attribution maps (CAMs) provide local explanations for the decisions of convolutional neural networks. While widely used in practice, the evaluation of CAMs remains challenging due to the lack of ground-truth explanations, making it difficult to evaluate the soundness of existing metrics. Independently, most commonly used CAM methods produce low-resolution attribution maps, which limits their usefulness for detailed interpretability. To address the evaluation challenge, we introduce a synthetic
dataset with ground-truth attributions that enables a rigorous comparison of CAM evaluation metrics. Using this
dataset, we analyze existing metrics and propose ARCC, a new composite metric that more reliably identifies faithful explanations. To address the low resolution issue, we introduce RefineCAM, a method that produces high-resolution attribution maps by aggregating CAMs across multiple network layers. Our results show that RefineCAM consistently outperforms existing methods according to the proposed evaluation.
Read the full abstract ▾
All authors Luca Domeniconi, Alessandra Stramiglio, Michele Lombardi, Samuele Salti
Computer Vision
MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models
Tianwei Chen, Takuya Furusawa, Yuki Hirakawa et al. (+3 more)
This paper introduces a multi-label visual
emotion analysis
benchmark dataset for comprehensively evaluating the ability of
multimodal large
language models (MLLMs) to predict the
emotions evoked by images. Recent user studies report an unintuitive finding: humans may prefer the predictions of MLLMs over the labels in existing
datasets. We argue that this phenomenon stems from the suboptimal annotation scheme used in existing
datasets, where each annotator is shown a single candidate
emotion for each image and judges whether it is evoked or not. This approach is clearly limited because a single image can evoke multiple
emotions with varying intensities. As a result, evaluations based on these
datasets may underestimate the capabilities of MLLMs, yet an appropriate
benchmark for evaluating such models remains lacking. To address this issue, we introduce a new multi-label
benchmark dataset for visual
emotion analysis toward MLLMs evaluation. We hire $20$ annotators per image and ask them to select all
emotions they feel from an image. Then, we aggregate the votes across all annotators, providing a more reliable and representative
dataset labeled with a distribution of
emotions. The resulting
dataset contains $10,344$ images with $236,998$ valid votes across eight
emotions. Based on this
benchmark dataset, we evaluate several recent models, including Qwen3-VL, OpenAI's GPT, Gemini, and Claude. We assess model performance on both dominant
emotion prediction and
emotion distribution prediction. Our results demonstrate the progress achieved by recent MLLMs while also indicating that substantial room for improvement remains. Furthermore, our experiments with LLM-as-a-judge show that the method does not consistently improve MLLMs' performance, indicating its limitations for the subjective task of visual
emotion analysis.
Read the full abstract ▾
All authors Tianwei Chen, Takuya Furusawa, Yuki Hirakawa, Ryotaro Shimizu, Mo Fan, Takashi Wada
Computer Vision · Cross-listed from cs.LG
Action-Inspired Generative Models
Eshwar R. A. and Debnath Pal
We introduce Action-Inspired Generative Models (AGMs), a dual-network generative framework motivated by the observation that existing bridge-matching methods assign uniform regression weight to every
stochastic transition in the transport landscape, regardless of whether a given bridge sample lies along a structurally coherent trajectory or a degenerate one. We address this by introducing a lightweight learned scalar potential $V_\phi$ that scores bridge samples online and modulates the drift objective via importance weights derived through a stop-gradient barrier – preventing
adversarial feedback between the two networks whilst preserving $V_\phi$'s guiding signal. Crucially, $V_\phi$ comprises only $\sim$1.4% of the primary drift network's parameter count, adds no overhead to the inference graph, and requires no iterative half-bridge fitting or auxiliary
stochastic differential equation (SDE) solvers: it is a plug-and-play enhancement to any bridge-matching training loop. At inference, $V_\phi$ is discarded entirely, leaving standard Euler-Maruyama integration of the exponential moving average (EMA) drift. We demonstrate that selectively penalising uninformative transport paths through the learned potential yields consistent improvements in generation quality across fidelity and coverage metrics.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Efficient Dense Matching for Enhanced Gaussian Splatting Using AV1 Motion Vectors
Julien Zouein, Vibhoothi Vibhoothi, François Pitié et al. (+1 more)
3D Gaussian Splatting (3DGS) has emerged as a prominent framework for real-time, photorealistic scene reconstruction, offering significant speed-ups over Neural Radiance Fields (NeRF). However, the fidelity of 3DGS representations remains heavily dependent on the quality of the initial point cloud. While standard Structure-from-Motion (SfM) pipelines using COLMAP provide adequate initialisation, they often suffer from high computational costs and sparsity in textureless regions, which degrades subsequent reconstruction accuracy and convergence speed. In this work, we introduce an AV1-based feature detection and matching pipeline that significantly reduces SfM processing overhead. By leveraging motion vectors inherent to the AV1
video codec, we bypass computationally expensive exhaustive matching while maintaining
geometric robustness. Our pipeline produces substantially denser point clouds, with up to eight times as many points as classical SfM. We demonstrate that this enhanced initialisation directly improves 3DGS performance, yielding an 9-point increase in VMAF and a 63% average reduction in training time required to reach baseline quality. The project page: https://sigmedia.tv/AV1-3DGS.github.io/
Read the full abstract ▾
All authors Julien Zouein, Vibhoothi Vibhoothi, François Pitié, Anil Kokaram
Computer Vision
UniTriGen: Unified Triplet Generation of Aligned Visible-Infrared-Label for Few-Shot RGB-T Semantic Segmentation
Ping Zhou, Haoyu Wang, Mengmeng Zheng et al. (+4 more)
RGB-T semantic
segmentation requires strictly aligned VIS-IR-Label triplets; however, such aligned triplet data are often scarce in real-world scenarios. Existing generative augmentation methods usually adopt cascaded generation paradigms, decomposing joint triplet generation into local conditional processes. As a result, consistency among VIS, IR, and Label in spatial structure, semantic content, and cross-modal details cannot be reliably maintained. To address this issue, we propose UniTriGen, a unified triplet generation framework that directly generates spatially aligned, semantically consistent, and modality complementary VIS-IR-Label triplets under the guidance of text
prompts. UniTriGen first introduces a unified triplet generation mechanism, where VIS, IR, and Label are jointly encoded into a shared latent space and modeled with a
diffusion process to enforce global cross-modal consistency. Lightweight modality-specific residual adapters are further integrated into this mechanism to accommodate modality-specific imaging characteristics and output formats. To mitigate generation bias caused by imbalanced scene and class distributions in limited paired triplets, UniTriGen also employs a scene-balanced and class-aware few-shot sampling strategy, which induces a more balanced sampling distribution and enhances the scene and class diversity of generated triplets. Experiments show that UniTriGen generates high-quality aligned triplets from limited real paired data, thereby achieving consistent performance improvements across various RGB-T semantic
segmentation models.
Read the full abstract ▾
All authors Ping Zhou, Haoyu Wang, Mengmeng Zheng, Lei Zhang, Wei Wei, Chen Ding, Fei Zhou
Computer Vision
Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attribution
Tian Qin, Junzhe Chen, Yuqing Shi et al. (+3 more)
Large
vision-
language models (LVLMs) often hallucinate when
language priors dominate weak or ambiguous visual evidence. Existing contrastive decoding methods mitigate this problem by comparing predictions from the original image with those from externally perturbed visual inputs, but such references can introduce off-
manifold artifacts and require costly extra forward passes. We propose SIRA, a training-free internal contrastive decoding framework that constructs a counterfactual reference inside the same LVLM by exploiting the staged information flow of
multimodal transformers. Instead of removing visual information from the input, SIRA first lets image and text
tokens interact through a shared prefix, forming an aligned
multimodal state that preserves
prompt interpretation, decoding history, positional structure, and early visual grounding. It then forks a counterfactual branch in later
transformer layers, where
attention to image-
token positions is masked. This branch retains the shared
multimodal context but lacks continued access to fine-grained visual evidence, yielding a
language-prior-dominated internal reference for
token-level contrast. During decoding, SIRA suppresses
tokens that remain strong without late visual access and favors predictions whose advantage depends on the full visual pathway. Experiments on POPE, CHAIR, and AMBER with Qwen2.5-VL and LLaVA-v1.5 show that SIRA consistently reduces
hallucinations while preserving descriptive coverage and incurring lower overhead than two-pass contrastive decoding. SIRA requires no training, external verifier, or perturbed input, and applies to open-weight LVLMs with white-box inference access.
Read the full abstract ▾
All authors Tian Qin, Junzhe Chen, Yuqing Shi, Tianshu Zhang, Qiang Ju, Lijie Wen
Computer Vision
CalibAnyView: Beyond Single-View Camera Calibration in the Wild
Boying Li, Cheng Zhang, Weirong Chen et al. (+5 more)
Camera calibration is fundamental to reliable
geometric perception, yet classical approaches rely on dedicated targets, successful reconstruction, or dense view coverage, which casually captured imagery rarely satisfies. Recent learning-based single-image methods lift these requirements by exploiting visual cues, and extend calibration beyond intrinsics to
gravity estimation. Yet in the common multi-view case, they predict each view independently and combine the estimates only afterwards, lacking full use of cross-view consistency. We bridge this gap with CalibAnyView, a framework that unifies single- and sparse multi-view calibration by enforcing that consistency inside the network: a
transformer with cross-view
attention predicts camera-model-agnostic perspective fields, followed by a multi-view optimization that fuses them into shared intrinsics and per-view
gravity directions, covering camera models from pinhole to severely distorted lenses. To support this, we construct a large-scale in-the-wild multi-view
video dataset spanning diverse camera models, dynamic scenes, realistic motion trajectories, and heterogeneous lens distortions. Extensive experiments show that CalibAnyView outperforms state-of-the-art methods in both sparse multi-view and single-view settings spanning pinhole to distorted optics.
Read the full abstract ▾
All authors Boying Li, Cheng Zhang, Weirong Chen, Guyuan Chen, Daniel Cremers, Jianfei Cai, Ian Reid, Hamid Rezatofighi