Computer Vision
Selective Prediction Forgetting for Adaptation of Weakly Supervised Localization of Histology Images
Alexis Guichemerre, Banafsheh Karimian, Soufiane Belharbi et al. (+6 more)
Under strong cross-domain shifts, weakly supervised object localization (WSOL) models often lose their capacity to discriminate and develop a strong prediction bias toward a subset of classes. As a result, the majority of target samples are assigned to these dominant classes. Since source-free (unsupervised) domain adaptation (SFDA) methods typically rely on pseudo-labels, inaccurate pseudo-label estimation causes suboptimal adaptation and poor classification and localization performance. This issue is particularly challenging in computational histopathology, where shifts in organ-specific morphology, staining protocols, and acquisition conditions can significantly impact class-discriminative visual cues. This paper introduces SFDA with selective prediction forgetting (SFDA-SPF), a method designed to mitigate excessive concentration of predictions for dominant classes during target-domain adaptation. SFDA-SPF identifies prediction-dominant classes based on the model's target predictions and selectively revises uncertain samples assigned to them. Importantly, this procedure does not require knowledge of the true target class distribution. In contrast with previous SFDA methods, SFDA-SPF introduces a 'forget set' composed of uncertain samples from dominant classes near their decision boundaries, enabling the classification boundaries to shift away from biased predictions. A second 'retain set' is used to anchor certain and accurate samples. To forget the predictions of dominant classes, SFDA-SPF introduces a complementary optimization objective that reduces their probability while allowing alternative class assignments to emerge. This iterative optimization process reassesses target images for forget and retain sets at regular intervals. In addition to improving image-level pseudo-label estimation, the pixel-level classifier is jointly optimized, further enhancing localization.
Read the full abstract ▾
All authors Alexis Guichemerre, Banafsheh Karimian, Soufiane Belharbi, Natacha Gillet, Nicolas Thome, Pourya Shamsolmoali, Mohammadhadi Shateri, Luke McCaffrey, Eric Granger
Computer Vision · Cross-listed from cs.LG
Bases of Steerable Kernels for Equivariant CNNs: From 2D Rotations to the Lorentz Group
Alan Garbarz
We present an alternative way of solving the steerable kernel constraint that appears in the design of steerable equivariant convolutional neural networks. We find explicit real and complex bases which are ready to use, for different
symmetry groups and for feature maps of arbitrary tensor type. A major advantage of this method is that it bypasses the need to numerically or analytically compute Clebsch-Gordan coefficients and works directly with the representations of the input and output feature maps. The strategy is to find a basis of kernels that respect a simpler invariance condition at some point $x_0$, and then steer it with the defining equation of steerability to move to some arbitrary point $x=g\cdot x_0$. This idea has already been mentioned in the literature before, but not advanced in depth and with some generality. Here we describe how it works with minimal technical tools to make it accessible for a general audience.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Unmasking Biases and Reliability Concerns in Convolutional Neural Networks Analysis of Cancer Pathology Images
Michael Okonoda, Eder Martinez, Abhilekha Dalal et al. (+1 more)
Convolutional Neural Networks have shown promising effectiveness in identifying different types of cancer from radiographs. However, the opaque nature of CNNs makes it difficult to fully understand the way they operate, limiting their assessment to empirical evaluation. Here we study the soundness of the standard practices by which CNNs are evaluated for the purpose of cancer pathology. Thirteen highly used cancer
benchmark datasets were analyzed, using four common CNN architectures and different types of cancer, such as melanoma, carcinoma, colorectal cancer, and lung cancer. We compared the accuracy of each model with that of
datasets made of cropped segments from the background of the original images that do not contain clinically relevant content. Because the rendered
datasets contain no clinical information, the null hypothesis is that the CNNs should provide mere chance-based accuracy when classifying these
datasets. The results show that the CNN models provided high accuracy when using the cropped segments, sometimes as high as 93%, even though they lacked biomedical information. These results show that some CNN architectures are more sensitive to bias than others. The analysis shows that the common practices of machine learning evaluation might lead to unreliable results when applied to cancer pathology. These biases are very difficult to identify, and might mislead researchers as they use available
benchmark datasets to test the efficacy of CNN methods.
Read the full abstract ▾
All authors Michael Okonoda, Eder Martinez, Abhilekha Dalal, Lior Shamir
Computer Vision
Revisiting Model Stitching In the Foundation Model Era
Zheda Mai, Ke Zhang, Fu-En Wang et al. (+6 more)
Model stitching, connecting early layers of one model (source) to later layers of another (target) via a light stitch layer, has served as a probe of representational compatibility. Prior work finds that models trained on the same
dataset remain stitchable (negligible accuracy drop) despite different initializations or objectives. We revisit stitching for
Vision Foundation Models (VFMs) that vary in objectives, data, and modality mix (e.g., CLIP, DINOv2, SigLIP 2) and ask: Are heterogeneous VFMs stitchable? We introduce a systematic protocol spanning the stitch points, stitch layer families, training losses, and downstream tasks. Three findings emerge. (1) Stitch layer training matters: conventional approaches that match the intermediate features at the stitch point or optimize the task loss end-to-end struggle to retain accuracy, especially at shallow stitch points. (2) With a simple feature-matching loss at the target model's penultimate layer, heterogeneous VFMs become reliably stitchable across
vision tasks. (3) For deep stitch points, the stitched model can surpass either constituent model at only a small inference overhead (for the stitch layer). Building on these findings, we further propose the VFM Stitch Tree (VST), which shares early layers across VFMs while retaining their later layers, yielding a controllable accuracy-latency trade-off for
multimodal LLMs that often leverage multiple VFMs. Taken together, our study elevates stitching from a diagnostic probe to a practical recipe for integrating complementary VFM strengths and pinpointing where their representations align or diverge.
Read the full abstract ▾
All authors Zheda Mai, Ke Zhang, Fu-En Wang, Zixiao Ken Wang, Albert Y. C. Chen, Lu Xia, Min Sun, Wei-Lun Chao, Cheng-Hao Kuo
Computer Vision
Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validation
Jian Jiang, Chenxi Lin, Yiming Gu et al. (+24 more)
Surgical scene understanding demands not only accurate predictions but also interpretable
reasoning that surgeons can verify against clinical expertise. However, existing
surgical vision-
language models generate predictions without
reasoning chains, and general-purpose
reasoning models fail on compositional
surgical tasks without domain-specific knowledge. We present Surg-R1, a
surgical Vision-
Language Model that addresses this gap through hierarchical
reasoning trained via a four-stage pipeline. Our approach introduces three key contributions: (1) a three-level
reasoning hierarchy decomposing
surgical interpretation into perceptual grounding, relational understanding, and contextual
reasoning; (2) the largest
surgical chain-of-thought
dataset with 320,000
reasoning pairs; and (3) a four-stage training pipeline progressing from supervised fine-tuning to group relative policy optimization and iterative self-improvement. Evaluation on SurgBench, comprising six public
benchmarks and six multi-center external validation
datasets from five institutions, demonstrates that Surg-R1 achieves the highest Arena Score (64.9%) on public
benchmarks versus Gemini 3.0 Pro (46.1%) and GPT-5.1 (37.9%), outperforming both proprietary
reasoning models and specialized
surgical VLMs on the majority of tasks spanning instrument localization, triplet recognition, phase recognition, action recognition, and critical view of safety assessment, with a 15.2 percentage point improvement over the strongest
surgical baseline on external validation.
Read the full abstract ▾
All authors Jian Jiang, Chenxi Lin, Yiming Gu, Zengyi Qin, Zhitao Zeng, Kun Yuan, Yonghao Long, Xiang Xia, Cheng Yuan, Yuqi Wang, Zijie Yue, Kunyi Yang, Yuting Zhang, Zhu Zhuo, Dian Qin, Xin Wang, NG Chi Fai, Brian Anthony, Daguang Xu, Guy Rosman, Ozanan Meireles, Zizhen Zhang, Nicolas Padoy, Hesheng Wang, Qi Dou, Yueming Jin, Yutong Ban
Computer Vision
A Neuro-Symbolic Framework Combining Inductive and Deductive Reasoning for Autonomous Driving Planning
Hongyan Wei and Wael AbdAlmageed
Existing end-to-end autonomous driving models rely heavily on purely data-driven inductive
reasoning. This "black-box" nature limits interpretability and makes safety-critical planning challenging in complex, long-tail scenarios. To overcome this bottleneck, we propose a novel neuro-symbolic trajectory planning (NS-Plan) framework, which integrates rigorous deductive
reasoning into end-to-end neural networks. Specifically, our framework utilizes a Large
Language Model (LLM) to dynamically extract scene rules and employs an Answer Set Programming (ASP) solver for deterministic logical arbitration, generating logically consistent and traceable discrete driving decisions. To bridge the gap between discrete symbols and continuous trajectories, we introduce a decision-conditioned decoding mechanism that transforms high-level logical decisions into learnable
embedding vectors, simultaneously constraining the planning query and the physical initial velocity of a differentiable Kinematic Bicycle Model (KBM). By combining KBM-generated baseline trajectories with neural residual corrections, our approach uses the KBM to guide trajectory generation while keeping the decision-making process traceable. On the nuScenes
benchmark, our method (NS-Plan) comprehensively outperforms state-of-the-art baselines, reducing the average $L_2$ error by 18.3% (to 0.49 m), the collision rate by 22.2% (to 0.07%), and improving the trajectory prediction consistency (TPC) by 13% (to 0.47 m) compared to MomAD. More importantly, in the complex Turning-nuScenes subset, our method reduces the collision rate by 78.1% (to 0.07%) over the baseline.
Read the full abstract ▾
Computer Vision
ABRA: Teleporting Fine-Tuned Knowledge Across Domains for Open-Vocabulary Object Detection
Mattia Bernardi, Chiara Cappellino, Matteo Mosconi et al. (+3 more)
Although recent Open-Vocabulary Object Detection architectures, such as Grounding DINO, demonstrate strong zero-shot capabilities, their performance degrades significantly under domain shifts. Moreover, many domains of practical interest, such as nighttime or foggy scenes, lack large annotated
datasets, preventing direct fine-tuning. In this paper, we introduce Aligned Basis Relocation for Adaptation(ABRA), a method that transfers class-specific detection knowledge from a labeled source domain to a target domain where no training images containing these classes are accessible. ABRA formulates this adaptation as a
geometric transport problem in the weight space of a pretrained detector, aligning source and target domain experts to transport class-specific knowledge. Extensive experiments across challenging domain shifts demonstrate that ABRA successfully teleports class-level specialization under multiple adverse conditions. Our code will be made public upon acceptance.
Read the full abstract ▾
All authors Mattia Bernardi, Chiara Cappellino, Matteo Mosconi, Enver Sangineto, Angelo Porrello, Simone Calderara
Computer Vision · Cross-listed from math.CO
Generation of Maximal Snake Polyominoes Using a Deep Neural Network
Benjamin Gauthier, Alain Goupil and Fadel Toure
Maximal snake polyominoes are difficult to study numerically in large rectangles, as computing them requires the complete enumeration of all snakes for a specific rectangle size, which corresponds to a brute force
algorithm. This hinders the study of maximal snakes in larger rectangles. Moreover, most enumerable snakes lie in small rectangles, obscuring large-scale patterns. In this paper, we investigate the contribution of a deep neural network to the generation of maximal snake polyominoes from a data-driven training, where the maximality and adjacency constraints are not encoded explicitly, but learned. To this extent, we experiment with a denoising
diffusion model, which we referred as Structured Pixel Space
Diffusion (SPS
Diffusion). We find that SPS
Diffusion generalizes from small rectangles to larger ones, generating valid snakes up to 28x28 squares and producing maximal snake candidates on squares close to the current computational limit. The model is, however, prone to errors such as branching, cycles, or multiple snake components. Overall, the
diffusion model is promising and suggests that complex combinatorial objects can be understood by deep neural networks, which is useful in their investigation.
Read the full abstract ▾
Computer Vision
Deployment-Oriented Session-wise Meta-Calibration for Landmark-Based Webcam Gaze Tracking
Chenkai Zhang
Practical webcam gaze tracking is constrained not only by error, but also by calibration burden, robustness to head motion and session drift, runtime footprint, and browser use. We therefore target a deployment-oriented operating point rather than the image large-backbone regime. We cast landmark-based point-of-regard estimation as session-wise adaptation: a shared
geometric encoder produces
embeddings that can be aligned to a new session from a small calibration set. We present Equivariant Meta-Calibrated Gaze (EMC-Gaze), a lightweight landmark-only method combining an E(3)-equivariant landmark-graph encoder, local eye geometry, binocular emphasis, auxiliary 3D gaze-direction supervision, and a closed-form ridge calibrator differentiated through episodic meta-training. To reduce pose leakage, we use a two-view canonicalization consistency loss. The deployed predictor uses only facial landmarks and fits a per-session ridge head from brief calibration. In a fixation-style interactive evaluation over 33 sessions at 100 cm, EMC-Gaze achieves 5.79 +/- 1.81 deg RMSE after 9-point calibration versus 6.68 +/- 2.34 deg for Elastic Net; the gain is larger on still-head queries (2.92 +/- 0.75 deg vs. 4.45 +/- 0.30 deg). Across three subject holdouts of 10 subjects each, EMC-Gaze retains an advantage (5.66 +/- 0.19 deg vs. 6.49 +/- 0.33 deg). On MPIIFaceGaze with short per-session calibration, the eye-focused model reaches 8.82 +/- 1.21 deg at 16-shot calibration, ties Elastic Net at 1-shot, and outperforms it from 3-shot onward. The exported eye-focused encoder has 944,423 parameters, is 4.76 MB in ONNX, and supports calibrated browser prediction in 12.58/12.58/12.90 ms per sample (mean/median/p90) in Chromium 145 with ONNX Runtime Web. These results position EMC-Gaze as a calibration-friendly operating point rather than a universal state-of-the-art claim against heavier appearance-based systems.
Read the full abstract ▾
Computer Vision
SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs
Mohamad Alansari, Naufal Suryanto, Divya Velayudhan et al. (+3 more)
Multimodal large
language models (MLLMs) have advanced from image-level
reasoning to pixel-level grounding, but extending these capabilities to
videos remains challenging as models must achieve spatial precision and temporally consistent reference tracking. Existing
video MLLMs often rely on a static
segmentation token ([SEG]) for frame-wise grounding, which provides semantics but lacks temporal context, causing spatial drift, identity switches, and unstable initialization when objects move or reappear. We introduce SPARROW, a pixel-grounded
video MLLM that unifies spatial accuracy and temporal stability through two key components: (i) Target-Specific Tracked Features (TSF), which inject temporally aligned referent cues during training, and (ii) a dual-
prompt design that decodes box ([BOX]) and
segmentation ([SEG])
tokens to fuse
geometric priors with semantic grounding. SPARROW is supported by a curated referential
video dataset of 30,646
videos and 45,231 Q&A pairs and operates end-to-end without external detectors via a class-agnostic SAM2-based proposer. Integrated into three recent open-source
video MLLMs (UniPixel, GLUS, and VideoGLaMM), SPARROW delivers consistent gains across six
benchmarks, improving up to +8.9 J&F on RVOS, +5 mIoU on visual grounding, and +5.4 CLAIR on GCG. These results demonstrate that SPARROW substantially improves referential stability, spatial precision, and temporal coherence in pixel-grounded
video understanding. Project page: https://risys-lab.github.io/SPARROW
Read the full abstract ▾
All authors Mohamad Alansari, Naufal Suryanto, Divya Velayudhan, Sajid Javed, Naoufel Werghi, Muzammal Naseer
Computer Vision
Human Knowledge Integrated Multi-modal Learning for Single Source Domain Generalization
Ayan Banerjee, Kuntal Thakur and Sandeep Gupta
Generalizing image classification across domains remains challenging in critical tasks such as fundus image-based diabetic retinopathy (DR) grading and resting-state fMRI seizure onset zone (SOZ) detection. When domains differ in unknown causal factors, achieving cross-domain generalization is difficult, and there is no established methodology to objectively assess such differences without direct metadata or protocol-level information from data collectors, which is typically inaccessible. We first introduce domain conformal bounds (DCB), a theoretical framework to evaluate whether domains diverge in unknown causal factors. Building on this, we propose GenEval, a
multimodal Vision Language Models (VLM) approach that combines foundational models (e.g., MedGemma-4B) with human knowledge via Low-Rank Adaptation (LoRA) to bridge causal gaps and enhance single-source domain generalization (SDG). Across eight DR and two SOZ
datasets, GenEval achieves superior SDG performance, with average accuracy of 69.2% (DR) and 81% (SOZ), outperforming the strongest baselines by 9.4% and 1.8%, respectively.
Read the full abstract ▾
Computer Vision
Alternating Gradient Flow Utility: A Unified Metric for Structural Pruning and Dynamic Routing in Deep Networks
Tianhao Qian, Zhuoxuan Li, Jinde Cao et al. (+2 more)
Efficient deep learning traditionally relies on static heuristics like weight magnitude or activation awareness (e.g., Wanda, RIA). While successful in unstructured settings, we observe a critical limitation when applying these metrics to the structural
pruning of deep
vision networks. These contemporary metrics suffer from a magnitude bias, failing to preserve critical functional pathways. To overcome this, we propose a decoupled kinetic paradigm inspired by Alternating Gradient Flow (AGF), utilizing an absolute feature-space Taylor expansion to accurately capture the network's structural "kinetic utility". First, we uncover a topological phase transition at extreme sparsity, where AGF successfully preserves baseline functionality and exhibits topological implicit regularization, avoiding the collapse seen in models trained from scratch. Second, transitioning to architectures without strict structural priors, we reveal a phenomenon of Sparsity Bottleneck in
Vision Transformers (ViTs). Through a gradient-magnitude decoupling analysis, we discover that dynamic signals suffer from signal compression in converged models, rendering them suboptimal for real-time routing. Finally, driven by these empirical constraints, we design a hybrid routing framework that decouples AGF-guided offline structural search from online execution via zero-cost physical priors. We validate our paradigm on large-scale
benchmarks: under a 75% compression stress test on ImageNet-1K, AGF effectively avoids the structural collapse where traditional metrics aggressively fall below random sampling. Furthermore, when systematically deployed for dynamic inference on ImageNet-100, our hybrid approach achieves Pareto-optimal efficiency. It reduces the usage of the heavy expert by approximately 50% (achieving an estimated overall cost of 0.92$\times$) without sacrificing the full-model accuracy.
Read the full abstract ▾
All authors Tianhao Qian, Zhuoxuan Li, Jinde Cao, Xinli Shi, Leszek Rutkowski
Computer Vision
VQQA: An Agentic Approach for Video Evaluation and Quality Improvement
Yiwen Song, Tomas Pfister and Yale Song
Despite rapid advancements in
video generation models, aligning their outputs with complex user intent remains challenging. Existing test-time optimization methods are typically either computationally expensive or require white-box access to model internals. To address this, we present VQQA (
Video Quality Question Answering), a unified, multi-
agent framework generalizable across diverse input modalities and
video generation tasks. By dynamically generating visual questions and using the resulting
Vision-
Language Model (VLM) critiques as semantic gradients, VQQA replaces traditional, passive evaluation metrics with human-interpretable, actionable feedback. This enables a highly efficient, closed-loop
prompt optimization process via a black-box natural
language interface. Extensive experiments demonstrate that VQQA effectively isolates and resolves visual artifacts, substantially improving generation quality in just a few refinement steps. Applicable to both text-to-
video (T2V) and image-to-
video (I2V) tasks, our method achieves absolute improvements of +11.57% on T2V-CompBench and +8.43% on VBench2 over vanilla generation, significantly outperforming state-of-the-art
stochastic search and
prompt optimization techniques.
Read the full abstract ▾
Computer Vision
EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation
Tianwei Xiong, Jun Hao Liew, Zilong Huang et al. (+3 more)
Autoregressive (AR)
video generative models rely on
video tokenizers that compress pixels into discrete
token sequences. The length of these
token sequences is crucial for balancing reconstruction quality against downstream generation computational cost. Traditional
video tokenizers apply a uniform
token assignment across temporal blocks of different
videos, often wasting
tokens on simple, static, or repetitive segments while underserving dynamic or complex ones. To address this inefficiency, we introduce $\textbf{EVATok}$, a framework to produce $\textbf{E}$fficient $\textbf{V}$ideo $\textbf{A}$daptive $\textbf{Tok}$enizers. Our framework estimates optimal
token assignments for each
video to achieve the best quality-cost trade-off, develops lightweight routers for fast prediction of these optimal assignments, and trains adaptive tokenizers that encode
videos based on the assignments predicted by routers. We demonstrate that EVATok delivers substantial improvements in efficiency and overall quality for
video reconstruction and downstream AR generation. Enhanced by our advanced training recipe that integrates
video semantic encoders, EVATok achieves superior reconstruction and state-of-the-art class-to-
video generation on UCF-101, with at least 24.4% savings in average
token usage compared to the prior state-of-the-art LARP and our fixed-length baseline.
Read the full abstract ▾
All authors Tianwei Xiong, Jun Hao Liew, Zilong Huang, Zhijie Lin, Jiashi Feng, Xihui Liu
Computer Vision
MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoning
Haozhan Shen, Shilin Yan, Hongwei Xue et al. (+5 more)
Multimodal Large
Language Models (MLLMs) are increasingly used to carry out visual workflows such as navigating GUIs, where the next step depends on verified visual compositional conditions (e.g., "if a permission dialog appears and the color of the interface is green, click Allow") and the process may branch or terminate early. Yet this capability remains under-evaluated: existing
benchmarks focus on shallow-compositions or independent-constraints rather than deeply chained compositional conditionals. In this paper, we introduce MM-CondChain, a
benchmark for visually grounded deep compositional
reasoning. Each
benchmark instance is organized as a multi-layer
reasoning chain, where every layer contains a non-trivial compositional condition grounded in visual evidence and built from multiple objects, attributes, or relations. To answer correctly, an MLLM must perceive the image in detail, reason over multiple visual elements at each step, and follow the resulting execution path to the final outcome. To scalably construct such workflow-style data, we propose an agentic synthesis pipeline: a Planner orchestrates layer-by-layer generation of compositional conditions, while a Verifiable Programmatic Intermediate Representation (VPIR) ensures each layer's condition is mechanically verifiable. A Composer then assembles these verified layers into complete instructions. Using this pipeline, we construct
benchmarks across three visual domains: natural images, data charts, and GUI trajectories. Experiments on a range of MLLMs show that even the strongest model attains only 53.33 Path F1, with sharp drops on hard negatives and as depth or predicate complexity grows, confirming that deep compositional
reasoning remains a fundamental challenge.
Read the full abstract ▾
All authors Haozhan Shen, Shilin Yan, Hongwei Xue, Shuaiqi Lu, Xiaojun Tang, Guannan Zhang, Tiancheng Zhao, Jianwei Yin
Computer Vision
OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams
Yibin Yan, Jilan Xu, Shangzhe Di et al. (+2 more)
Modern visual
agents require representations that are general, causal, and physically structured to operate in real-time streaming environments. However, current
vision foundation models remain fragmented, specializing narrowly in image semantic perception, offline temporal modeling, or spatial geometry. This paper introduces OmniStream, a unified streaming visual backbone that effectively perceives, reconstructs, and acts from diverse visual inputs. By incorporating causal spatiotemporal
attention and 3D rotary positional
embeddings (3D-RoPE), our model supports efficient, frame-by-frame online processing of
video streams via a persistent KV-cache. We pre-train OmniStream using a synergistic multi-task framework coupling static and temporal representation learning, streaming
geometric reconstruction, and
vision-
language alignment on 29
datasets. Extensive evaluations show that, even with a strictly frozen backbone, OmniStream achieves consistently competitive performance with specialized experts across image and
video probing, streaming
geometric reconstruction, complex
video and spatial
reasoning, as well as robotic manipulation (unseen at training). Rather than pursuing
benchmark-specific dominance, our work demonstrates the viability of training a single, versatile
vision backbone that generalizes across semantic, spatial, and temporal
reasoning, i.e., a more meaningful step toward general-purpose visual understanding for interactive and
embodied agents.
Read the full abstract ▾
All authors Yibin Yan, Jilan Xu, Shangzhe Di, Haoning Wu, Weidi Xie
Computer Vision
GRADE: Benchmarking Discipline-Informed Reasoning in Image Editing
Mingxin Liu, Ziqian Fan, Zhaokai Wang et al. (+13 more)
Unified
multimodal models target joint understanding,
reasoning, and generation, but current image editing
benchmarks are largely confined to natural images and shallow commonsense
reasoning, offering limited assessment of this capability under structured, domain-specific constraints. In this work, we introduce GRADE, the first
benchmark to assess discipline-informed knowledge and
reasoning in image editing. GRADE comprises 520 carefully curated samples across 10 academic domains, spanning from natural science to social science. To support rigorous evaluation, we propose a multi-dimensional evaluation protocol that jointly assesses Discipline
Reasoning, Visual Consistency, and Logical Readability. Extensive experiments on 20 state-of-the-art open-source and closed-source models reveal substantial limitations in current models under implicit, knowledge-intensive editing settings, leading to large performance gaps. Beyond quantitative scores, we conduct rigorous analyses and ablations to expose model shortcomings and identify the constraints within disciplinary editing. Together, GRADE pinpoints key directions for the future development of unified
multimodal models, advancing the research on discipline-informed image editing and
reasoning. Our
benchmark and evaluation code are publicly released.
Read the full abstract ▾
All authors Mingxin Liu, Ziqian Fan, Zhaokai Wang, Leyao Gu, Zirun Zhu, Yiguo He, Yuchen Yang, Changyao Tian, Xiangyu Zhao, Ning Liao, Shaofeng Zhang, Qibing Ren, Zhihang Zhong, Xuanhe Zhou, Junchi Yan, Xue Yang
Computer Vision
Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously
Yiran Guan, Liang Yin, Dingkang Liang et al. (+5 more)
Online
Video Large
Language Models (VideoLLMs) play a critical role in supporting responsive, real-time interaction. Existing methods focus on streaming perception, lacking a synchronized logical
reasoning stream. However, directly applying test-time scaling methods incurs unacceptable response latency. To address this trade-off, we propose
Video Streaming Thinking (VST), a novel paradigm for streaming
video understanding. It supports a thinking while watching mechanism, which activates
reasoning over incoming
video clips during streaming. This design improves timely comprehension and coherent cognition while preserving real-time responsiveness by amortizing LLM
reasoning latency over
video playback. Furthermore, we introduce a comprehensive post-training pipeline that integrates VST-SFT, which structurally adapts the offline VideoLLM to causal streaming
reasoning, and VST-RL, which provides end-to-end improvement through self-exploration in a multi-turn
video interaction environment. Additionally, we devise an automated training-data synthesis pipeline that uses
video knowledge graphs to generate high-quality streaming QA pairs, with an entity-relation grounded streaming Chain-of-Thought to enforce multi-evidence
reasoning and sustained
attention to the
video stream. Extensive evaluations show that VST-7B performs strongly on online
benchmarks, e.g. 79.5% on StreamingBench and 59.3% on OVO-Bench. Meanwhile, VST remains competitive on offline long-form or
reasoning benchmarks. Compared with
Video-R1, VST responds 15.7 times faster and achieves +5.4% improvement on VideoHolmes, demonstrating higher efficiency and strong generalization across diverse
video understanding tasks. Code, data, and models will be released at https://github.com/1ranGuan/VST.
Read the full abstract ▾
All authors Yiran Guan, Liang Yin, Dingkang Liang, Jianzhong Ju, Zhenbo Luo, Jian Luan, Yuliang Liu, Xiang Bai
Computer Vision · Cross-listed from cs.LG
The Latent Color Subspace: Emergent Order in High-Dimensional Chaos
Mateusz Pach, Jessica Bader, Quentin Bouniot et al. (+2 more)
Text-to-image generation models have advanced rapidly, yet achieving fine-grained control over generated images remains difficult, largely due to limited understanding of how semantic information is encoded. We develop an interpretation of the color representation in the Variational Autoencoder latent space of FLUX.1 [Dev], revealing a structure reflecting Hue, Saturation, and Lightness. We verify our Latent Color Subspace (LCS) interpretation by demonstrating that it can both predict and explicitly control color, introducing a fully training-free method in FLUX based solely on closed-form latent-space manipulation. Code is available at https://github.com/ExplainableML/LCS.
Read the full abstract ▾
All authors Mateusz Pach, Jessica Bader, Quentin Bouniot, Serge Belongie, Zeynep Akata
Computer Vision
DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning
Yujie Wei, Xinyu Liu, Shiwei Zhang et al. (+12 more)
While large-scale
diffusion models have revolutionized
video synthesis, achieving precise control over both multi-subject identity and multi-granularity motion remains a significant challenge. Recent attempts to bridge this gap often suffer from limited motion granularity, control ambiguity, and identity degradation, leading to suboptimal performance on identity preservation and motion control. In this work, we present DreamVideo-Omni, a unified framework enabling harmonious multi-subject customization with omni-motion control via a progressive two-stage training paradigm. In the first stage, we integrate comprehensive control signals for joint training, encompassing subject appearances, global motion, local dynamics, and camera movements. To ensure robust and precise controllability, we introduce a condition-aware 3D rotary positional
embedding to coordinate heterogeneous inputs and a hierarchical motion injection strategy to enhance global motion guidance. Furthermore, to resolve multi-subject ambiguity, we introduce group and role
embeddings to explicitly anchor motion signals to specific identities, effectively disentangling complex scenes into independent controllable instances. In the second stage, to mitigate identity degradation, we design a latent identity reward feedback learning paradigm by training a latent identity reward model upon a pretrained
video diffusion backbone. This provides motion-aware identity rewards in the latent space, prioritizing identity preservation aligned with human preferences. Supported by our curated large-scale
dataset and the comprehensive DreamOmni Bench for multi-subject and omni-motion control evaluation, DreamVideo-Omni demonstrates superior performance in generating high-quality
videos with precise controllability.
Read the full abstract ▾
All authors Yujie Wei, Xinyu Liu, Shiwei Zhang, Hangjie Yuan, Jinbo Xing, Zhekai Chen, Xiang Wang, Haonan Qiu, Rui Zhao, Yutong Feng, Ruihang Chu, Yingya Zhang, Yike Guo, Xihui Liu, Hongming Shan
Computer Vision
Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training
Fangfu Liu, Diankun Wu, Jiawei Chi et al. (+7 more)
Humans perceive and understand real-world spaces through a stream of visual observations. Therefore, the ability to streamingly maintain and update spatial evidence from potentially unbounded
video streams is essential for spatial intelligence. The core challenge is not simply longer context windows but how spatial information is selected, organized, and retained over time. In this paper, we propose Spatial-TTT towards streaming visual-based spatial intelligence with test-time training (TTT), which adapts a subset of parameters (fast weights) to capture and organize spatial evidence over long-horizon scene
videos. Specifically, we design a hybrid architecture and adopt large-chunk updates parallel with sliding-window
attention for efficient spatial
video processing. To further promote spatial awareness, we introduce a spatial-predictive mechanism applied to TTT layers with 3D spatiotemporal convolution, which encourages the model to capture
geometric correspondence and temporal continuity across frames. Beyond architecture design, we construct a
dataset with dense 3D spatial descriptions, which guides the model to update its fast weights to memorize and organize global 3D spatial signals in a structured manner. Extensive experiments demonstrate that Spatial-TTT improves long-horizon spatial understanding and achieves state-of-the-art performance on
video spatial
benchmarks. Project page: https://liuff19.github.io/Spatial-TTT.
Read the full abstract ▾
All authors Fangfu Liu, Diankun Wu, Jiawei Chi, Yimo Cai, Yi-Hsin Hung, Xumin Yu, Hao Li, Han Hu, Yongming Rao, Yueqi Duan
Computer Vision
Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing
Baifeng Shi, Stephanie Fu, Long Lian et al. (+10 more)
Multi-modal large
language models (MLLMs) have advanced general-purpose
video understanding but struggle with long, high-resolution
videos – they process every pixel equally in their
vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-
token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the
video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual
tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution
videos and achieving superior results on
video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form
video QA
benchmark with 5-minute 4K-resolution
videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: https://autogaze.github.io/.
Read the full abstract ▾
All authors Baifeng Shi, Stephanie Fu, Long Lian, Hanrong Ye, David Eigen, Aaron Reite, Boyi Li, Jan Kautz, Song Han, David M. Chan, Pavlo Molchanov, Trevor Darrell, Hongxu Yin
Computer Vision
EndoCoT: Scaling Endogenous Chain-of-Thought Reasoning in Diffusion Models
Xuanlang Dai, Yujie Zhou, Long Xing et al. (+6 more)
Recently,
Multimodal Large
Language Models (MLLMs) have been widely integrated into
diffusion frameworks primarily as text encoders to tackle complex tasks such as spatial
reasoning. However, this paradigm suffers from two critical limitations: (i) MLLMs text encoder exhibits insufficient
reasoning depth. Single-step encoding fails to activate the Chain-of-Thought process, which is essential for MLLMs to provide accurate guidance for complex tasks. (ii) The guidance remains invariant during the decoding process. Invariant guidance during decoding prevents DiT from progressively decomposing complex instructions into actionable denoising steps, even with correct MLLM encodings. To this end, we propose Endogenous Chain-of-Thought (EndoCoT), a novel framework that first activates MLLMs'
reasoning potential by iteratively refining latent thought states through an iterative thought guidance module, and then bridges these states to the DiT's denoising process. Second, a terminal thought grounding module is applied to ensure the
reasoning trajectory remains grounded in textual supervision by aligning the final state with ground-truth answers. With these two components, the MLLM text encoder delivers meticulously reasoned guidance, enabling the DiT to execute it progressively and ultimately solve complex tasks in a step-by-step manner. Extensive evaluations across diverse
benchmarks (e.g., Maze, TSP, VSP, and Sudoku) achieve an average accuracy of 92.1%, outperforming the strongest baseline by 8.3 percentage points. The code and
dataset are publicly available at https://internlm.github.io/EndoCoT/.
Read the full abstract ▾
All authors Xuanlang Dai, Yujie Zhou, Long Xing, Jiazi Bu, Xilin Wei, Yuhong Liu, Beichen Zhang, Kai Chen, Yuhang Zang
Computer Vision
DVD: Deterministic Video Depth Estimation with Generative Priors
Hongfei Zhang, Harold Haodong Chen, Chenfei Liao et al. (+12 more)
Existing
video depth estimation faces a fundamental trade-off: generative models suffer from
stochastic geometric hallucinations and scale drift, while discriminative models demand massive labeled
datasets to resolve semantic ambiguities. To break this impasse, we present DVD, the first framework to deterministically adapt pre-trained
video diffusion models into single-pass depth regressors. Specifically, DVD features three core designs: (i) repurposing the
diffusion timestep as a structural anchor to balance global stability with high-frequency details; (ii) latent
manifold rectification (LMR) to mitigate regression-induced over-smoothing, enforcing differential constraints to restore sharp boundaries and coherent motion; and (iii) global affine coherence, an inherent property bounding inter-window divergence, which enables seamless long-
video inference without requiring complex temporal
alignment. Extensive experiments demonstrate that DVD achieves state-of-the-art zero-shot performance across
benchmarks. Furthermore, DVD successfully unlocks the profound
geometric priors implicit in
video foundation models using 163x less task-specific data than leading baselines. Notably, we fully release our pipeline, providing the whole training suite for SOTA
video depth estimation to benefit the open-source community.
Read the full abstract ▾
All authors Hongfei Zhang, Harold Haodong Chen, Chenfei Liao, Jing He, Zixin Zhang, Haodong Li, Yihao Liang, Kanghao Chen, Bin Ren, Xu Zheng, Shuai Yang, Kun Zhou, Yinchuan Li, Nicu Sebe, Ying-Cong Chen
Computer Vision · Cross-listed from cs.CL
SciMDR: Advancing Scientific Multimodal Document Reasoning
Ziyu Chen, Yilun Zhao, Chengye Wang et al. (+3 more)
Constructing scientific
multimodal document
reasoning datasets for foundation model training involves an inherent trade-off among scale, faithfulness, and realism. To address this challenge, we introduce the synthesize-and-reground framework, a two-stage pipeline comprising: (1) Claim-Centric QA Synthesis, which generates faithful, isolated QA pairs and
reasoning on focused segments, and (2) Document-Scale Regrounding, which programmatically re-embeds these pairs into full-document tasks to ensure realistic complexity. Using this framework, we construct SciMDR, a large-scale training
dataset for cross-modal comprehension, comprising 300K QA pairs with explicit
reasoning chains across 20K scientific papers. We further construct SciMDR-Eval, an expert-annotated
benchmark to evaluate
multimodal comprehension within full-length scientific workflows. Experiments demonstrate that models fine-tuned on SciMDR achieve significant improvements across multiple scientific QA
benchmarks, particularly in those tasks requiring complex document-level
reasoning.
Read the full abstract ▾
All authors Ziyu Chen, Yilun Zhao, Chengye Wang, Rilyn Han, Manasi Patwardhan, Arman Cohan
Computer Vision
Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation
Xiangyu Zhao, Peiyuan Zhang, Junming Lin et al. (+7 more)
Reinforcement learning (RL) has emerged as a promising paradigm for enhancing image editing and text-to-image (T2I) generation. However, current reward models, which act as critics during RL, often suffer from
hallucinations and assign noisy scores, inherently misguiding the optimization process. In this paper, we present FIRM (Faithful Image Reward Modeling), a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing. First, we design tailored data curation pipelines to construct high-quality scoring
datasets. Specifically, we evaluate editing using both execution and consistency, while generation is primarily assessed via instruction following. Using these pipelines, we collect the FIRM-Edit-370K and FIRM-Gen-293K
datasets, and train specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) that accurately reflect these criteria. Second, we introduce FIRM-Bench, a comprehensive
benchmark specifically designed for editing and generation critics. Evaluations demonstrate that our models achieve superior
alignment with human judgment compared to existing metrics. Furthermore, to seamlessly integrate these critics into the RL pipeline, we formulate a novel "Base-and-Bonus" reward strategy that balances competing objectives: Consistency-Modulated Execution (CME) for editing and Quality-Modulated
Alignment (QMA) for generation. Empowered by this framework, our resulting models FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial performance breakthroughs. Comprehensive experiments demonstrate that FIRM mitigates
hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models. All of our
datasets, models, and code have been publicly available at https://firm-reward.github.io.
Read the full abstract ▾
All authors Xiangyu Zhao, Peiyuan Zhang, Junming Lin, Tianhao Liang, Yuchen Duan, Shengyuan Ding, Changyao Tian, Yuhang Zang, Junchi Yan, Xue Yang
Computer Vision
One Model, Many Budgets: Elastic Latent Interfaces for Diffusion Transformers
Moayed Haji-Ali, Willi Menapace, Ivan Skorokhodov et al. (+6 more)
Diffusion transformers (DiTs) achieve high generative quality but lock FLOPs to image resolution, limiting principled latency-quality trade-offs, and allocate computation uniformly across input spatial
tokens, wasting resource allocation to unimportant regions. We introduce Elastic Latent Interface
Transformer (ELIT), a drop-in, DiT-compatible mechanism that decouples input image size from compute. Our approach inserts a latent interface, a learnable variable-length
token sequence on which standard
transformer blocks can operate. Lightweight Read and Write cross-
attention layers move information between spatial
tokens and latents and prioritize important input regions. By training with random dropping of tail latents, ELIT learns to produce importance-ordered representations with earlier latents capturing global structure while later ones contain information to refine details. At inference, the number of latents can be dynamically adjusted to match compute constraints. ELIT is deliberately minimal, adding two cross-
attention layers while leaving the rectified flow objective and the DiT stack unchanged. Across
datasets and architectures (DiT, U-ViT, HDiT, MM-DiT), ELIT delivers consistent gains. On ImageNet-1K 512px, ELIT delivers an average gain of $35.3\%$ and $39.6\%$ in FID and FDD scores. Project page: https://snap-research.github.io/elit/
Read the full abstract ▾
All authors Moayed Haji-Ali, Willi Menapace, Ivan Skorokhodov, Dogyun Park, Anil Kag, Michael Vasilkovsky, Sergey Tulyakov, Vicente Ordonez, Aliaksandr Siarohin
Computer Vision
BiGain: Unified Token Compression for Joint Generation and Classification
Jiacheng Liu, Shengkun Tang, Jiacheng Cui et al. (+2 more)
Acceleration methods for
diffusion models (e.g.,
token merging or downsampling) typically optimize synthesis quality under reduced compute, yet often ignore discriminative capacity. We revisit
token compression with a joint objective and present BiGain, a training-free, plug-and-play framework that preserves generation quality while improving classification in accelerated
diffusion models. Our key insight is frequency separation: mapping feature-space signals into a frequency-aware representation disentangles fine detail from global semantics, enabling compression that respects both generative fidelity and discriminative utility. BiGain reflects this principle with two frequency-aware operators: (1)
Laplacian-gated
token merging, which encourages merges among spectrally smooth
tokens while discouraging merges of high-contrast
tokens, thereby retaining edges and textures; and (2) Interpolate-Extrapolate KV Downsampling, which downsamples keys/values via a controllable interextrapolation between nearest and average pooling while keeping queries intact, thereby conserving
attention precision. Across DiT- and U-Net-based backbones and ImageNet-1K, ImageNet-100, Oxford-IIIT Pets, and COCO-2017, our operators consistently improve the speed-accuracy trade-off for
diffusion-based classification, while maintaining or enhancing generation quality under comparable acceleration. For instance, on ImageNet-1K, with 70%
token merging on Stable
Diffusion 2.0, BiGain increases classification accuracy by 7.15% while improving FID by 0.34 (1.85%). Our analyses indicate that balanced spectral retention, preserving high-frequency detail and low/mid-frequency semantics, is a reliable design rule for
token compression in
diffusion models. To our knowledge, BiGain is the first framework to jointly study and advance both generation and classification under accelerated
diffusion, supporting lower-cost deployment.
Read the full abstract ▾
All authors Jiacheng Liu, Shengkun Tang, Jiacheng Cui, Dongkuan Xu, Zhiqiang Shen
Computer Vision
SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation
Jun Luo, Jiaxiang Tang, Ruijie Lu et al. (+1 more)
Text-to-3D scene generation from natural
language is highly desirable for digital content creation. However, existing methods are largely domain-restricted or reliant on predefined spatial relationships, limiting their capacity for unconstrained, open-vocabulary 3D scene synthesis. In this paper, we introduce SceneAssistant, a visual-feedback-driven
agent designed for open-vocabulary 3D scene generation. Our framework leverages modern 3D object generation model along with the spatial
reasoning and planning capabilities of
Vision-
Language Models (VLMs). To enable open-vocabulary scene composition, we provide the VLMs with a comprehensive set of atomic operations (e.g., Scale, Rotate, FocusOn). At each interaction step, the VLM receives rendered visual feedback and takes actions accordingly, iteratively refining the scene to achieve more coherent spatial arrangements and better
alignment with the input text. Experimental results demonstrate that our method can generate diverse, open-vocabulary, and high-quality 3D scenes. Both qualitative analysis and quantitative human evaluations demonstrate the superiority of our approach over existing methods. Furthermore, our method allows users to instruct the
agent to edit existing scenes based on natural
language commands. Our code is available at https://github.com/ROUJINN/SceneAssistant
Read the full abstract ▾
All authors Jun Luo, Jiaxiang Tang, Ruijie Lu, Gang Zeng
Computer Vision
Andy Li, Aiden Durrant, Milan Markovic et al. (+1 more)
Vision Transformers require significant computational resources and
memory bandwidth, severely limiting their deployment on resource-constrained hardware. Most structured
pruning methods reduce theoretical cost effectively, yet they typically operate at a single structural granularity and depend on multi-stage pipelines with importance ranking, auxiliary solvers or post-hoc magnitude thresholding, followed by a separate fine-tuning phase to recover accuracy. We propose Hierarchical Auto-
Pruning (HiAP), which casts ViT
pruning as a single budget-aware learning problem and jointly allocates sparsity across four granularities in one end-to-end phase. HiAP introduces
stochastic Gumbel-Sigmoid gates at macro level (
attention heads and FFN blocks) and micro level (intra-head dimensions and FFN
neurons), and trains them against the task loss together with an analytical MAC cost term. Two penalty weights set how much compute is removed, while the gates gradually harden into a dense, smaller sub-network at convergence. It does not require importance heuristics, ranking metrics, auxiliary solvers or secondary fine-tuning. On ImageNet, HiAP compresses DeiT-Base to 7.4G MACs at 80.88% top-1 80.40% at 7.8G without
distillation and DeiT-Small to 3.1G at 79.33%, competitive with substantially more complex pipelines at matched compute. The structurally pruned network can be accelerated natively on stock kernels, and more than 90% of the theoretical MAC reduction is realized as measured throughput at batch 256 on an A100.
Read the full abstract ▾
All authors Andy Li, Aiden Durrant, Milan Markovic, Georgios Leontidis