Computer Vision
SpurCon: Weighted Supervised Contrastive Learning for Mitigating Spurious Cues in Medical Imaging
Shenhav Nadir, Meir Yossef Levi, Eyal Gofer et al. (+1 more)
Despite the rapid progress of deep neural networks in visual recognition, their adoption in high-risk medical applications remains limited due to reliability and robustness concerns. Models may exploit spurious correlations, particularly in medical imaging, where devices or treatment artifacts often co-occur with pathology. In small or imbalanced
datasets, such cues further reduce worst-group performance and undermine clinical trust. To solve these issues, two major challenges should be addressed: identifying
dataset-specific spurious cues, which typically require domain knowledge, and mitigating reliance on them. To tackle both, we propose SpurCon, a lightweight framework based on a novel supervised contrastive loss formulation that leverages available metadata and predicted spurious labels to enhance robustness. We introduce a fast few-shot procedure, without network training, to estimate spurious labels using a small number of expert-annotated samples. We then propose a weighted supervised contrastive objective, WtSupCon, that reshapes the representation geometry by assigning sample-specific weights that depend on the [pathology, spurious, metadata] combination. For example, the highest weight is assigned to samples that differ only in their spurious label. This yields highly similar representations for images with the same metadata and pathology, differing only in the predicted spurious label. Our method operates on pretrained image encoders (such as BiomedCLIP) and trains only a lightweight projection head. We evaluate SpurCon on a synthetic setting and on Waterbirds, CheXpert, a chest X-ray classification
dataset, and ISIC 2020, a skin cancer classification
dataset. Our approach delivers the best spurious-mitigation performance, balancing well worst-group and overall accuracy on multiple
datasets.
Read the full abstract ▾
All authors Shenhav Nadir, Meir Yossef Levi, Eyal Gofer, Guy Gilboa
Computer Vision
CoinVE-200K: A Large-Scale High-Quality Dataset for Compositional Instruction-Guided Video Editing
Fuchen Long, Cong Wang, Zitao Gao et al. (+8 more)
The quality and diversity of instruction-based
video editing
datasets are steadily improving, yet existing
datasets mainly focus on single editing operations and fall short in supporting compositional instruction-guided
video editing. In particular, multiple editing intents must be jointly understood and faithfully executed within the same
video. To address this issue, we introduce CoinVE-200K, a large-scale, high-quality
dataset for Compositional Instruction-Guided
Video Editing. CoinVE-200K contains 1080p
video-editing pairs of up to 201 frames, covering diverse compositional scenarios where each sample involves 2 to 5 atomic editing operations. The instructions target humans, objects, and backgrounds, and cover edit types such as addition, removal, modification, and stylization. All samples are built through a carefully designed generation and filtering pipeline to ensure instruction faithfulness, visual quality, temporal consistency, and compositional diversity. We also introduce CoinVE-Bench, a
benchmark for compositional-instruction
video editing across diverse subjects, operation types, and instruction complexities. Furthermore, we present CoinVE-Edit, a 22B compositional
video editing model built upon Wan2.1-T2V-14B and Qwen3-VL-8B-Instruct. CoinVE-Edit disentangles region-aware
attention for different editing instructions, enabling precise multi-region editing while preserving irrelevant content and temporal coherence. Experiments on CoinVE-Bench show that CoinVE-Edit achieves strong performance in instruction following, compositional editing accuracy, visual quality, and temporal consistency.
Read the full abstract ▾
All authors Fuchen Long, Cong Wang, Zitao Gao, Wenhao Zhong, Yu Cheng, Xiaolu Hou, Yan Li, Xiao Cao, Xinlong Sun, Xi Chen, Yu Liu
Computer Vision
Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models
Zongyang Qiu, Yihan Wu, Kaixuan Fan et al. (+2 more)
Unified
multimodal models (UMMs) are motivated by the hope that understanding and generation reinforce each other but controlled ablations repeatedly find that adding a generation objective leaves understanding flat. Joint-training studies cannot settle the disagreement: with overlapping supervision, a gain cannot be attributed to the architecture rather than the data. To further investigate the relationship between the two directions in UMMs, we separate them by construction. A novel visual entity, a rendered 3D asset paired with a pseudo-word screened for absence from the frozen model's behavior, is bound through exactly one task direction, and the untrained direction is then measured. We find that the channel is real in both directions, but the directions differ in kind: generation training installs a name the model can only match among candidates; understanding training installs one it can also produce. What governs cross-task usability is where the binding enters the shared computation. An
alignment probe predicts export across 36 configurations (Spearman $\rho = +0.68$). That objective's
alignment term, maximized in closed form over activations with every weight frozen, makes a concept drawable when injected at layer 7 of 28 and is indistinguishable from the base model from layer 14 on, while the weight-based version of the same edit peaks at layers 10-14. In an observational series of four models, this window appears only where the understanding pathway is a semantic
vision encoder, suggesting that unified weights are not enough: the two directions must share a semantic format at the entry point. Exploiting the rule, a mid-stack
alignment objective acquires the concept for a $0.1\%$ relative loss of the model's general text-to-image ability, against $41\%$ for the standard generative route. Our code is at https://github.com/Zane-ZYQiu/entry-point-umm.
Read the full abstract ▾
All authors Zongyang Qiu, Yihan Wu, Kaixuan Fan, Bo Li, Hui Xiong
Computer Vision
Leveraging existing sparse point annotations for benthic imagery dense segmentation
Cesar Borja, Breck A. McCollum, Jarret E. Byrnes et al. (+2 more)
The health of marine ecosystems is a critical indicator of global environmental change, yet the physical constraints of
underwater observation and the intrinsic challenges of processing marine imagery severely limit the scalability of systematic monitoring. While recent visual foundation models such as the Segment Anything Model (SAM) series show great promise, they still struggle with the fine-grained recognition required in these complex scenarios and still require expert supervision. Our work addresses this gap by bridging state-of-the-art foundation models with existing sparse supervision. Because historical benthic surveys are typically annotated with only a few sparse expert points per image, we utilize these legacy point-labels as visual
prompts for SAM2. Our primary contribution is a novel mechanism to automatically identify which of these points are suitable, and which are actively harmful, when used for propagation. By filtering out unreliable points, we extract high-quality pseudo-ground-truth masks capable of training more accurate, fine-grained semantic
segmentation models. We demonstrate the effectiveness of our approach on public benthic data and introduce a new, challenging
benchmark featuring real-world sparse expert annotations, paving the way for scalable ecological analysis.
Read the full abstract ▾
All authors Cesar Borja, Breck A. McCollum, Jarret E. Byrnes, Kenneth Sebens, Ana C. Murillo
Computer Vision
MSEditor: Toward Consistent Multi-Shot Video Editing
Kunyu Feng, Yue Ma, Bingyuan Wang et al. (+6 more)
In this paper, we tackle the problem of performing consistent, unified modifications to a multi-shot
video sequence. This task is particularly challenging because multi-shot
videos consist of discontinuous temporal segments that vary significantly in viewpoint, camera scale, and subject pose, leading to severe identity drift and cumulative error propagation. Achieving coherent edits requires establishing reliable cross-shot semantic awareness to maintain stable subject appearance and visual continuity across these disjointed boundaries. To address this, we propose MSEditor, the first framework designed specifically for consistent multi-shot
video editing. To overcome the scarcity of high-quality multi-shot training data, we repurpose existing multi-view
video datasets to provide robust cross-shot supervision. Architecturally, we introduce a Supervisory Adapter that injects this cross-shot information into the
diffusion backbone, enabling the model to learn identity-consistent representations. Furthermore, to effectively mitigate cumulative errors and ensure long-range temporal coherence, we design a Cross-Shot Packing strategy that dynamically aggregates information from semantically related shots within the self-
attention window. Extensive experiments demonstrate that MSEditor significantly outperforms existing methods on our curated multi-shot
video editing
benchmark in terms of identity preservation, temporal stability, and overall visual quality.
Read the full abstract ▾
All authors Kunyu Feng, Yue Ma, Bingyuan Wang, Yuefeng Wang, Zhiyuan Qin, Hao Cheng, Hao Li, Qifeng Chen, Zeyu Wang
Computer Vision
Code as Representation: A Compilable Parsing Paradigm for Academic Documents
Rihui Jin, Jun Wang, chengyuan zhu et al. (+11 more)
Academic papers are a primary carrier of scientific knowledge, yet most of this knowledge remains locked in PDFs that are optimized for human reading rather than machine use. For
Multimodal Large
Language Models (MLLMs), the core challenge is not only perception, but representation: scientific pages interleave text with Structured Academic Elements (SAEs) such as tables, formulas, charts, and pseudocode, whose structure, data, and logic are poorly preserved by common surrogates like Markdown. We therefore propose Compilable Academic Document Parsing (CADP), a paradigm that reconstructs a full page as contextual \LaTeX{} plus executable Python, so that structure-preserving elements and executable chart representations can be reconstructed, recompiled, and directly verified against the source page. To support this setting, we introduce CADP-Bench, an expert-verified
benchmark of full academic pages containing tightly coupled text and multiple SAE types, evaluated through a re-injection compilation protocol. We further study current capabilities using SOTA MLLMs and an exploratory multi-
agent baseline that incorporates common agentic techniques. Results show that even frontier models still struggle to produce high-fidelity executable reconstructions, highlighting substantial room for improvement in structure-aware scientific document parsing. CADP-Bench is released for future research.
Read the full abstract ▾
All authors Rihui Jin, Jun Wang, chengyuan zhu, Liang Mingyu, Yue Gao, Li Yunxuan, Kuicai Dong, Guilin Qi, Lin Ren, Yongrui Chen, Xinbang Dai, Jiaqi Li, Tongtong Wu, Gholamreza Haffari
Computer Vision
GroupForward: Building Referable 3D Scenes via Instance-Grouped Feed-Forward Gaussian Splatting
Qijian Tian, Zimeng Wu, Xuhong Wang et al. (+2 more)
Simultaneously reconstructing and understanding 3D environments is essential for
embodied agents. Toward this goal, feed-forward semantic 3D Gaussian Splatting (3DGS) efficiently constructs semantic scene representations from sparse multi-view observations. However, existing methods lack explicit instance discrimination and mainly support category- or phrase-based semantic queries. To this end, we propose GroupForward, an instance-grouped feed-forward Gaussian splatting model that reconstructs geometry, appearance, instance structure, and semantics from sparse, unposed, and uncalibrated multi-view images. Unlike existing methods that attach high-dimensional semantic features to each Gaussian, GroupForward learns compact instance
embeddings that group Gaussians into cross-view consistent 3D instances, reformulating feed-forward semantic 3DGS from per-Gaussian semantic feature rendering to instance-level semantic aggregation and propagation. Building on these instance groups, we further propose a Referential Scene
Reasoning Framework (RSRF) for complex 3D referring
segmentation. RSRF constructs an instance-grouped 3D scene graph and retrieves candidate instances for a given referring expression. A
vision-
language model then reasons over structured instance evidence and multi-view observations to identify the referred instance among the candidates. RSRF thereby extends
language interaction from simple semantic querying to complex referential scene
reasoning. Experiments on semantic reconstruction and referential
reasoning demonstrate the effectiveness of our instance-grouped reconstruction and
reasoning framework.
Read the full abstract ▾
All authors Qijian Tian, Zimeng Wu, Xuhong Wang, Lizhuang Ma, Xin Tan
Computer Vision
Explainable AI-Powered Framework for Video-Based Skill Assessment in Cataract Surgery
Mohammad Javad Ahmadi and Hamid D. Taghirad
Persistent shortages in the
surgical workforce and inherent limitations of traditional training methods highlight the necessity of automated, data-driven approaches in
surgical education. This study addresses these challenges by introducing a novel, explainable AI-powered framework for automated skill assessment, specifically focusing on cataract surgery. We present the world's largest
dataset of cataract surgery
videos, comprising 2,000 recordings. Additionally, we propose an AI-powered analytical framework that employs advanced computer
vision and signal-processing techniques to automatically evaluate
surgical videos to derive objective, quantitative performance indicators that complement or potentially replace subjective scoring methods. A significant advantage of our framework over previous methods lies precisely in its explainability of outputs, elevating it beyond merely an opaque skill classification tool. Through experimental analysis of 83 cataract surgery
videos, we demonstrate that the automatically computed metrics exhibit strong correlations with expert-based subjective evaluations, achieving up to 87% accuracy in
surgical skill assessment. Each metric was individually examined, and expert surgeons provided subjective ratings using the newly introduced Capsulorhexis Skill Assessment System (CSAS). These subjective assessments were compared with ten objective motion-based metrics extracted through our framework. The results indicated a robust correlation between subjective ratings and automated indicators, underscoring the framework's capacity to accurately model
surgical expertise.
Read the full abstract ▾
Computer Vision
BrainNorm: A Foundation Model that knows Normal via Semantic Atlas Pretraining
Madhumitha Venkatesh, Shanawaj S Madarkar and Konda Reddy Mopuri
We introduce BrainNorm, a normative foundation model, trained and tested on ~66,000 T1-weighted structural MRI (T1w sMRI) scans. By leveraging
language-image style contrastive pretraining on healthy cohorts across ages, BrainNorm learns a Semantic Atlas Latent space (SAL), where each scan is represented as a set of atlas-parcel
embeddings. This yields parcel-specific healthy aging template trajectories that support age-consistent template matching and localized deviation scoring relative to a subject's chronological age. Across 6 downstream cohorts, BrainNorm demonstrates generalization evaluated across 25 task-setting combinations spanning age estimation,
brain-age gap estimation, parcel identification, and single- & multi-disease classification tasks under direct inference, zero-shot, few-shot & full-data linear-probe settings. The resulting deviation patterns in SAL space enable zero-shot tasks for disease prediction using parcel-wise abnormalities. Fine-tuning on healthy-only cohorts of downstream
datasets further improves the performance of various tasks. Across all classification tasks, linear probing on BrainNorm's frozen
embeddings outperforms 9 baselines finetuned under end-to-end supervision. Furthermore, the localized deviations identified by BrainNorm across various neurodegenerative disorders closely align with established neurodegeneration pathology in clinical literature.
Read the full abstract ▾
Computer Vision
Looking Beyond the Scale: Do Surgical Skill Models Learn Transferable Representations Across Assessment Rubrics?
Hanna Hoffmann, Felix von Bechtolsheim, Stefanie Speidel et al. (+1 more)
Vision-based
surgical skill assessment has shown strong in-domain results, yet a fundamental question remains unasked: do these models learn transferable representations of
surgical proficiency, or do they merely encode
dataset-specific visual patterns? This paper systematically analyzes what limits cross-domain skill transfer between the GOALS and OSATS assessment scales using the LASANA and JIGSAWS
datasets. Each evaluated method serves a targeted diagnostic purpose: end-to-end training to test whether supervised skill learning transfers directly, Adaptive Sharpness-Aware Minimization (ASAM) to probe whether flatter loss landscapes improve generalization, and augmentation-based self-supervised and contrastive learning to assess whether domain-invariant pretraining decouples skill from visual context. Transfer is evaluated in both directions using a disjoint-participant held-out test set for JIGSAWS. Results reveal an asymmetry: backbones pretrained on JIGSAWS achieve CCC values of 0.77 to 0.80 on LASANA, closely matching the end-to-end baseline, showing cross-rubric transfer is feasible when the target domain provides consistent supervision. Transfer to JIGSAWS fails across all methods, likely due to annotation inconsistencies. Control experiments with a Kinetics-pretrained backbone suggest task-specific heads carry the majority of the skill prediction burden, while the backbone need only provide adequate spatiotemporal features. These findings offer a new perspective on
vision-based skill assessment: the central question of whether skill representations transfer across scoring systems has not been previously investigated. Results indicate the visual component is dominant but not solely responsible for skill prediction; further work is needed to conclusively disentangle transferable skill features from those bound to a specific visual domain.
Read the full abstract ▾
All authors Hanna Hoffmann, Felix von Bechtolsheim, Stefanie Speidel, Rebecca Hisey
Computer Vision
SE-MoLoRA: Shared-Expert LoRA Adapters for Domain-Specific Photographic Assessment
Bishwash Khanal, Anlan Zhang, Sasu Tarkoma et al. (+2 more)
Vision-
language models can describe images fluently, but they often fail to provide actionable photographic critique because semantic content and aesthetic judgment remain entangled. We propose SE-MoLoRA, a modular parameter-efficient adaptation framework for domain-specific photographic assessment. The method separates general photographic knowledge from specialist residual judgments using an always-active shared LoRA expert and routed adapters for composition, lighting, and technical quality. A lightweight query router selects the relevant specialist, enabling targeted critique without training separate full models. A rank-64 shared adapter captures broad photographic vocabulary, while rank-32 specialists learn domain-specific residuals with an orthogonal regularization penalty that encourages disentangled representations. Training data is obtained by distilling the Reddit Photo Critique
Dataset into domain-labeled critique samples. On held-out critique generation, SE-MoLoRA improves BERTScore-F1 from 0.2317 to 0.4215 over monolithic LoRA and is preferred in 84.6% of pairwise comparisons, while using fewer active parameters than separate specialist models. SVD-based ablation study shows that shared-specialist decomposition and orthogonal regularization reduce expert overlap. These results demonstrate that modular adaptation improves controllability and specificity in
multimodal photographic critique.
Read the full abstract ▾
All authors Bishwash Khanal, Anlan Zhang, Sasu Tarkoma, Tommi Mikkonen, Abhishek Kumar
Computer Vision
When More Foundation Models Means Less: Diagnosing and Addressing Multi-View Fusion Failure
Yibo Liu and Bowen Jiang
Foundation-model hubs turn multi-view fusion into a selection problem: from a large heterogeneous encoder pool, which views should be fused, and how many? We show that downstream performance is non-monotonic in the number of fused encoders; later views can be redundant or task-misaligned, causing accuracy to saturate or decline. We formalise this setting as view-set composition and propose KAGES (Kernel-
Alignment Greedy Encoder Selector), a label-aware method that orders frozen encoders by their marginal gain in centred kernel-target
alignment. KAGES requires no downstream classifier training during selection, evaluates each candidate in $\mathcal{O}(n^2)$ time independent of encoder dimension, and admits a conditional $(1-e^{-\gamma})$ prefix-wise guarantee under monotonicity and a positive submodularity ratio. Across five recognition regimes and low-shot, larger-pool, and full-data protocols, KAGES improves average AULC over full fusion by 3.9, 5.8, and 3.3 points, respectively, and exceeds DPP and facility-location selection in average AULC. Image
retrieval exhibits later, task-dependent saturation along the KAGES ordering, while peak-then-decline reproduces in frozen-LLM fusion. These results show that effective large-pool fusion depends on selecting a compact, task-aligned set of views rather than indiscriminately fusing more encoders.
Read the full abstract ▾
Computer Vision
NeuroPath: Brain-Inspired Dual-Pathway Graph Convolutional Networks for Skeleton-Based Action Recognition
Kanglei Zhou, Ruizhi Cai, Hubert P. H. Shum et al. (+2 more)
Skeleton-based action recognition aims to recognize human actions from sequences of human joint coordinates. Most existing Spatial-Temporal Graph Convolutional Networks (STGCNs) have achieved promising results by modeling skeletal structures with implicit spatial-temporal representations. However, our empirical study reveals a clear performance imbalance across different skeletal modalities, indicating that implicitly coupling spatial and temporal information limits the full exploitation of complementary structural and motion cues. Inspired by the ventral and dorsal pathways in human perception, we propose Dual-Pathway Graph Convolutional Networks (NeuroPath), which adopt a dual-pathway architecture for separate yet collaborative modeling of spatial and temporal information. Specifically, transformation units first convert the input into pathway-specific skeletal representations, allowing each pathway to focus on complementary aspects of human motion. To further capture coordinated joint behaviors and their interrelationships, we introduce a group graph convolution block that dynamically identifies key body parts and models their spatial-temporal dependencies. In addition, inter-pathway dynamic fusion modules integrate complementary inter-modal information across pathways, facilitating higher-level semantic interpretation of actions. Extensive experiments on Kinetics Skeleton 400, NTU RGB+D 60, and NTU RGB+D 120 demonstrate consistent performance improvements, validating the effectiveness of dual-pathway spatial-temporal modeling for skeleton-based action recognition.
Read the full abstract ▾
All authors Kanglei Zhou, Ruizhi Cai, Hubert P. H. Shum, Frederick W. B. Li, Xiaohui Liang
Computer Vision
S$^3$AM: A Single-Stream SAM with Reliability-Calibrated Frequency Adapter for Multi-modal Salient Object Detection
Ruichao Hou, Boyue Xu, Tongwei Ren et al. (+3 more)
Vision foundation models have recently advanced multi-modal salient object detection (MSOD) through parameter-efficient tuning and
prompt learning. However, existing Segment Anything Model (SAM)-adapted MSOD methods often rely on dual-stream encoders or auxiliary
prompt generators, leading to redundant computation. Although a single-stream alternative can reduce this cost, early fusion may also propagate noisy or misaligned auxiliary high-frequency cues through the backbone. In this paper, we propose a novel single-stream framework that integrates reliability-calibrated frequency adaptation into the adopted SAM backbone for MSOD. It avoids duplicated foundation backbones while explicitly controlling auxiliary frequency injection. Specifically, we design a mixture of frequency experts module, which uses the stationary wavelet transform to decompose each modality and aggregate cross-modal frequency information. We further introduce a reliability-calibrated frequency adapter with a dual-gate calibration mechanism, which selectively propagates the calibrated residual across
transformer stages while jointly controlling its injection strength and cross-modal reliability. A hypernetwork-guided semantic-structural decoder then combines semantic mask features from the adopted backbone with Mamba-based structural detail recovery. Comprehensive experiments on RGB-D, RGB-T, and RGB-NIR salient object detection
benchmarks validate that the proposed framework achieves competitive performance with only 12.20M trainable parameters, accounting for 5.4% of the total parameters. The code will be available at https://github.com/xuboyue1999/SSSAM.
Read the full abstract ▾
All authors Ruichao Hou, Boyue Xu, Tongwei Ren, Dongming Zhou, Gangshan Wu, Jinde Cao
Computer Vision
NGS-Marker: Robust Native Watermarking for 3D Gaussian Splatting
Hao Qin, Yukai Sun, Luyuan Chen et al. (+5 more)
With the rapid development and adoption of 3D Gaussian Splatting (3DGS), the need for effective copyright protection has become increasingly critical. Existing watermarking techniques for 3DGS mainly focus on protecting rendered images via pre-trained decoders, leaving the underlying 3D Gaussian primitives vulnerable to misuse. In particular, they are ineffective against Partial Infringement, where an adversary extracts and reuses only a subset of Gaussians. In this paper, we propose NGS-Marker, a novel native watermarking framework for 3DGS. It integrates a jointly trained watermark injector and message decoder, and employs a gradientbased progressive injection strategy to ensure full-scene coverage. This enables robust ownership decoding from any local region. We further extend NGS-Marker with hybrid protection (combining native and indirect watermarks) and support for
multimodal watermarking. Extensive experiments demonstrate that NGS-Marker effectively defends against partial infringement while offering practical flexibility for real-world deployment.
Read the full abstract ▾
All authors Hao Qin, Yukai Sun, Luyuan Chen, Mengxu Lu, Feng Zhang, Ming Kong, Zhenhong Du, Qiang Zhu
Computer Vision
Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs
Yifan Lu, Adinath Dukre, Abhijit Das et al. (+4 more)
Medical
vision-
language models (Med-VLMs) have demonstrated strong performance on medical visual question answering, yet they remain prone to
hallucination, generating clinically unsupported statements that are insufficiently grounded in image evidence. Mitigation methods applied during decoding offer a practical solution, but they typically lack anatomical awareness or rely heavily on ground truth annotations, which limits their applicability. We propose Counterfactual Anatomy-guided Spatial-Temporal decoding (CAST), a framework that operates entirely during inference and requires no manual annotations for anatomically grounded
hallucination mitigation. CAST automatically discovers anatomical regions relevant to the given query through broad medical
segmentation. It then selects a compact, causally informative area using counterfactual intervention based on the drop in answer likelihood under occlusion. Guided by this chosen region, CAST performs a unified contrastive decoding process, combining classifier-free guidance to correct spatial
attention with stepwise temporal contrast to regulate generation dynamics. Experiments on the SLAKE and MIMIC-CXR
datasets across three Med-VLMs demonstrate that CAST consistently outperforms strong baselines and surpasses decoding strategies reliant on ground truth. Our results indicate that compact, automatically selected regions provide highly effective contrastive guidance without expert annotations, offering a practical and generalizable solution for improving spatial grounding and reducing
hallucinations. Code is available at https://github.com/csyifan/CAST.
Read the full abstract ▾
All authors Yifan Lu, Adinath Dukre, Abhijit Das, Ziyun Zou, Haolin Yang, Yutong Xie, Imran Razzak
Computer Vision
SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation
Keyu Tu, Zhuowei Chen, Mengqi Huang et al. (+4 more)
We introduce Semantic Task Completion
Video Generation, an outcome-oriented
video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation
dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric
video clip. A scalable four-stage curation pipeline converts raw
videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a
vision-
language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative
video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.
Read the full abstract ▾
All authors Keyu Tu, Zhuowei Chen, Mengqi Huang, Yuxin Wang, Jiahao Zhu, Zhendong Mao, Yongdong Zhang
Computer Vision
GSToken: Geometry-Structured Gaussian Tokens for Compact 3D Medical Image Representation
Xiaoduo Li and Quan Gu
Effective
segmentation of multi-modal MRI is central to improving neural network accuracy in
brain tumor recognition. Existing methods typically compress 3D volumes into
token sequences via fixed patch encoding or learned
attention pooling (e.g., TokenLearner). However, these compression schemes discard explicit spatial shape information; the resulting
tokens convey no notion of lesion morphology or spatial extent. Meanwhile, end-to-end evaluation entangles a tokenizer's information retention with the reconstruction capacity of the downstream decoder, and the lack of a unified capacity contract across methods makes performance differences difficult to attribute. In this paper, we introduce Gaussian
tokens to multi-modal
brain tumor
segmentation for the first time: each
token carries not only a semantic feature but also a learned 3D center, anisotropic scale, and orientation, endowing the representation with explicit
geometric support at negligible parameter cost. We further propose a frozen-
token utility evaluation protocol: the trained tokenizer is frozen, its output is cast into a fixed-capacity serialized contract, and a shared lightweight
Transformer probe independently measures each tokenizer's retained information under strictly matched conditions. Multi-seed paired statistical testing shows that GSToken consistently and substantially outperforms capacity-matched adaptive baselines under frozen probing, with uniform advantages across all tumor sub-regions, surface, and distance metrics. These results demonstrate that explicitly encoding spatial geometry within
tokens significantly improves the information density of volumetric representations, offering a new design principle for compact 3D medical image representation and downstream reading.
Read the full abstract ▾
Computer Vision
TF-CADE: Foreground-Concentrated Text-Video Alignment for Zero-Shot Temporal Action Detection
Yearang Lee, Ho-Joong Kim and Seong-Whan Lee
Zero-Shot Temporal Action Detection (ZSTAD) aims to lo- calize and recognize action instances from unseen action categories in untrimmed
videos. Although existing meth- ods have shown effectiveness by advancing architectural text-
video alignment, they still struggle with capturing se- mantic distinctions between action classes, resulting in text- irrelevant predictions. To address this issue, we propose a Text-Foreground Concentrated
Alignment for zero-shot temporal action DEtector (TF-CADE) that explicitly aligns textual information with action-relevant foreground regions. Specifically, we introduce Action Concentrate Aggregation (ACA), which extracts action concentrate scores to aggregate temporally informative
video segments into a foreground- weighted
video embedding. This foreground concentrated
alignment enhances the semantic consistency between text and
video features and improves inter-class discriminabil- ity. In addition, a Certainty-based Confidence Re-weighting (CCR) strategy refines per-snippet confidence scores by lever- aging foreground-aware similarity, effectively suppressing irrelevant action classes during inference. Extensive evalua- tions show that our TF-CADE not only achieves state-of-the- art performance under in-distribution settings but also excels in cross-
dataset generalization to unseen action classes.
Read the full abstract ▾
Computer Vision
TEAMS: Text-prompted spatiotEmporal dual-heAd Mamba Snake
Ruicheng Zhang, Jianhui Lei, Kaiwen Shen et al. (+6 more)
Deep snake is a promising family of instance
segmentation methods that accurately predicts object-level contours, thereby overcoming common pixel-level misclassification issues such as mask cavities and jagged edges in semantic
segmentation approaches. However, existing deep snake methods face challenges in handling complex morphological variations, accurately capturing fine-grained organ details, and correcting base detection errors. To mitigate these limitations, we propose a cohesive Text-prompted spatiotEmporal dual-heAd Mamba Snake (TEAMS), a novel
vision-
language Mamba snake framework with three key innovations: (1) A Spatiotemporal Snake
Evolution Strategy (SSES) is introduced to tackle complex morphological variations by capturing bidirectional spatial dependencies along the snake contour and temporal dynamics across
evolution steps in a state space model. (2) A Contour Morphology-Aware Mamba (CMAM) is proposed to quantify local contour morphologies to modulate the structured
attention mask in the Mamba2 SSD dual form, which extends Mamba's capability to perceive the relative importance of its input sequence elements for better delineation of fine-grained organ details. (3) A Text-prompted Collaborative Dual-Head Snake (TCDHS) is designed to incorporate cues from textual
prompts and transfer the evolved contour information to the base detection head, which enhances the deep snake workflow and mitigates wrong detections. Comprehensive evaluations on five
datasets covering different organs and imaging modalities demonstrate that TEAMS outperforms existing semantic and deep snake
segmentation methods (e.g., relative mDice/mBF improvements of 6.9%/9.1% in a spinal
dataset), underscoring its potential as a reliable tool across diverse medical image
segmentation scenarios.
Read the full abstract ▾
All authors Ruicheng Zhang, Jianhui Lei, Kaiwen Shen, Haowei Guo, Jun Zhou, Bin Chen, Mengtang Li, Shen Zhao, Shuo Li
Computer Vision
SPVC: Structured and Panoptic Video Fixing for Cross-Dataset Driving Scene Rendering
Gen Li, Shu Han, Yun Xi Qiao et al. (+5 more)
Driving scene reconstruction and rendering, especially with 3D Gaussian Splatting, has become an important component of autonomous driving simulation. However, rendered views often degrade under extrapolated ego trajectories and scene edits, producing blurry structures, temporal flicker, and foreground-background misalignment. Existing refinement methods are commonly designed for a specific setting, such as image-level novel-view repair or object-editing correction. In this paper, we introduce SPVC, a structured and panoptic
video fixing framework for cross-
dataset driving scene rendering. The name summarizes four design principles. (1) Structured fixing denotes the use of explicit spatial conditions, including camera pose, 3D bounding boxes, and HD maps, to guide the repair process and reduce uncontrolled
hallucination. (2) Panoptic fixing refers to correcting both background rendering artifacts, such as distorted roads, buildings, and lanes, and foreground vehicle artifacts introduced by scene editing, such as inconsistent object appearance. (3)
Video fixing means that the model operates on driving sequences rather than isolated frames, allowing temporal cues to be used during artifact correction. (4) Cross-
dataset fixing means that a single shared network is trained and applied across multiple driving
datasets, reducing the need for
dataset-specific or scene-specific fixers. Concretely, we construct paired degraded-clean training data by simulating under-constrained 3DGS rendering and foreground vehicle insertion artifacts, and train a two-stage controllable
video diffusion model that first addresses
video-level appearance and then refines scene layout with structured controls.
Read the full abstract ▾
All authors Gen Li, Shu Han, Yun Xi Qiao, Hua Chen, Xuyang Dai, Bohan Li, Hao Zhao, Chaojian Li
Computer Vision
Spectral Gradient Orthogonalization Improves Differentially Private Training at Scale
Sabari Shanmugam, Nick Barnes and Kerry Taylor
Differentially private training adds isotropic Gaussian noise to clipped gradients, corrupting every singular direction equally. In
vision models, where spatial correlation concentrates gradient energy into a low-rank subspace, most of this noise falls in directions that carry little signal. Spectral gradient orthogonalization via polar decomposition is introduced as a post-processing step that recovers directional signal from the noisy gradient's low-rank structure at zero additional
privacy cost. A phase transition governs the utility of this approach: orthogonalization improves accuracy only when the per-direction spectral signal-to-noise ratio (SNR) suffices for singular vector recovery; in low-SNR regimes, the directional bias of the gradient is replaced by a nearly random orthogonal update, and the transformation is harmful. The recovery threshold is determined by the spectral gap of the gradient and is surpassed at large batch sizes. Empirically, the benefit scales with model capacity: spectral orthogonalization achieves a +20.9% improvement over DP-SGD on WRN-28-10 (B = 4096) and +14.9% on ResNet-18, while reducing inter-run variance by a factor of two to three. In the fine-tuning regime, spectral orthogonalization matches the stability of DP-Adam while maintaining a first-order
memory footprint. Combining spectral with temporal denoising yields 50.3% on CIFAR-10 (epsilon = 4), the highest accuracy in any tested configuration. These gains are specific to moderate-to-high-SNR regimes such as large-batch training of higher-capacity models. Small-batch or low-SNR settings are better served by DP-SGD or temporal denoising.
Read the full abstract ▾
Computer Vision
REChart: Reasoning-Efficient Chart Editing with Large Reasoning Models
Yuanbang Liu, Chenxi Ruan, Yihan Hou et al. (+2 more)
Chart editing requires inferring and modifying visualization code from a reference chart image based on an editing instruction, challenging fine-grained visual
reasoning, instruction following, and executable code synthesis capabilities of MLLMs. Large
reasoning models (LRMs) with extended Chain-of-Thought (CoT)
reasoning are suitable for tackling such complex
multimodal tasks. However, our preliminary study reveals an “inverted-U” relationship between
reasoning length and chart-editing performance: Excessive
reasoning often leads to “overthinking,” where models drift toward hallucinated visual details or get stuck in redundant
reasoning loops. To address the gap, we introduce REChart, a two-stage training framework that provides process-level supervision over intermediate
reasoning steps, improving both editing fidelity and
reasoning efficiency. First, we synthesize 200k high-quality
reasoning trajectories for supervised fine-tuning from a large image-instruction-code pool, using a role-specialized agentic Reason-Score-Refine workflow that iteratively refine the chart code toward higher quality. Second, we optimize the model via reinforcement learning with two complementary rewards: a fidelity reward evaluating code correctness, visual fidelity, and structural consistency, and an efficiency reward that assigns each rollout a random thinking budget, truncates the
reasoning process, and credits the final
reasoning segment according to its contribution to the output. On the ChartEdit and ChartMIMIC
benchmarks, our model achieves state-of-the-art chart-editing performance among open-source models of comparable scale, while mitigating overthinking and reducing average
reasoning token usage by 79.0% under a maximum thinking budget of 16,384
tokens compared with the base model.
Read the full abstract ▾
All authors Yuanbang Liu, Chenxi Ruan, Yihan Hou, Qiong Luo, Wei Zeng
Computer Vision
MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding
Bonan Zhang, Shiyu Dong, Quan Hung Tran et al. (+9 more)
Vision encoders are a critical component of
vision-
language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style
vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for
vision encoder scaling and find that fine-grained MoE
topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance
video capabilities while preserving image knowledge, we introduce frame-level
distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts
Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and
video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.
Read the full abstract ▾
All authors Bonan Zhang, Shiyu Dong, Quan Hung Tran, Katharina Gschwind, Shuqi Yang, Sijia Chen, Adel Ahmadyan, Seungwhan Moon, Lu Zhang, Ahmed Kirmani, Babak Damavandi, Anuj Kumar
Computer Vision
To Remove or Not to Remove Clouds: A Comparative Analysis and Fusion of Raw SAR and Synthetic NDWI for Overcast Water Segmentation
Saleh Sakib Ahmed, Sara Nowreen and M. Sohel Rahman
Persistent clouds blind optical satellites during floods. While Synthetic Aperture Radar (SAR) penetrates clouds, its raw data is noisy and lacks clear contrast. To mitigate this, recent studies utilize deep learning models to translate SAR into cloud-free synthetic optical imagery for downstream tasks like water body
segmentation. However, because raw SAR is the original source for both of these operations, a critical methodological dilemma arises: during complete overcast should
segmentation models process the raw SAR directly, or rely on a translated synthetic Normalized Difference Water Index (NDWI) proxy? This study resolves the debate by demonstrating that synthetic NDWI yields better results, as the translation process acts as a powerful filter against radar noise. This raises a natural second question: what if we utilize both? Building on our findings, we introduce a Combined Framework that integrates both raw SAR and synthetic NDWI into a unified model. By fusing the sharp physical boundaries of raw SAR with the high contrast of synthetic NDWI, this hybrid approach consistently outperforms all standalone methods.
Read the full abstract ▾
Computer Vision
Noisy group neurons with synchronous resetting for high-performance spiking neural networks
Yajie Zhai, Yanmei Kang, Meng Li et al. (+1 more)
Spiking neural networks (SNNs), characterized by bio-inspired neuronal dynamics and event-driven communication, have attained significant progress in recent years. Nevertheless, training deep SNNs remains challenging due to spatiotemporal information loss and gradient mismatching. To simultaneously address these issues, we propose a noisy group
neuron (NGN) model, which incorporates population-level synchronous resetting and neural stochasticity as fundamental computational mechanisms. We then develop the NGN method as a framework that combines the NGN model with backpropagation learning based on mean-field dynamics. We demonstrate the advantages of the NGN method through theoretical analysis and experimental validation on CIFAR-10, CIFAR-100, Tiny-ImageNet, DVS-Gesture, N-Caltech101, and CIFAR10-DVS. The proposed approach achieves an accuracy of 87.35% on CIFAR10-DVS within 10 inference time steps. These results support NGN as a practical approach to high-performance neuromorphic computing.
Read the full abstract ▾
All authors Yajie Zhai, Yanmei Kang, Meng Li, Zigang Huang
Computer Vision
GeoWeaver: Accurate Long-Sequence 3D Reconstruction via Hierarchical Geometric Assembly
Tinghao Jiang, Sheng Tang, Shengzhe Wei et al. (+4 more)
Long-sequence 3D reconstruction from RGB
videos requires both accurate local geometry and globally consistent camera motion. Feed-forward models provide strong depth and pose predictions, but their
memory cost prevents joint inference over long sequences. Chunk-wise processing improves scalability, yet independently predicted chunks often exhibit scale drift, pose errors, and point-cloud misalignment. We present GeoWeaver, a unified framework comprising a
Geometric Prior Model (GPM) and Test-Time Adaptation (TTA). The GPM predicts chunk-wise depth, confidence, and camera parameters as adjustable
geometric priors. TTA then performs sequential initialization, global chunk-level Sim(3)
alignment, and coarse-to-fine refinement of camera poses, affine depth corrections, and intrinsics. Dense correspondences provide adjacent, cross-chunk, and long-range constraints, while a robust CDF-style objective jointly optimizes weighted 2D reprojection and 3D consistency residuals. This design preserves local
geometric accuracy while correcting accumulated pose, scale, depth, and calibration errors. Experiments across diverse long-sequence
benchmarks demonstrate improved camera accuracy, global consistency, and point-cloud quality. Ablations verify the contribution of each adaptation stage, and applying the same TTA procedure to different
geometric prior models consistently improves their trajectory estimates, demonstrating that GeoWeaver is not tied to a specific GPM.
Read the full abstract ▾
All authors Tinghao Jiang, Sheng Tang, Shengzhe Wei, Juntong Fang, Weiqi Zhang, Junsheng Zhou, Zesong Li
Computer Vision
Continuity-Driven Representation Learning for Industrial Defect Detection
Minjong Kim, Hyun Jun Kim, Jeongrae Kim et al. (+2 more)
Industrial defect detection differs from natural-image object detection because inspection images are captured under controlled conditions and contain large normal-dominant regions with repetitive structures. Defects therefore appear as localized disruptions of otherwise predictable patterns, while conventional detectors rely mainly on sparse bounding-box supervision, resulting in weakly constrained normal-region representations. We propose a continuity-driven representation regularization framework that exploits normal-dominant regions as dense auxiliary supervision. The framework introduces two detector-agnostic objectives: Multi-Continuity Loss, which combines 1D patch-sequence prediction and 2D masked spatial prediction, and Differencing Loss, which regularizes first-order feature variation and second-order
curvature between neighboring patch
embeddings. Both objectives are applied with box-derived region weighting to stabilize normal-region representations while preserving defect-related discontinuities. Experiments on two real-world industrial
datasets and the public NEU-DET
benchmark, using six detector architectures including YOLO-family models, MambaYOLO, and DETR, demonstrate consistent improvements over native detector baselines. In the full-data setting, the proposed regularizers improve average mAP@0.5:0.95 by up to 3.49 percentage points on Industrial Metal, 5.38 percentage points on MEA, and 5.03 percentage points on NEU-DET. Under limited-data conditions, the gains become more pronounced, with Differencing Loss achieving improvements of up to 21.07 percentage points in mAP@0.5 and 8.23 percentage points in mAP@0.5:0.95 on NEU-DET using only 25% of the training data. These results suggest that continuity-driven regularization provides an effective prior for improving industrial defect detection, particularly when annotated data are scarce.
Read the full abstract ▾
All authors Minjong Kim, Hyun Jun Kim, Jeongrae Kim, Heeseung Shin, Changwon Lim
Computer Vision
Primitive-Driven Compositional Forensic Visual Prompting for Open-World Face Anti-Spoofing
Fangling Jiang, Qi Li, Bing Liu et al. (+4 more)
Open-world face anti-spoofing must address both
covariate and semantic shifts: source and target domains differ in imaging conditions, while target domains contain diverse attack types absent from training. Existing
prompt-based approaches often express spoofing through category semantics or
language guidance, which is effective for modeling high-level concepts but is less suited to explicitly capturing the evolving fine-grained and spatially heterogeneous forensic evidence of unseen attacks. Motivated by the hypothesis that many unseen attacks can be characterized by new combinations of recurring visual cues, we propose a compositional forensic visual
prompt learning framework that operates entirely in the visual feature space. Built on a frozen ViT-based
vision foundation model, the framework employs patch-aware
attention to refine a shared set of learnable micro-forensic primitives into localized forensic evidence units derived from image patches. Class-specific global contextual
prompts then provide input-dependent routing weights that adaptively select and compose these primitives into compositional forensic visual
prompts for real/spoof discrimination. The primitives are not assigned predefined semantic meanings; instead, their specialization and reuse emerge from shared parameterization and joint optimization across categories. Extensive experiments on nine open-world protocols demonstrate state-of-the-art performance, strong cross-domain generalization, and robust adaptation to unseen attacks.
Read the full abstract ▾
All authors Fangling Jiang, Qi Li, Bing Liu, Weining Wang, Quilin Huang, Zhenan Sun, Ming-Hsuan Yang
Computer Vision
Learning latent progression states from spatial heterogeneity in uterine histopathology
Qiming He, Yan Liu, Shuang Ge et al. (+20 more)
Tumor progression is accompanied by changes in architecture, morphology and microenvironmental organization, yet progression-associated heterogeneity is usually compressed into static diagnostic categories in histopathology. Here we present SpaTIE, a uterus-specific computational pathology framework that learns morphology-aware representations and organizes spatial histopathological heterogeneity into progression-associated tumor states. SpaTIE was developed using 10,426 uterine hematoxylin and eosin whole-slide images and evaluated in TCGA-UCEC and TCGA-UCS cohorts. The learned representations formed morphology
manifolds, supported diagnostic, molecular and survival-related prediction tasks, and localized
attention to informative tumor regions. Beyond supervised prediction, SpaTIE inferred tumor-state axes from cross-sectional morphology without temporal or molecular supervision. These morphology-derived states were spatially coherent and showed associations with clinicopathological variables and survival outcomes, while not simply recapitulating staging or diagnostic labels. Integrative multi-omics analyses linked the inferred states to DNA methylation, somatic copy-number variation, mutation, RNA-seq and RPPA profiles, highlighting molecular programs related to chromatin regulation, copy-number-associated structural variation, receptor tyrosine kinase signaling,
cell adhesion, extracellular-
matrix remodeling and metabolic adaptation. Progression-guided virtual perturbation further prioritized molecular features coupled to the morphology-derived state organization. Together, these findings suggest that uterine histopathology contains recoverable progression-associated tumor-state information and establish SpaTIE as a framework for connecting spatial morphology with multi-omics-informed tumor-state discovery.
Read the full abstract ▾
All authors Qiming He, Yan Liu, Shuang Ge, Fan Yang, Yuxiang Wang, Ieng Man Zhang, Jing Yang, Zihao Jia, Ajin Hu, Yexing Zhang, Zixiu Song, Qiang Huang, Xiaoya Zhao, Zihan Wang, Xianjing Zheng, Yijun Zheng, Liling Lin, Shuxing Liu, Bin Bao, Yue Xie, Tian Guan, Yonghong He, Congrong Liu