Computer Vision
Mutual Enhancement Between Global Tokens and Patch Tokens: From Theory to Practice
Xiusheng Huang, Xin Jiang, Jun Zhao et al. (+2 more)
Accurate and effective discrete image tokenization is crucial for long image sequence processing. However, current methods rigidly compress all content at a fixed rate, ignoring the variable information density of images and leading to either redundancy or information loss. Inspired by information
entropy, we propose TaTok, a Theoretically grounded adaptive image Tokenization framework. We rigorously identify two key drawbacks in existing methods: information insufficiency when reconstructing images with patch
tokens alone, and information redundancy among patch
tokens. To address these, we introduce global
tokens that model mutual information across patch
tokens, and a Dynamic
Token Filtering (DTF)
algorithm based on cumulative conditional
entropy to eliminate redundancy. Experiments confirm TaTok's state-of-the-art performance, delivering a 1.3x gFID improvement and 8.7x inference speedup. By allocating
tokens according to information richness, TaTok enables more compressed yet accurate image tokenization, offering valuable insights for future research.
Read the full abstract ▾
All authors Xiusheng Huang, Xin Jiang, Jun Zhao, Kang Liu, Yequan Wang
Computer Vision
A neurosymbolic Approach with Epistemic Deep Learning for Hierarchical Image Classification
Ezel Kilicdere, Shireen Kudukkil Manchingal and Fabio Cuzzolin
Deep neural networks achieve high accuracy on image classification tasks. Yet, they often produce overconfident predictions as which fail to express epistemic uncertainty, and frequently violate logical or structural constraints present in the data. These limitations are particularly pronounced in hierarchical classification, where predictions across fine and coarse levels must remain coherent. We propose, for the first time, a unified neurosymbolic and epistemic modelling framework that augments Swin
Transformers with focal set
reasoning and differentiable fuzzy logic. Rather than treating labels as isolated categories, our method induces data-driven focal sets within the learnt
embedding space, which helps capture epistemic uncertainty over multiple plausible fine-grained classes. These focal sets form the basis of a belief-theoretic layer that uses fuzzy membership functions and t-norm conjunctions to encourage consistency between fine- and coarse-grained predictions. A learnable loss further balances calibration, mass regularisation, and logical consistency, allowing the model to adaptively trade off symbolic structure with data-driven evidence. In experiments on hierarchical image classification, our framework maintains accuracy on par with
transformer baselines while providing more calibrated and interpretable predictions, reducing overconfidence and enforcing high logical consistency across hierarchical outputs. Our experimental results show that combining focal set
reasoning with fuzzy logic provides a practical step toward deep learning models that are both accurate and epistemically aware.
Read the full abstract ▾
Computer Vision
StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video
Ao Li, Zihan Xiao, Zihao Yue et al. (+7 more)
Proactive streaming
video understanding requires models to continuously process
video streams and decide when to respond, rather than merely what to respond. This naturally introduces a decision-making problem under partial observations, where models must balance early prediction against sufficient evidence. However, existing
benchmarks largely follow a "see-then-answer" paradigm, where responses are triggered only after explicit evidence appears, effectively reducing proactive
reasoning to delayed perception. As a result, they fail to evaluate a model's ability to make timely and reliable decisions under incomplete observations. Moreover, training proactive models is inherently challenging due to the extreme imbalance between silence and response signals in streaming trajectories, as well as the need to jointly optimize response correctness and timing. To address these challenges, we introduce StreamPro-Bench, a new
benchmark that evaluates streaming models from three complementary perspectives: Perception Understanding, Temporal
Reasoning, and Proactive Agency, where the last measures a model's ability to make early yet reliable decisions under partial observations. We further propose StreamPro, a two-stage training framework for proactive learning. First, we introduce CB-Stream Loss to mitigate the severe supervision imbalance during supervised fine-tuning (SFT). Then, we apply Group Relative Policy Optimization (GRPO) with a multi-grained reward design that involves both turn-level and trajectory-level rewards. Experiments show that StreamPro significantly improves proactive performance. On StreamPro-Bench, it achieves 41.5, substantially outperforming the previous best (10.4), while also maintaining strong performance on real-time streaming
benchmarks, achieving 78.9 on StreamingBench-RTVU.
Read the full abstract ▾
All authors Ao Li, Zihan Xiao, Zihao Yue, Boshen Xu, Linli Yao, Jiaze Li, Pei Fu, Jianzhong Ju, Jian Luan, Qin Jin
Computer Vision · Cross-listed from eess.IV
Brain Tumor Classification in MRI Images: A Computationally Efficient Convolutional Neural Network
Md Fahimul Kabir Chowdhury and Jannatul Ferdous
Improving patient outcomes depends on the
prompt and accurate diagnosis of
brain tumors, but manual MRI scan analysis is still time-consuming and unreliable. Although deep learning has shown promise, many of the models that are now in use are computationally intensive and have difficulty handling the intrinsic complexity and variety of different types of
brain tumors. In this work, we propose a lightweight yet high-performing Convolutional Neural Network (CNN) for multi-class
brain tumor classification, employing MRI images to target gliomas, meningiomas, pituitary tumors, and healthy (no tumor) instances. The model was rigorously evaluated on two publicly accessible
datasets from Figshare and Kaggle. Leveraging efficient feature extraction and optimized training strategies, our CNN achieved classification accuracies of 99.03% and 99.28%, along with ROC scores of 99.88% and 99.94% on
Dataset 1 and
Dataset 2, respectively-all while utilizing significantly fewer parameters than popular pre-trained architectures. In contrast to cutting-edge models like DenseNet201, MobileNetV2, VGG19, Xception, InceptionV3, and ResNet50, our approach consistently demonstrated superior performance with reduced computational overhead. These findings highlight the potential of the proposed model as a practical and reliable diagnostic aid in clinical environments.
Read the full abstract ▾
Computer Vision
M2Retinexformer: Multi-Modal Retinexformer for Low-Light Image Enhancement
Youssef Aboelwafa, Hicham G. Elmongui and Marwan Torki
Low-light image enhancement is challenging due to complex degradations, including amplified noise, artifacts, and color distortion. While Retinex-based deep learning methods have achieved promising results, they primarily rely on single-modality RGB information. We propose M2Retinexformer (Multi-Modal Retinexformer), a novel framework that extends Retinexformer by incorporating depth cues, luminance priors, and semantic features within a progressive refinement pipeline. Depth provides
geometric context that is invariant to lighting variations, while luminance and semantic features offer explicit guidance on brightness distribution and scene understanding. Modalities are extracted at multiple scales and fused through cross-
attention, with adaptive gating dynamically balancing illumination-guided self-
attention and cross-
attention based on the reliability of auxiliary cues. Evaluations on the LOL, SID, SMID, and SDSD
benchmarks demonstrate overall improvements over Retinexformer and recent state-of-the-art methods. Code and pretrained weights are available at https://github.com/YoussefAboelwafa/M2Retinexformer
Read the full abstract ▾
Computer Vision
Quantifying Rodda and Graham Gait Classification from 3D Markerless Kinematics derived from a Single-view Video in a Heterogeneous Pediatric Clinical Cohort
Lauhitya Reddy, Seth Donahue, Jeremy Bauer et al. (+8 more)
Cerebral Palsy (CP) is a neurological disorder of movement and the most common cause of lifelong physical disability in childhood. Approximately 75% of children with CP are ambulatory, and accurate gait assessment is central to preserving walking function, which deteriorates by mid-adulthood in a quarter to half of adults with CP. The Rodda and Graham classification system quantifies sagittal-plane gait deviations using ankle and knee z-scores derived from 3D Instrumented Gait Analysis (3D-IGA), but 3D-IGA is expensive and limited to specialized centers, while observational assessment shows only moderate inter-rater agreement. We developed a markerless gait analysis pipeline that quantifies Rodda and Graham knee and ankle z-scores directly from single-view clinical gait
videos. Across 1,058 bilateral limb samples from 529 trials of 152 children (88 male, 63 female; age 12.1 $\pm$ 4.0 years; 60 distinct primary diagnoses, cerebral palsy the most common at $n=54$), the sagittal-view model achieved $R^2 = 0.80 \pm 0.02$ and CCC $= 0.89 \pm 0.02$ for knee z-scores and $R^2 = 0.57 \pm 0.02$ and CCC $= 0.72 \pm 0.02$ for ankle z-scores against 3D-IGA. Binary screening for excess knee flexion achieves AUROC $= 0.88$, correctly identifying 83% of affected children, and applying Rodda and Graham rules yields $43 \pm 1$% 7-class accuracy with macro-AUROC $= 0.78 \pm 0.01$, ankle prediction error remaining the primary bottleneck. Beyond cross-sectional screening, continuous z-scores support longitudinal trajectory tracking across visits, providing a quantitative substrate for monitoring disease progression and treatment response unavailable from observational scales. These results demonstrate the feasibility of
video-based z-score estimation, excess-flexion screening, and longitudinal trajectory tracking as a path toward scalable, objective gait assessment in low-resource clinical settings.
Read the full abstract ▾
All authors Lauhitya Reddy, Seth Donahue, Jeremy Bauer, Susan Sienko, Anita Bagley, Joseph Krzak, Maura Eveld, Karen Kruger, Ross Chafetz, Vedant Kulkarni, Hyeokhyen Kwon
Computer Vision · Cross-listed from cs.LG
Couple to Control: Joint Initial Noise Design in Diffusion Models
Jing Jia, Liyue Shen and Guanyang Wang
Diffusion models typically generate image batches from independent Gaussian initial noises. We argue that this independence assumption is only one choice within a broader class of valid joint noise designs. Instead, one can specify a coupling of the initial noises: each noise remains marginally standard Gaussian, so the pretrained
diffusion model receives the same single-sample input distribution, while the dependence across samples is chosen by design. This reframes initial-noise control from selecting or optimizing individual seeds to designing the dependence structure of a multi-sample gallery. This view gives a general framework for initial-noise design, covering several existing methods as special cases and leading naturally to new coupled-noise constructions. Coupled noise can improve generation on its own without adding sampling cost, and it is flexible enough to serve as a structured initialization for optimization-based pipelines when additional computation is available. Empirically, repulsive Gaussian coupling improves gallery diversity on SD1.5, SDXL, and SD3 while largely preserving
prompt alignment and image quality. It matches or outperforms recent test-time noise-optimization baselines on several diversity metrics at the same sampling cost as independent generation. Subspace couplings also support fixed-object background generation, producing diverse, natural backgrounds compared with specialized inpainting baselines, with a tunable trade-off in foreground fidelity.
Read the full abstract ▾
Computer Vision
Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation
Ajay Vikram Periasami, Junlin Wang and Bhuwan Dhingra
We study whether
vision-
language models (VLMs) can reconstruct structured, graphics-based images by writing executable code to generate them. Existing
benchmarks on image-to-code generation are generally limited in the domains they cover, rely on reference source code for evaluation, and do not provide metrics to quantify domain-specific issues. To address this, we introduce
Vision2Code, a multi-domain, reference-code-free
benchmark containing 539 test examples from 15 different
datasets, spanning charts and plots, geometry, graphs, scientific diagrams, documents and 3D spatial scenes. Evaluation is done by executing the generated programs and comparing their rendered outputs to the source images using
dataset- and domain-specific rubrics for a VLM judge. We find that the final scores obtained in this manner have substantially higher correlation with human ratings than those obtained via generic VLM rubrics or standard pixel-based,
embedding-based or perceptual similarity metrics. We further demonstrate the reliability of our evaluation under repeated sampling from the VLM judges, varying rubric weights, and in the presence of controlled errors. Our results highlight the variability across domains of existing image-to-code generation models, with accuracy on charts and graphs being significantly higher than other domains. Lastly, we demonstrate the utility of our scoring method for rejection sampling of generated programs for fine-tuning, improving Qwen3.5-9B by 16% and Phi-4-MM by 28% compared to their base checkpoints. Our code and data are publicly available at https://image2code.github.io/
vision2code/.
Read the full abstract ▾
Computer Vision
CheXTemporal: A Dataset for Temporally-Grounded Reasoning in Chest Radiography
Eva Prakash, Yunhe Gao, Chong Wang et al. (+10 more)
Chest radiograph interpretation requires temporal
reasoning over prior and current studies, yet most
vision-
language models are trained on static image-report pairs and lack explicit supervision for modeling longitudinal change. We introduce CheXTemporal, a
dataset for temporally grounded
reasoning in chest radiography consisting of paired prior-current chest X-rays (CXR) with finding-level temporal and spatial annotations. The
dataset includes a five-class progression taxonomy (new, worse, stable, improved, resolved), localized spatial supervision of pathology, explicit spatial-temporal
alignment across paired studies, and multi-source coverage for cross-domain evaluation. We additionally construct a 280K-pair silver
dataset with automatically derived temporal and anatomical supervision for large-scale evaluation under weaker supervision. Using these resources, we evaluate multiple state-of-the-art
vision-
language CXR models on grounding and progression-classification tasks in a zero-shot setting. Across both gold and silver evaluations, current models exhibit consistent limitations in spatial grounding, fine-grained temporal
reasoning, and robustness under distribution shift. In particular, models perform substantially better on salient progression categories such as worse than on temporally subtle states such as stable and resolved, suggesting limited modeling of longitudinal disease
evolution in chest radiography.
Read the full abstract ▾
All authors Eva Prakash, Yunhe Gao, Chong Wang, Justin Xu, Neal Prakash, Arne Michalson, Seena Dehkharghani, Eun Kyoung Hong, Julie Bauml, Roger Boodoo, Jean-Benoit Delbrouck, Sophie Ostmeier, Curtis Langlotz
Computer Vision · Cross-listed from cs.AI
LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?
Xueqi Cheng and Yushun Dong
Multimodal large
language models (MLLMs) have heterogeneous strengths across OCR, chart understanding, spatial
reasoning, visual question answering, cost, and latency. Effective MLLM routing therefore requires more than estimating query difficulty: a router must match the
multimodal requirements of the current image-question input with the capabilities of each candidate model. We propose LatentRouter, a router that formulates MLLM routing as counterfactual
multimodal utility prediction. Given an image-question query, LatentRouter extracts learned
multimodal routing capsules, represents each candidate MLLM with a model capability
token, and performs latent communication between these states to estimate how each model would perform if selected. A distributional outcome head predicts model-specific counterfactual quality, while a bounded capsule correction refines close decisions without allowing residual signals to dominate the prediction. The resulting utility-based policy supports performance-oriented and performance-cost routing, and handles changing candidate pools through shared per-model scoring with availability masking. Experiments on MMR-Bench and VL-RouterBench show that LatentRouter outperforms fixed-model, feature-level, and learned-router baselines. Additional analyses show that the gains are strongest on
multimodal task groups where model choice depends on visual, layout-sensitive, or
reasoning-oriented requirements, and that latent communication is the main contributor to the improvement. The code is available at: https://github.com/LabRAI/LatentRouter.
Read the full abstract ▾
Computer Vision
Can Graphs Help Vision SSMs See Better?
Dhruv Parikh, Anvitha Ramachandran, Haoyang Fan et al. (+3 more)
Vision state space models inherit the efficiency and long-range modeling ability of Mamba-style selective scans. However, their performance depends critically on the representation of two-dimensional visual features as one-dimensional
token sequences. Existing scan operators range from predefined
geometric traversals to dynamic coordinate-based samplers that reroute
tokens through predicted offsets and interpolation. While effective, these mechanisms primarily adapt paths or sampling locations, rather than explicitly modeling which local patches should exchange information before global state-space mixing. This motivates a simple question: can graphs help
vision state space models see better? We introduce GraphScan, a graph-induced dynamic scanning operator for
Vision SSMs. For each
token, GraphScan constructs a spatially bounded local graph, learns feature-conditioned affinities with relative positional bias, and produces the output
token by one-step message passing over its semantic neighborhood. The resulting
tokens are locally grounded before being processed by the selective SSM for global aggregation. GraphScan preserves
token count and linear scaling in image size, while replacing coordinate-conditioned interpolation with feature-conditioned semantic routing. Integrated into a hierarchical backbone, GraphScan-Mamba achieves state-of-the-art performance among
Vision SSMs across image classification, object detection, instance
segmentation, and semantic
segmentation, with modest computational overhead. Our analysis further shows that GraphScan induces interpretable displacement fields over the
token lattice, providing a semantic and spatially grounded view of dynamic scanning. These results suggest that future
Vision SSMs should treat scanning not merely as
geometric serialization, but as learned local semantic routing before global state-space modeling.
Read the full abstract ▾
All authors Dhruv Parikh, Anvitha Ramachandran, Haoyang Fan, Mustafa Munir, Rajgopal Kannan, Viktor Prasanna
Computer Vision
Generative AI for Visualizing Highway Construction Hazards Through Synthetic Images and Temporal Sequences
Trevor Neece, Mason Smetana and Lev Khazanovich
Highway construction workers face a high risk of serious injury or death. Image-based training materials depicting hazardous scenarios are essential for engaging safety instruction but remain scarce due to ethical and logistical barriers. This study develops and evaluates a generative AI methodology for producing synthetic visualizations of highway construction hazards from OSHA Severe Injury Report narratives. Two modes were developed: a single-pass approach yielding one image per incident, and a temporal approach producing a four-stage sequence. A sample of 75 incident records yielded 750 images, evaluated using CLIP-based semantic
retrieval and expert assessment across dimensions such as educational utility, fidelity, and
alignment. Single-pass images achieved 81.1% educational acceptability with fidelity and
alignment scores of 4.14/5 and 4.07/5, respectively, while temporal sequences achieved 60.9% acceptability with comparable
alignment (3.94/5) but lower fidelity (3.51/5). CLIP-based
retrieval revealed that both modes produce images with statistically significant
retrieval capabilities. This is among the first studies to leverage modern autoregressive image generation models for visualizing construction hazards from reported severe injuries and to generate temporally sequenced hazard imagery, and a new multi-dimensional evaluation framework was developed to support future research in this domain. The work enables safety trainers to pair narrative storytelling with visual learning material without photographing real-world hazards, and the framework could be applied to
datasets across diverse domains, enabling synthetic image generation tailored to new application areas.
Read the full abstract ▾
Computer Vision
Real-Scale Island Area and Coastline Estimation using Only its Place Name or Coordinates
Quanyun Wu, Kyle Gao, Wentao Sun et al. (+4 more)
Accurate measurement of island area and coastline length is crucial for coastal zone monitoring and oceanographic analysis. However, traditional measurement and mapping methods usually rely heavily on orthophotos, expensive airborne depth sensors, or dense ground control points, which face serious limitations of high labor costs, time-consuming efforts, and low operational efficiency in vast and inaccessible open sea environments. To overcome these challenges and break away from the reliance on manual field exploration, this paper proposes a geometrically consistent, real-scale island measurement framework based on pure
monocular vision. This project significantly reduces the mapping cost through a fully automated process and achieves high-efficiency measurement without prior GIS data. In our system pipeline, only the geographical coordinates or names of the target area need to be input to obtain a low-altitude surrounding image sequence. After obtaining the point clouds, a lightweight trajectory
alignment algorithm (Umeyama) is used to restore the global physical scale, and the scaled model is orthorectified, enabling high-precision area and perimeter extraction directly on the 2D rasterized plane. We have fully verified this pipeline on four islands with different terrain features (covering natural landform islands and islands with complex artificial facilities). The experimental results show that the final measurement error of the system is stable at around 10%, demonstrating excellent accuracy and robustness. Moreover, this framework has outstanding inference speed, requiring only 70 ms to process a single high-resolution image and generate point clouds, providing a highly practical new paradigm for large-scale marine and coastline
Read the full abstract ▾
All authors Quanyun Wu, Kyle Gao, Wentao Sun, Hongjie He, Yuhao Chen, David A. Clausi, Jonathan Li
Computer Vision
PG-3DGS: Optimizing 3D Gaussian Splatting to Satisfy Physics Objectives
Zachary Lee, Maxwell Jacobson and Yexiang Xue
Recent advances in Gaussian Splatting have enabled fast, high-fidelity 3D scene generation, yet these methods remain purely visual and lack an understanding of how shapes behave in the physical world. We introduce Physics-Guided 3D Gaussian Splatting (PG-3DGS), a framework that couples differentiable physics simulation with 3D Gaussian representations to generate 3D structures satisfying physics functionalities. By allowing physical objectives to guide the shape optimization process alongside visual losses, our approach produces geometries that are not only photometrically accurate but also physically functional. The model learns to adjust shapes so that the generated objects exhibit physically meaningful behaviors, for example, teapots that can pour and airplanes that can generate lift, without sacrificing visual quality. Experiments on pouring and aerodynamic lift tasks show that PG-3DGS improves physical functionality while preserving visual quality. In addition to simulation gains, bench-top physical lift tests with 3D-printed aircraft (Cessna, B-2 Spirit, and paper plane) under identical airflow conditions show higher scale-measured lift for PG-3DGS, generated structures than an appearance-matching baseline in all three cases. Our unified framework connects appearance-based reconstruction with physics-based
reasoning, enabling end-to-end generation of 3D structures that both look realistic and function correctly.
Read the full abstract ▾
Computer Vision
DenseTRF: Texture-Aware Unsupervised Representation Adaptation for Surgical Scene Dense Prediction
Guiqiu Liao, Matjaž Jogan and Daniel A. Hashimoto
Dense prediction tasks in
surgical computer
vision, such as
segmentation and
surgical zone prediction, can provide valuable guidance for laparoscopic and robotic surgery. However, these models often suffer from distribution shifts, as training
datasets rarely cover the variability encountered during deployment, leading to poor generalization. We propose DenseTRF, a self-supervised representation adaptation framework based on texture-centric
attention. Our method leverages slot
attention to learn texture-aware representations that capture invariant visual structures. By adapting these representations to the target distribution without supervision, DenseTRF significantly improves robustness to domain shifts. The framework is implemented through conditioning dense prediction on slot
attention and model merging strategies. Experiments across multiple
surgical procedures demonstrate improved cross-distribution generalization in comparison to state-of-the-art
segmentation models and test-distribution adaptation methods for dense prediction tasks.
Read the full abstract ▾
Computer Vision
ABRA: Agent Benchmark for Radiology Applications
Bulat Maksudov, Vladislav Kurenkov, Kathleen M. Curran et al. (+1 more)
Existing medical-
agent benchmarks deliver imaging as pre-selected samples, never as an environment the
agent must navigate. We introduce ABRA, a radiology-
agent benchmark in which the
agent operates an OHIF viewer and an Orthanc DICOM server through twenty-one function-calling tools that span slice
navigation, windowing, series selection, pixel-coordinate annotation, and structured reporting. ABRA contains 655 programmatically generated tasks across three difficulty tiers and eight types (viewer control, metadata QA,
vision probe, annotation, longitudinal comparison, BI-RADS reporting, and oracle variants of annotation and BI-RADS reporting), drawn from LIDC-IDRI, Duke Breast Cancer MRI, and NLST New-Lesion LongCT. Each episode is scored along Planning, Execution, and Outcome (Bluethgen et al., 2025) by task-type-specific automatic scorers. Ten current models, five closed-weight and five open-weight, reach at least 89% Execution on real annotation but only 0-25% Outcome; on the paired oracle variant where a simulated detector supplies the finding, Outcome on the same task reaches 69-100% across the models evaluated, localising the bottleneck to perception rather than tool orchestration. Code, task generators, and scorers are released at https://github.com/Luab/ABRA
Read the full abstract ▾
All authors Bulat Maksudov, Vladislav Kurenkov, Kathleen M. Curran, Alessandra Mileo
Computer Vision
Hi-GaTA: Hierarchical Gated Temporal Aggregation Adapter for Surgical Video Report Generation
Kedi Sun, Chaohui Dang, Yue Feng et al. (+3 more)
Automated, clinician-grade assessment reports for
surgical procedures could reduce documentation burden and provide objective feedback, yet remain challenging due to the difficulty of aligning dense spatio-temporal
video representations with
language-based
reasoning and the scarcity of high-quality,
privacy-preserving
datasets. To address this gap, we establish a
benchmark comprising 214 high-quality simulated
surgical videos paired with surgeon-authored evaluation reports. Building on this resource, we propose a Perception-
Alignment-
Reasoning framework for
surgical video report generation, featuring Hi-GaTA, a novel lightweight temporal adapter that efficiently compresses long
video sequences into compact, LLM-compatible visual prefix
tokens through short-to-long-range temporal aggregation. For robust visual perception, we pretrain Sur40k, a
surgical-specific ViViT-style
video encoder on 40,000 minutes of public
surgical videos to capture fine-grained spatio-temporal procedural priors. Hi-GaTA employs a temporal pyramid with text-conditioned dual cross-
attention, and improves multi-scale consistency through cross-level gated fusion and an increasing-depth strategy. Finally, we fine-tune the LLM backbone using LoRA to enable coherent and stylistically consistent
surgical report generation under limited supervision. Experiments show our approach achieves the best overall performance, with consistent gains over strong
Multimodal Large
Language Model (MLLM) baselines. Ablation studies further validate the effectiveness of each proposed component.
Read the full abstract ▾
All authors Kedi Sun, Chaohui Dang, Yue Feng, James Glasbey, Theodoros N. Arvanitis, Le Zhang
Computer Vision · Cross-listed from cs.LG
How Far Does a Shared Linear Map Go? Probing Feature-Space Manipulability for Image Editing
Elias Krey, Nils Neukirch and Nils Strodthoff
Understanding how image-space transformations manifest in a model's internal representations is a longstanding goal in representation analysis. Prior work has shown that
geometric transformations can often be captured by learned linear operators between feature maps, but it remains unclear whether this extends to photometric, local, and semantically defined edits. We train probes of increasing capacity from a spatially shared linear map to nonlinear per-vector, receptive-field, and global
transformer models to predict feature-space changes induced by
geometric transforms, photometric edits, occlusions, and
diffusion-generated semantic edits. Across ConvNeXt, SwinV2, and DINOv3, a single shared linear map often predicts held-out manipulation outcomes nearly as well as substantially more expressive probes for the supervised backbones, with sufficiency generally increasing with depth; this pattern is less consistent for DINOv3. These results suggest that a simple spatially shared linear operator is often sufficient to represent diverse image manipulations, while its leading singular components capture semantic content and higher-rank components primarily refine image details. We frame these findings as predictive representational sufficiency rather than evidence of intrinsic linear feature-space geometry.
Read the full abstract ▾
Computer Vision
Unpacking the Eye of the Beholder: Social Location, Identity, and the Moving Target of Political Perspectives
Elena Sirotkina
Political and social identities structure how people evaluate political information, a finding decades deep in political science and routinely discarded by computational tools that often produce single scores that treat a piece of text, an image, or a
video as if it means the same thing to everyone. This paper shows that it does not, and that the difference is consequential. To address this problem, I develop the Perspectivist Visual Political Sentiment (PVPS) classifier, which learns from approximately 82,000 evaluations by 5,575 U.S. adults to predict how audiences defined by political and social identities will evaluate the same image. Unlike standard tools that average systematic disagreement away, PVPS preserves it, returning an evaluative profile that records who agrees, who diverges, and along which identity lines. Applied to several influential studies of visual sentiment, PVPS shows that perceived violence in protest imagery and the emotional mechanisms behind protest image engagement both change substantively once audience identity is taken into account. It follows that what a political image conveys is a moving target, and measuring it requires knowing whom it is moving.
Read the full abstract ▾
Computer Vision
USEMA: a Scalable Efficient Mamba Like Attention for Medical Image Segmentation
Elisha Dayag, Nhat Thanh Tran and Jack Xin
Accurate medical image
segmentation is an integral part of the medical image analysis pipeline that requires the ability to merge local and global information. While
vision transformers are able to capture global interactions using vanilla self-
attention, their quadratic computational complexity in the input size remains a struggle for medical image
segmentation tasks. Motivated by the dispersion property of vanilla self-
attention and recent development of Mamba form of
attention, Scalable and Efficient Mamba like
Attention (SEMA) utilizes
token localization via local window
attention to avoid dispersion and maintain focusing, complemented by theoretically consistent arithmetic averaging to capture global aspect of
attention. In this work, we present USEMA, a hybrid UNet architecture that merges the local feature extraction ability of convolutional neural networks (CNNs) with SEMA
attention. We conduct experiments with USEMA across a variety of modalities and image sizes, demonstrating improved computational efficiency compared to
transformer based models using full self-
attention, and superior
segmentation performance relative to purely convolution and Mamba-based models.
Read the full abstract ▾
Computer Vision
LatentHDR: Decoupling Exposure from Diffusion via Conditional Latent-to-Latent Mapping for Text/Image-to-Panoramic HDR
Pedram Fekri, WenChen Li, William Chen et al. (+1 more)
High Dynamic Range (HDR) generation remains challenging for generative models, which are largely limited to low dynamic range outputs. Recent diffusionbased approaches approximate HDR by generating multiple exposure-conditioned samples, incurring high computational cost and structural inconsistencies across exposures. We propose LatentHDR, a framework that decouples scene generation from exposure modeling in latent space. A pretrained
diffusion backbone produces a single coherent scene representation, while a lightweight conditional latent to-latent head deterministically maps it to exposure-specific representations. This enables the generation of a dense, structurally consistent exposure stack in a single pass. This design eliminates multi-pass
diffusion, ensures cross-exposure
alignment, and enables scalable HDR synthesis. LatentHDR supports both textand image-conditioned HDR generation for perspective and panoramic scenes. Experiments on synthetic data and the SI-HDR
benchmark show that LatentHDR achieves state-of-the-art dynamic range with competitive perceptual quality, while reducing computation by an order of magnitude. Our results demonstrate that high-quality HDR generation can be achieved through structured latent modeling, challenging the need for
stochastic multi-exposure generation.
Read the full abstract ▾
All authors Pedram Fekri, WenChen Li, William Chen, Peter Altamirano
Computer Vision · Cross-listed from physics.geo-ph
Deploying Self-Supervised Learning for Real Seismic Data Denoising
Giovanny A. M. Arboleda, Claudio D. T. de Souza, Carlos E. M. dos Anjos et al. (+6 more)
Self-supervised learning (SSL) has emerged as a promising approach to seismic data denoising as it does not require clean reference data. In this work, the deployment of the Noisy-as-Clean (NaC) method was evaluated for real seismic data denoising under controlled conditions. Two independent seismic acquisitions, each comprising noisy and filtered data, were organized into four real
datasets. The NaC SSL method was adapted to add real noise to the noisy input, controlled by a parameter. An experimental protocol with ten experiments was designed to compare different strategies for deploying the NaC SSL method with the supervised learning baseline, using identical network
topology and hyperparameters. The models were evaluated in terms of denoising performance, computational cost, and generalization capability. The results show that the synthetic additive white Gaussian noise (AWGN) is inadequate for the denoising of seismic data within the NaC method, and performance strongly depends on the compatibility between the injected and actual noise characteristics. Furthermore, both the characteristics of the seismic data and the noise level influence the performance of the model. Self-supervised fine-tuning on test data has improved SSL performance, whereas no such gain was observed for fine-tuning of supervised models. Finally, NaC has shown to be a simple, effective, and model-independent method that offers a feasible solution for the denoising of real seismic data.
Read the full abstract ▾
All authors Giovanny A. M. Arboleda, Claudio D. T. de Souza, Carlos E. M. dos Anjos, Lessandro de S. S. Valente, Roosevelt de L. Sardinha, Albino Aveleda, Pablo M. Barros, André Bulcão, Alexandre G. Evsukoff
Computer Vision
Birds of a Feather Flock Together: Background-Invariant Representations via Linear Structure in VLMs
Youssef Zaazou and Mark Thomas
Vision-
language models (VLMs), such as CLIP and SigLIP 2, are widely used for image classification, yet their
vision encoders remain vulnerable to systematic biases that undermine robustness. In particular, correlations between foreground objects and their backgrounds constitute a salient and practically important class of spurious dependencies. In this work, we revisit the well-known property of high linear additivity in VLM
embedding spaces and show that it enables a decomposition of scene representations into foreground and background components. Leveraging this insight, we introduce a pre-training approach that exploits this property to construct background-invariant representations using synthetic data. Our method achieves, to our knowledge, the first worst-group accuracy exceeding $90\%$ on Waterbirds under perfect ($100\%$) spurious correlation (i.e., no minority-group examples in the training data). Furthermore, it demonstrates strong sim-to-real transfer and requires no access to real-world debiased data, making it practical for real-world deployment.
Read the full abstract ▾
Computer Vision
HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer
Qi Cai, Jingwen Chen, Chengmin Gao et al. (+22 more)
The
evolution of visual generative models has long been constrained by fragmented architectures relying on disjoint text encoders and external VAEs. In this report, we present HiDream-O1-Image, a natively unified generative foundation model via pixel-space
Diffusion Transformer, that pioneers a paradigm shift from modular architectures to an end-to-end in-context visual generation engine. By mapping raw image pixels, text
tokens, and task-specific conditions into a single shared
token space, HiDream-O1-Image achieves a structural unification of
multimodal inputs within an Unified
Transformer (UiT) architecture. This native encoding paradigm eliminates the need for separate VAEs or disjoint pre-trained text encoders, allowing the model to treat diverse generation and editing tasks as a consistent in-context
reasoning process. Extensive experiments show that HiDream-O1-Image excels across various generation tasks, including text-to-image generation, instruction-based editing, and subject-driven personalization. Notably, with only 8B parameters, HiDream-O1-Image (8B) achieves performance parity with or even surpasses established state-of-the-art models with significantly larger parameters (e.g., 27B Qwen-Image). Crucially, to validate the immense scalability of this paradigm, we successfully scale the architecture up to over 200B parameters. Experimental results demonstrate that this massive-scale version HiDream-O1-Image-Pro (200B+) unlocks unprecedented generative capabilities and superior performance, establishing new state-of-the-art
benchmarks. Ultimately, HiDream-O1-Image highlights the immense potential of natively unified architectures and charts a highly scalable path toward next-generation
multimodal AI.
Read the full abstract ▾
All authors Qi Cai, Jingwen Chen, Chengmin Gao, Zijian Gong, Yehao Li, Yingwei Pan, Yi Peng, Zhaofan Qiu, Kai Yu, Yiheng Zhang, Hao Ai, Siying Bai, Yang Chen, Zhihui Chen, Fengbin Gao, Ying Guo, Dong Li, Zhen Shen, Leilei Shi, Jing Wang, Siyu Wang, Yimeng Wang, Rui Zheng, Ting Yao, Tao Mei
Computer Vision · Cross-listed from eess.IV
SplitFed-CL: A Split Federated Co-Learning Framework for Medical Image Segmentation with Inaccurate Labels
Zahra Hafezi Kafshgari, Hadi Hadizadeh and Parvaneh Saeedi
Split
Federated Learning (SplitFed) combines
federated and split learning to preserve
privacy while reducing client-side computation. However, in medical image
segmentation, heterogeneous label quality across clients can significantly degrade performance. We propose SplitFed-CL, a co-learning framework where a global teacher guides local students to detect and refine unreliable annotations. Reliable labels supervise training directly, while unreliable labels are corrected via weighted student–teacher refinement. SplitFed-CL further incorporates consistency regularization for robustness to input perturbations and a trainable weighting module to balance loss terms adaptively. We also introduce a novel difficulty guided strategy to simulate human like boundary centric annotation errors, where the degree of perturbation is governed by shape complexity and the associated annotation difficulty. Experiments on two multiclass
segmentation datasets with controlled synthetic noise, together with a binary
segmentation dataset containing real-world annotation errors, demonstrate that SplitFed-CL consistently outperforms seven state-of-the-art baselines, yielding improved
segmentation quality and robustness.
Read the full abstract ▾
Computer Vision
The first global agricultural field boundary map at 10m resolution
Caleb Robinson, Gedeon Muhawenayo, Subash Khanal et al. (+9 more)
The agricultural field is the natural unit at which crops are planted, managed, regulated, and reported, yet most global remote-sensing products for agriculture are only available at the pixel level. While some high-quality field-level data products exist, they come from parcel registries covering only parts of Europe or from ML-derived products for individual countries. No openly available, globally consistent map of agricultural field boundaries exists to date. Here we present the first global field boundary
dataset at 10\,m resolution for the years 2024 and 2025, comprising 3.17 billion remote-sensing field polygons (1.62 B in 2024 and 1.55 B in 2025) across 241 countries and territories, produced by applying a U-Net
segmentation model trained on the Fields of The World
dataset to cloud-free Sentinel-2 mosaics. Validated against ground-truth field boundaries in 24 countries, the map achieved a mean pixel-level recall of 0.85 with 14 countries exceeding 0.90. Evaluation against full-country ground-truth
datasets in Austria, Latvia, and Finland yielded F1 scores of 0.89, 0.88, and 0.74, respectively. Because reference data for global validation is inherently incomplete, we accompanied the map with a 500 m confidence layer that identifies regions where predictions are reliable. We release the
dataset openly as three global maps: the confidence-thresholded default field boundary
dataset, the full unfiltered
dataset, and the continuous-valued confidence raster. These maps provide the first globally consistent field-level unit of analysis for crop monitoring, food security, and downstream agricultural science.
Read the full abstract ▾
All authors Caleb Robinson, Gedeon Muhawenayo, Subash Khanal, Zhanpei Fang, Isaac Corley, Ana M. Tárano, Lyndon Estes, Jennifer Marcus, Nathan Jacobs, Hannah Kerner, Inbal Becker-Reshef, Juan M. Lavista Ferres
Computer Vision
Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping
Haoyuan Sun, Jing Wang, Yuxin Song et al. (+9 more)
Recently, post-training methods based on reinforcement learning, with a particular focus on Group Relative Policy Optimization (GRPO), have emerged as the robust paradigm for further advancement of text-to-image (T2I) models. However, these methods are often prone to reward hacking, wherein models exploit biases in imperfect reward functions rather than yielding genuine performance gains. In this work, we identify that normalization could lead to miscalibration and directly removing the
prompt-level standard deviation term yields an optimal policy ascent direction that is linear in the advantage but still limits the separation of genuine signals from noise. To mitigate the above issues, we propose Super-Linear Advantage Shaping (SLAS) by revisiting the functional update from an information geometry perspective. By extending the Fisher-Rao information metric with advantage-dependent weighting, SLAS introduces a non-linear
geometric structure that reshapes the local policy space. This design relaxes constraints along high-advantage directions to amplify informative updates, while tightening those in low-advantage regions to suppress illusory gradients. In addition, batch-level normalization is applied to stabilize training under varying reward scales. Extensive evaluations demonstrate that SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and
benchmarks. In particular, it yields faster training dynamics, improved out-of-domain performance on GenEval and UniGenBench++, and enhanced robustness to model scaling, while mitigating reward hacking and preserving semantic and compositional fidelity in generations.
Read the full abstract ▾
All authors Haoyuan Sun, Jing Wang, Yuxin Song, Yu Lu, Bo Fang, Yifu Luo, Jun Yin, Pengyu Zeng, Miao Zhang, Tiantian Zhang, Xueqian Wang, Shijian Lu
Computer Vision
Personal Visual Context Learning in Large Multimodal Models
Zihui Xue, Ami Baid, Sangho Kim et al. (+2 more)
As wearable devices like smart glasses integrate Large
Multimodal Models (LMMs) into the continuous first-person visual streams of individual users, the
evolution of these models into true personal assistants hinges on visual personalization: the ability to reason over visual information unique to the wearer. We formalize this capability as Personal Visual Context Learning (Personal VCL), the
prompt-time capability of using user-specific visual context to resolve personalized queries. To systematically evaluate this, we present Personal-VCL-Bench, a comprehensive
benchmark capturing the personal visual world across persons, objects, and behaviors. Our analysis of frontier LMMs identifies a profound context utilization gap, revealing that the mechanisms for leveraging visual evidence, as well as aggregating multiple visual observations, remain critically understudied. Motivated by these findings, we propose the Agentic Context Bank, a strong inference-time baseline that structures a user's visual context into a self-refining
memory bank and employs query-adaptive evidence selection. Our baseline approach consistently improves over standard context prompting regimes across tasks and evaluated backbones, demonstrating a practical path towards future personalized LMMs.
Read the full abstract ▾
All authors Zihui Xue, Ami Baid, Sangho Kim, Mi Luo, Kristen Grauman
Computer Vision · Cross-listed from cs.LG
Variational Inference for Lévy Process-Driven SDEs via Neural Tilting
Yaman Kindap, Manfred Opper, Benjamin Dupuis et al. (+2 more)
Modelling extreme events and heavy-tailed phenomena is central to building reliable predictive systems in domains such as finance,
climate science, and safety-critical AI. While Lévy processes provide a natural mathematical framework for capturing jumps and heavy tails,
Bayesian inference for Lévy-driven
stochastic differential equations (SDEs) remains intractable with existing methods: Monte Carlo approaches are rigorous but lack scalability, whereas neural variational inference methods are efficient but rely on Gaussian assumptions that fail to capture discontinuities. We address this tension by introducing a neural exponential tilting framework for variational inference in Lévy-driven SDEs. Our approach constructs a flexible variational family by exponentially reweighting the Lévy measure using neural networks. This parametrization preserves the jump structure of the underlying process while remaining computationally tractable. To enable efficient inference, we develop a quadratic neural parametrization that yields closed-form normalization of the tilted measure, a conditional Gaussian representation for stable processes that facilitates simulation, and
symmetry-aware Monte Carlo
estimators for scalable optimization. Empirically, we demonstrate that the method accurately captures jump dynamics and yields reliable posterior inference in regimes where Gaussian-based variational approaches fail, on both synthetic and real-world
datasets.
Read the full abstract ▾
All authors Yaman Kindap, Manfred Opper, Benjamin Dupuis, Umut Simsekli, Tolga Birdal
Computer Vision
Pixal3D: Pixel-Aligned 3D Generation from Images
Dong-Yang Li, Wang Zhao, Yuxin Chen et al. (+5 more)
Recent advances in 3D generative models have rapidly improved image-to-3D synthesis quality, enabling higher-resolution geometry and more realistic appearance. Yet fidelity, which measures pixel-level faithfulness of the generated 3D asset to the input image, still remains a central bottleneck. We argue this stems from an implicit 2D-3D correspondence issue: most 3D-native generators synthesize shape in canonical space and inject image cues via
attention, leaving pixel-to-3D associations ambiguous. To tackle this issue, we draw inspiration from 3D reconstruction and propose Pixal3D, a pixel-aligned 3D generation paradigm for high-fidelity 3D asset creation from images. Instead of generating in a canonical pose, Pixal3D directly generates 3D in a pixel-aligned way, consistent with the input view. To enable this, we introduce a pixel back-projection conditioning scheme that explicitly lifts multi-scale image features into a 3D feature volume, establishing direct pixel-to-3D correspondence without ambiguity. We show that Pixal3D is not only scalable and capable of producing high-quality 3D assets, but also substantially improves fidelity, approaching the fidelity level of reconstruction. Furthermore, Pixal3D naturally extends to multi-view generation by aggregating back-projected feature volumes across views. Finally, we show pixel-aligned generation benefits scene synthesis, and present a modular pipeline that produces high-fidelity, object-separated 3D scenes from images. Pixal3D for the first time demonstrates 3D-native pixel-aligned generation at scale, and provides a new inspiring way towards high-fidelity 3D generation of object or scene from single or multi-view images. Project page: https://ldyang694.github.io/projects/pixal3d/
Read the full abstract ▾
All authors Dong-Yang Li, Wang Zhao, Yuxin Chen, Wenbo Hu, Meng-Hao Guo, Fang-Lue Zhang, Ying Shan, Shi-Min Hu