Computer Vision
Scaling Remote Sensing Foundation Models: Data Domain Tradeoffs at the Peta-Scale
Charith Wickrema, Eliza Mace, Hunter Brown et al. (+7 more)
We explore the scaling behaviors of artificial intelligence to establish practical techniques for training foundation models on high-resolution electro-optical (EO)
datasets that exceed the current state-of-the-art scale by orders of magnitude. Modern
multimodal machine learning (ML) applications, such as generative artificial intelligence (GenAI) systems for image captioning, search, and
reasoning, depend on robust, domain-specialized encoders for non-text modalities. In natural image domains where internet-scale data is plentiful, well-established scaling laws help optimize the joint scaling of model capacity, training compute, and
dataset size. Unfortunately, these relationships are much less well understood in high-value domains like remote sensing (RS). Using over a quadrillion pixels of commercial satellite EO data and MITRE's Federal AI Sandbox, we train progressively larger
vision transformer (ViT) backbones, report successes and failure modes observed at peta-scale, and analyze implications for bridging domain gaps across additional RS modalities. We observe that even at this scale, performance is consistent with a data-limited regime rather than a model parameter-limited one. These practical insights are intended to inform data collection strategies, compute budgets, and optimization schedules that advance the future development of frontier scale RS foundation models.
Read the full abstract ▾
All authors Charith Wickrema, Eliza Mace, Hunter Brown, Heidys Cabrera, Nick Krall, Matthew O'Neill, Shivangi Sarkar, Lowell Weissman, Eric Hughes, Guido Zarrella
Computer Vision
MRI-to-CT Synthesis With Cranial Suture Segmentations Using A Variational Autoencoder Framework
Krithika Iyer, Austin Tapp, Athelia Paulli et al. (+4 more)
Quantifying normative pediatric cranial development and suture ossification is crucial for diagnosing and treating growth-related cephalic disorders. Computed
tomography (CT) is widely used to evaluate cranial and sutural deformities; however, its ionizing radiation is contraindicated in children without significant abnormalities.
Magnetic resonance imaging (MRI) offers radiation free scans with superior soft tissue contrast, but unlike CT, MRI cannot elucidate cranial sutures, estimate skull bone density, or assess cranial vault growth. This study proposes a deep learning driven pipeline for transforming T1 weighted MRIs of children aged 0.2 to 2 years into synthetic CTs (sCTs), predicting detailed cranial bone
segmentation, generating suture probability heatmaps, and deriving direct suture
segmentation from the heatmaps. With our in-house pediatric data, sCTs achieved 99% structural similarity and a Frechet inception distance of 1.01 relative to real CTs. Skull
segmentation attained an average Dice coefficient of 85% across seven cranial bones, and sutures achieved 80% Dice. Equivalence of skull and suture
segmentation between sCTs and real CTs was confirmed using two one sided tests (TOST p < 0.05). To our knowledge, this is the first pediatric cranial CT synthesis framework to enable suture
segmentation on sCTs derived from MRI, despite MRI's limited depiction of bone and sutures. By combining robust, domain specific variational autoencoders, our method generates perceptually indistinguishable cranial sCTs from routine pediatric MRIs, bridging critical gaps in non invasive cranial evaluation.
Read the full abstract ▾
All authors Krithika Iyer, Austin Tapp, Athelia Paulli, Gabrielle Dickerson, Syed Muhammad Anwar, Natasha Lepore, Marius George Linguraru
Computer Vision · Cross-listed from cs.RO
Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation
Guo Ye, Zexi Zhang, Xu Zhao et al. (+4 more)
Vision-
Language-Action (VLA) models have shown remarkable generalization by mapping web-scale knowledge to robotic control, yet they remain blind to physical contact. Consequently, they struggle with contact-rich manipulation tasks that require
reasoning about force, texture, and slip. While some approaches incorporate low-dimensional tactile signals, they fail to capture the high-resolution dynamics essential for such interactions. To address this limitation, we introduce DreamTacVLA, a framework that grounds VLA models in contact physics by learning to feel the future. Our model adopts a hierarchical perception scheme in which high-resolution tactile images serve as micro-
vision inputs coupled with wrist-camera local
vision and third-person macro
vision. To reconcile these multi-scale sensory streams, we first train a unified policy with a Hierarchical Spatial
Alignment (HSA) loss that aligns tactile
tokens with their spatial counterparts in the wrist and third-person views. To further deepen the model's understanding of fine-grained contact dynamics, we finetune the system with a tactile world model that predicts future tactile signals. To mitigate tactile data scarcity and the wear-prone nature of tactile sensors, we construct a hybrid large-scale
dataset sourced from both high-fidelity digital twin and real-world experiments. By anticipating upcoming tactile states, DreamTacVLA acquires a rich model of contact physics and conditions its actions on both real observations and imagined consequences. Across contact-rich manipulation tasks, it outperforms state-of-the-art VLA baselines, achieving up to 95% success, highlighting the importance of understanding physical contact for robust, touch-aware robotic
agents.
Read the full abstract ▾
All authors Guo Ye, Zexi Zhang, Xu Zhao, Shang Wu, Haoran Lu, Shihan Lu, Han Liu
Computer Vision
Lifelong Domain Adaptive 3D Human Pose Estimation
Qucheng Peng, Hongfei Xue, Pu Wang et al. (+1 more)
3D Human Pose Estimation (3D HPE) is vital in various applications, from person re-identification and action recognition to virtual reality. However, the reliance on annotated 3D data collected in controlled environments poses challenges for generalization to diverse in-the-wild scenarios. Existing domain adaptation (DA) paradigms like general DA and source-free DA for 3D HPE overlook the issues of non-stationary target pose
datasets. To address these challenges, we propose a novel task named lifelong domain adaptive 3D HPE. To our knowledge, we are the first to introduce the lifelong domain adaptation to the 3D HPE task. In this lifelong DA setting, the pose
estimator is pretrained on the source domain and subsequently adapted to distinct target domains. Moreover, during adaptation to the current target domain, the pose
estimator cannot access the source and all the previous target domains. The lifelong DA for 3D HPE involves overcoming challenges in adapting to current domain poses and preserving knowledge from previous domains, particularly combating catastrophic forgetting. We present an innovative Generative
Adversarial Network (GAN) framework, which incorporates 3D pose generators, a 2D pose discriminator, and a 3D pose
estimator. This framework effectively mitigates domain shifts and aligns original and augmented poses. Moreover, we construct a novel 3D pose generator paradigm, integrating pose-aware, temporal-aware, and domain-aware knowledge to enhance the current domain's adaptation and alleviate catastrophic forgetting on previous domains. Our method demonstrates superior performance through extensive experiments on diverse domain adaptive 3D HPE
datasets.
Read the full abstract ▾
All authors Qucheng Peng, Hongfei Xue, Pu Wang, Chen Chen
Computer Vision
TinyHistory: Lightweight Video History Embeddings via Two-Stage Context Learning
Lvmin Zhang, Shengqu Cai, Muyang Li et al. (+6 more)
History context is central to autoregressive
video generation, driving consistency and storytelling for both commercial models and personal use cases. For example, personal users, offline workflows, and individual-scale finetuning need to encode longer
video histories under tight compute and
memory budgets. We observe that content and identity consistency is an essential requirement, and that complete, uninterrupted history coverage together with content query and interpretation capabilities is broadly desired. We present TinyHistory, a lightweight history
embedding learned through two-stage context learning. In the first stage, we pretrain the encoder on large-scale
video data with a randomized frame query objective; in the second stage, we repurpose the pretrained encoder within an autoregressive
video diffusion model to learn content-level consistency. As a result, we show that the learned lightweight
embeddings achieve consistency comparable (by VLM, VBench, ELO, etc) to heavier alternatives, while reducing training overhead and extending the encodable history length within a given
memory budget. We conduct ablation studies to analyze the influence and trade-offs of each component.
Read the full abstract ▾
All authors Lvmin Zhang, Shengqu Cai, Muyang Li, Chong Zeng, Beijia Lu, Anyi Rao, Song Han, Gordon Wetzstein, Maneesh Agrawala
Computer Vision
Video-Based Performance Evaluation for ECR Drills in Synthetic Training Environments
Surya Rayala, Marcos Quinones-Grueiro, Naveeduddin Mohammed et al. (+5 more)
Effective urban warfare training requires situational awareness and muscle
memory, developed through repeated practice in realistic yet controlled environments. A key drill, Enter and Clear the Room (ECR), demands threat assessment, coordination, and securing confined spaces. The military uses Synthetic Training Environments that offer scalable, controlled settings for repeated exercises. However, automatic performance assessment remains challenging, particularly when aiming for objective evaluation of cognitive, psychomotor, and teamwork skills. Traditional methods often rely on costly, intrusive sensors or subjective human observation, limiting scalability and accuracy. This paper introduces a
video-based assessment pipeline that derives performance analytics from training
videos without requiring additional hardware. By utilizing computer
vision models, the system extracts 2D skeletons, gaze vectors, and movement trajectories. From these data, we develop task-specific metrics that measure psychomotor fluency, situational awareness, and team coordination. These metrics feed into an extended Cognitive Task Analysis (CTA) hierarchy, which employs a weighted combination to generate overall performance scores for teamwork and cognition. We demonstrate the approach with a case study of real-world ECR drills, providing actionable, domain specific metrics that capture individual and team performance. We also discuss how these insights can support After Action Reviews with interactive dashboards within Gamemaster and the Generalized Intelligent Framework for Tutoring (GIFT), providing intuitive and understandable feedback. We conclude by addressing limitations, including tracking difficulties, ground-truth validation, and the broader applicability of our approach. Future work includes expanding analysis to 3D
video data and leveraging
video analysis to enable scalable evaluation within STEs.
Read the full abstract ▾
All authors Surya Rayala, Marcos Quinones-Grueiro, Naveeduddin Mohammed, Ashwin T S, Benjamin Goldberg, Randall Spain, Paige Lawton, Gautam Biswas
Computer Vision
Bridging the Ex-Vivo to In-Vivo Gap: Synthetic Priors for Monocular Depth Estimation in Specular Surgical Environments
Ankan Aich, Emma D. Ryan, Kris Moe et al. (+3 more)
Accurate
Monocular Depth Estimation (MDE) is critical for autonomous robotic surgery. However, existing self-supervised methods often exhibit a severe "ex-vivo to in-vivo gap": they achieve high accuracy on public
datasets but struggle in actual clinical deployments. This disparity arises because the severe specular reflections and fluid-filled deformations inherent to real surgeries. Models trained on noisy real-world pseudo-labels consequently suffer from severe boundary collapse. To address this, we leverage the high-fidelity synthetic priors of the Depth Anything V2 architecture, which inherently capture precise
geometric details, and efficiently adapt them to the medical domain using Dynamic Vector Low-Rank Adaptation (DV-LORA). Our contributions are two-fold. Technically, our approach establishes a new state-of-the-art on the public SCARED
dataset; under a novel physically-stratified evaluation protocol, it reduces Squared Relative Error by over 17% in high-specularity regimes compared to strong baselines. Furthermore, to provide a rigorous reality check for the field, we introduce ROCAL-T 90 (Real Operative CT-Aligned Laparoscopic Trajectories 90), the first real-surgery validation
dataset featuring 90 clinical endoscopic sequences with sub-millimeter ($< 1$mm) ground-truth trajectories. Evaluations on ROCAL-T 90 demonstrate our model's superior robustness in true clinical settings.
Read the full abstract ▾
All authors Ankan Aich, Emma D. Ryan, Kris Moe, Isaac Schmale, Li-Xing Man, Yangming Lee
Computer Vision · Cross-listed from cs.LG
A Granular Grassmannian Clustering Framework via the Schubert Variety of Best Fit
Karim Salta, Michael Kirby and Chris Peterson
In many classification and clustering tasks, it is useful to compute a
geometric representative for a
dataset or a cluster, such as a mean or median. When
datasets are represented by subspaces, these representatives become points on the Grassmann or flag
manifold, with distances induced by their geometry, often via principal angles. We introduce a subspace clustering
algorithm that replaces subspace means with a trainable prototype defined as a Schubert Variety of Best Fit (SVBF) - a subspace that comes as close as possible to intersecting each cluster member in at least one fixed direction. Integrated in the Linde-Buzo-Grey (LBG) pipeline, this SVBF-LBG scheme yields improved cluster purity on synthetic, image, spectral, and
video action data, while retaining the mathematical structure required for downstream analysis.
Read the full abstract ▾
Computer Vision
Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion
Hau-Shiang Shiu, Chin-Yang Lin, Zhixiang Wang et al. (+4 more)
Diffusion-based
video super-resolution (VSR) methods deliver strong perceptual quality but are often unsuitable for latency-sensitive scenarios due to reliance on future frames and expensive multi-step denoising. We propose Stream-DiffVSR, a causally conditioned
diffusion framework for efficient online VSR. Operating strictly on past frames, Stream-DiffVSR integrates a four-step distilled denoiser for fast inference, an Auto-regressive Temporal Guidance (ARTG) module that injects motion-aligned cues during latent denoising, and a lightweight temporal-aware decoder with a Temporal Processor Module (TPM) to enhance detail and temporal coherence. Unlike chunk-wise streaming inference, our strictly frame-by-frame causal design avoids sequence-level waiting, substantially reducing time-to-first-frame and end-to-end latency. Stream-DiffVSR processes 720p frames in 0.328 seconds on an RTX 4090 and consistently outperforms prior
diffusion-based baselines. Compared with the online state-of-the-art TMP, it improves perceptual quality (LPIPS +0.095). Compared with prior
diffusion-based VSR methods such as MGLD-VSR, it reduces per-frame runtime by over 130x. Moreover, Stream-DiffVSR substantially lowers time-to-first-frame for
diffusion-based VSR, reducing initial delay from over 4600 seconds to 0.328 seconds, making
diffusion-based VSR markedly more practical for low-latency online and streaming deployment. Project page: https://jamichss.github.io/stream-diffvsr-project-page/
Read the full abstract ▾
All authors Hau-Shiang Shiu, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao, Po-Fan Yu, Yu-Chih Chen, Yu-Lun Liu
Computer Vision
Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation
Shaocong Xu, Songlin Wei, Qizhe Wei et al. (+12 more)
Transparent objects remain notoriously hard for perception systems: refraction, reflection and transmission break the assumptions behind stereo, ToF and purely discriminative
monocular depth, causing holes and temporally unstable estimates. Our key observation is that modern
video diffusion models already synthesize convincing transparent phenomena, suggesting they have internalized the optical rules. We build TransPhy3D, a synthetic
video corpus of transparent/reflective scenes: 11k sequences rendered with Blender/Cycles. Scenes are assembled from a curated bank of category-rich static assets and shape-rich procedural assets paired with glass/plastic/metal materials. We render RGB + depth + normals with physically based ray tracing and OptiX denoising. Starting from a large
video diffusion model, we learn a
video-to-
video translator for depth (and normals) via lightweight LoRA adapters. During training we concatenate RGB and (noisy) depth latents in the DiT backbone and co-train on TransPhy3D and existing frame-wise synthetic
datasets, yielding temporally consistent predictions for arbitrary-length input
videos. The resulting model, DKT, achieves zero-shot SOTA on real and synthetic
video benchmarks involving transparency: ClearPose, DREDS (CatKnown/CatNovel), and TransPhy3D-Test. It improves accuracy and temporal consistency over strong image/
video baselines, and a normal variant sets the best
video normal estimation results on ClearPose. A compact 1.3B version runs at ~0.17 s/frame. Integrated into a grasping stack, DKT's depth boosts success rates across translucent, reflective and diffuse surfaces, outperforming prior
estimators. Together, these results support a broader claim: "
Diffusion knows transparency." Generative
video priors can be repurposed, efficiently and label-free, into robust, temporally coherent perception for challenging real-world manipulation.
Read the full abstract ▾
All authors Shaocong Xu, Songlin Wei, Qizhe Wei, Zheng Geng, Hong Li, Licheng Shen, Qianpu Sun, Shu Han, Bin Ma, Bohan Li, Chongjie Ye, Yuhang Zheng, Nan Wang, Saining Zhang, Hao Zhao
Computer Vision · Cross-listed from cs.AI
Web World Models
Jichen Feng, Yifan Zhang, Chenggong Zhang et al. (+3 more)
Language agents increasingly require persistent worlds in which they can act, remember, and learn. Existing approaches sit at two extremes: conventional web frameworks provide reliable but fixed contexts backed by databases, while fully generative world models aim for unlimited environments, but the world is constructed primarily through generation, making it harder to maintain a fixed, deterministic global framework and reducing controllability. In this work, we introduce the Web World Model (WWM), a middle ground where world state and “physics” are implemented in ordinary web code to ensure logical consistency, while large
language models generate context, narratives, and high-level decisions on top of this structured latent state. We build a suite of WWMs on a realistic web stack, including an infinite travel atlas grounded in real geography, fictional
galaxy explorers, web-scale encyclopedic and narrative worlds, and simulation- and game-like environments. Across these systems, we identify practical design principles for WWMs: separating code-defined rules from model-driven imagination, representing latent state as typed web interfaces, and utilizing deterministic generation to achieve unlimited but structured exploration. Our results suggest that web stacks themselves can serve as a scalable substrate for world models, enabling controllable yet open-ended environments.
Read the full abstract ▾
All authors Jichen Feng, Yifan Zhang, Chenggong Zhang, Yifu Lu, Shilong Liu, Mengdi Wang
Computer Vision
MVID: Feed-Forward Multi-View Intrinsic Image Decomposition
Kang Du, Duotun Wang, Wanling Li et al. (+2 more)
Intrinsic image decomposition aims to recover material and illumination factors from RGB observations, but real-world images entangle reflectance with illumination, visibility, shadows, and non-diffuse appearance. Recent single-image methods address this
entanglement with a residual image formation model, decomposing RGB into albedo, diffuse shading, and a non-diffuse residual. However, applying such decomposition independently to consecutive frames lacks scene-level context, leading to inconsistent factor assignments, albedo drift, and leakage across views. Meanwhile, multi-view inverse-rendering methods recover properties, such as material, lighting, and geometry, but they rely on synthetic data due to the highly uncertain estimation, and do not generalize to supervision by real images. We present MVID, Multi-View Intrinsic image Decomposition, a feed-forward framework built on the residual image formation model. MVID builds a scene-level multi-view representation and decodes a view-consistent albedo together with coherent per-view shading and residual factors through a factor query adapter, while using the same image formation model for self-supervised RGB reconstruction on unlabeled real-world sequences. Experiments on indoor, real-world, and outdoor
benchmarks show that MVID improves both per-frame decomposition quality and cross-view consistency over single-view intrinsic, generative intrinsic, and multi-view inverse-rendering baselines. The resulting view-stable factors support practical image-space applications, including multi-view consistent illumination editing and specularity removal.
Read the full abstract ▾
All authors Kang Du, Duotun Wang, Wanling Li, Yirui Guan, Zeyu Wang
Computer Vision · Cross-listed from cs.RO
RoboMirror: Understand Before You Imitate for Video to Humanoid Locomotion
Zhe Li, Boan Zhu, Yangyang Wei et al. (+11 more)
Humans learn locomotion through visual observation, interpreting visual content first before imitating actions. However, state-of-the-art
humanoid locomotion systems rely on either curated motion capture trajectories or sparse text commands, leaving a critical gap between visual understanding and control. Text-to-motion methods suffer from semantic sparsity and staged pipeline errors, while
video-based approaches only perform mechanical pose mimicry without genuine visual understanding. We propose RoboMirror, the first retargeting-free
video-to-locomotion framework embodying "understand before you imitate". Leveraging VLMs, it distills raw
egocentric/third-person
videos into visual motion intents, which directly condition a
diffusion-based policy to generate physically plausible, semantically aligned locomotion without explicit pose reconstruction or retargeting. Extensive experiments validate the effectiveness of RoboMirror, it enables telepresence via
egocentric videos, drastically reduces third-person control latency by 80%, and achieves a 3.7% higher task success rate than baselines. By reframing
humanoid control around
video understanding, we bridge the visual understanding and action gap.
Read the full abstract ▾
All authors Zhe Li, Boan Zhu, Yangyang Wei, Shuanghao Bai, Yuheng Ji, Yibo Peng, Tao Huang, Pengwei Wang, Zhongyuan Wang, S.-H. Gary Chan, Chang Xu, Cheng Chi, Jianfei Yang, Shanghang Zhang
Computer Vision
Active Perception Agent for Omnimodal Audio-Video Understanding
Keda Tao, Wenjie Du, Bohan Yu et al. (+3 more)
Omnimodal large
language models have made significant strides in unifying audio and visual modalities; however, they often face challenges in fine-grained cross-modal understanding and have difficulty with
multimodal alignment. To address these limitations, we introduce OmniAgent, to our best knowledge, the first fully active perception
agent that dynamically orchestrates specialized unimodal tools to achieve more fine-grained omnimodal
reasoning. Unlike previous works that rely on rigid, static workflows and dense frame-captioning, we demonstrate a paradigm shift from passive response generation to active
multimodal inquiry. OmniAgent employs dynamic planning to autonomously orchestrate tool invocation on demand, strategically concentrating perceptual
attention on task-relevant cues. Central to our approach is a novel coarse-to-fine audio-guided perception paradigm, which leverages audio cues to localize temporal events and guide subsequent
reasoning. Extensive empirical evaluations on three audio-
video understanding
benchmarks demonstrate that OmniAgent achieves state-of-the-art performance, surpassing leading open-source and closed-source models by substantial margins of 10% - 20% accuracy without training.
Read the full abstract ▾
All authors Keda Tao, Wenjie Du, Bohan Yu, Weiqiang Wang, Jian Liu, Huan Wang
Computer Vision
Rethinking the Spatio-Temporal Alignment of End-to-End 3D Perception
Xiaoyu Li, Peidong Li, Xian Wu et al. (+7 more)
Spatio-temporal
alignment is crucial for temporal modeling of end-to-end (E2E) perception in autonomous driving (AD), providing valuable structural and textural prior information. Existing methods typically rely on the
attention mechanism to align objects across frames, simplifying the motion model with a unified explicit physical model (constant velocity, etc.). These approaches prefer semantic features for implicit
alignment, challenging the importance of explicit motion modeling in the traditional perception paradigm. However, variations in motion states and object features across categories and frames render this
alignment suboptimal. To address this, we propose HAT, a spatio-temporal
alignment module that allows each object to adaptively decode the optimal
alignment proposal from multiple hypotheses without direct supervision. Specifically, HAT first utilizes multiple explicit motion models to generate spatial anchors and motion-aware feature proposals for historical instances. It then performs multi-hypothesis decoding by incorporating semantic and motion cues embedded in cached object queries, ultimately providing the optimal
alignment proposal for the target frame. On nuScenes, HAT consistently improves 3D temporal detectors and trackers across diverse baselines. It achieves state-of-the-art tracking results with 46.0% AMOTA on the test set when paired with the DETR3D detector. In an object-centric E2E AD method, HAT enhances perception accuracy (+1.3% mAP, +3.1% AMOTA) and reduces the collision rate by 32%. When semantics are corrupted (nuScenes-C), the enhancement of motion modeling by HAT enables more robust perception and planning in the E2E AD.
Read the full abstract ▾
All authors Xiaoyu Li, Peidong Li, Xian Wu, Long Shi, Dedong Liu, Yitao Wu, Jiajia Fu, Dixiao Cui, Lijun Zhao, Lining Sun
Computer Vision
Memorization in 3D Shape Generation: An Empirical Study
Shu Pu, Boya Zeng, Kaichen Zhou et al. (+2 more)
Generative models are increasingly used in 3D
vision to synthesize novel shapes, yet it remains unclear whether their generation relies on memorizing training shapes. Understanding their memorization could help prevent training data leakage and improve the diversity of generated results. In this paper, we design an evaluation framework to quantify memorization in 3D generative models and study the influence of different data and modeling designs on memorization. We first apply our framework to quantify memorization in existing methods. Next, through controlled experiments with a latent vector-set (Vecset)
diffusion model, we find that, on the data side, memorization depends on data modality, and increases with data diversity and finer-grained conditioning; on the modeling side, it peaks at a moderate guidance scale and can be mitigated by longer Vecsets and simple rotation augmentation. Together, our framework and analysis provide an empirical understanding of memorization in 3D generative models and suggest simple yet effective strategies to reduce it without degrading generation quality. Our code is available at https://github.com/zlab-princeton/3d_mem.
Read the full abstract ▾
All authors Shu Pu, Boya Zeng, Kaichen Zhou, Mengyu Wang, Zhuang Liu
Computer Vision
Scalable Residual Feature Aggregation Framework with Hybrid Metaheuristic Optimization for Robust Early Pancreatic Neoplasm Detection in Multimodal CT Imaging
Janani Annur Thiruvengadam, Kiran Mayee Nabigaru and Anusha Kovi
The early detection of pancreatic neoplasm is a major clinical dilemma, and it is predominantly so because tumors are likely to occur with minimal contrast margins and a large spread anatomy-wide variation amongst patients on a CT scan. These complexities require to be addressed with an effective and scalable system that can assist in enhancing the salience of the subtle visual cues and provide a high level of the generalization on the
multimodal imaging data. A Scalable Residual Feature Aggregation (SRFA) framework is proposed to be used to meet these conditions in this study. The framework integrates a pipeline of preprocessing followed by the
segmentation using the MAGRes-UNet that is effective in making the pancreatic structures and isolating regions of interest more visible. DenseNet-121 performed with residual feature storage is used to extract features to allow deep hierarchical features to be aggregated without properties loss. To go further, hybrid HHO-BA metaheuristic feature selection strategy is used, which guarantees the best feature subset refinement. To be classified, the system is trained based on a new hybrid model that integrates the ability to pay
attention on the world, which is the
Vision Transformer (ViT) with the high representational efficiency of EfficientNet-B3. A dual optimization mechanism incorporating SSA and GWO is used to fine-tune hyperparameters to enhance greater robustness and less overfitting. Experimental results support the significant improvement in performance, with the suggested model reaching 96.23% accuracy, 95.58% F1-score and 94.83% specificity, the model is significantly better than the traditional CNNs and contemporary
transformer-based models. Such results highlight the possibility of the SRFA framework as a useful instrument in the early detection of pancreatic tumors.
Read the full abstract ▾
Computer Vision
Detection Fire in Camera RGB-NIR
Nguyen Truong Khai and Luong Duc Vinh
Improving the accuracy of fire detection using infrared night
vision cameras remains a challenging task. Previous studies have reported strong performance with popular detection models. For example, YOLOv7 achieved an mAP50-95 of 0.51 using an input image size of 640 x 1280, RT-DETR reached an mAP50-95 of 0.65 with an image size of 640 x 640, and YOLOv9 obtained an mAP50-95 of 0.598 at the same resolution. Despite these results, limitations in
dataset construction continue to cause issues, particularly the frequent misclassification of bright artificial lights as fire. This report presents three main contributions: an additional NIR
dataset, a two-stage detection model, and Patched-YOLO. First, to address data scarcity, we explore and apply various data augmentation strategies for both the NIR
dataset and the classification
dataset. Second, to improve night-time fire detection accuracy while reducing false positives caused by artificial lights, we propose a two-stage pipeline combining YOLOv11 and EfficientNetV2-B0. The proposed approach achieves higher detection accuracy compared to previous methods, particularly for night-time fire detection. Third, to improve fire detection in RGB images, especially for small and distant objects, we introduce Patched-YOLO, which enhances the model's detection capability through patch-based processing. Further details of these contributions are discussed in the following sections.
Read the full abstract ▾
Computer Vision
Same or Not? Enhancing Visual Perception in Vision-Language Models
Damiano Marsili, Aditya Mehta, Ryan Y. Lin et al. (+1 more)
Vision-
language models (VLMs) excel at broad visual understanding but remain coarse-grained, exhibit visual biases, and miss subtle visual details. Existing training corpora reinforce this limitation by emphasizing general recognition ("Is it a cat or a dog?") over fine-grained perception. To address this, we introduce a new training corpus and task designed to enhance the perceptual abilities of VLMs. TWIN is a large-scale
dataset of 561,000 image-pair queries that task models to determine whether two visually similar images depict the same object, encouraging
attention to nuanced visual cues. The
dataset spans a diverse range of everyday objects across contexts, viewpoints, and appearances. Fine-tuning VLMs on TWIN yields notable gains in fine-grained recognition, even on unseen domains such as art, animals, plants, and landmarks. To quantify these gains, we introduce FGVQA, a
benchmark suite of 12,000 queries that repurposes fine-grained recognition and
retrieval datasets from multiple domains. While existing VLMs struggle on FGVQA, when fine-tuned on TWIN they improve by up to 19.3%, without compromising performance on general VQA
benchmarks. Finally, our TWIN
dataset scales favorably with object annotations, and our analysis shows that scale is key to performance. We envision TWIN as a drop-in addition to open-source VLM training corpora, advancing perceptual precision of future models. Project webpage: https://glab-caltech.github.io/twin/
Read the full abstract ▾
All authors Damiano Marsili, Aditya Mehta, Ryan Y. Lin, Georgia Gkioxari
Computer Vision
Ethan Chern, Zhulin Hu, Bohao Tang et al. (+4 more)
Real-time
video generation via
diffusion is essential for building general-purpose
multimodal interactive AI systems. However, the simultaneous denoising of all
video frames with bidirectional
attention via an iterative process in
diffusion models prevents real-time interaction. While existing
distillation methods can make the model autoregressive and reduce sampling steps to mitigate this, they focus primarily on text-to-
video generation, leaving the human-AI interaction unnatural and less efficient. This paper targets real-time interactive
video diffusion conditioned on a
multimodal context, including text, image, and audio, to bridge the gap. Given the observation that the leading on-policy
distillation approach Self Forcing encounters challenges (visual artifacts like flickering, black frames, and quality degradation) with
multimodal conditioning, we investigate an improved
distillation recipe with emphasis on the quality of condition inputs as well as the initialization and schedule for the on-policy optimization. On
benchmarks for
multimodal-conditioned (audio, image, and text) avatar
video generation including HDTF, AVSpeech, and CelebV-HQ, our distilled model matches the visual quality of the full-step, bidirectional baselines of similar or larger size with 20x less inference cost and latency. Further, we integrate our model with audio
language models and long-form
video inference technique Anchor-Heavy Identity Sinks to build LiveTalk, a real-time
multimodal interactive avatar system. System-level evaluation on our curated multi-turn interaction
benchmark shows LiveTalk outperforms state-of-the-art models (Sora2, Veo3) in multi-turn
video coherence and content quality, while reducing response latency from 1 to 2 minutes to real-time generation, enabling seamless human-AI
multimodal interaction.
Read the full abstract ▾
All authors Ethan Chern, Zhulin Hu, Bohao Tang, Jiadi Su, Steffi Chern, Zhijie Deng, Pengfei Liu
Computer Vision
ProGuard: Towards Proactive Multimodal Safeguard
Shaohan Yu, Lijun Li, Chenyang Si et al. (+2 more)
The rapid
evolution of generative models has led to a continuous emergence of
multimodal safety risks, exposing the limitations of existing defense methods. To address these challenges, we propose ProGuard, a
vision-
language proactive guard that identifies and describes out-of-distribution (OOD) safety risks without the need for model adjustments required by traditional reactive approaches. We first construct a modality-balanced
dataset of 87K samples, each annotated with both binary safety labels and risk categories under a hierarchical
multimodal safety taxonomy, effectively mitigating modality bias and ensuring consistent moderation across text, image, and text-image inputs. Based on this
dataset, we train our
vision-
language base model purely through reinforcement learning (RL) to achieve efficient and concise
reasoning. To approximate proactive safety scenarios in a controlled setting, we further introduce an OOD safety category inference task and augment the RL objective with a synonym-bank-based similarity reward that encourages the model to generate concise descriptions for unseen unsafe categories. Experimental results show that ProGuard achieves performance comparable to closed-source large models on binary safety classification, substantially outperforms existing open-source guard models on unsafe content categorization. Most notably, ProGuard delivers a strong proactive moderation ability, improving OOD risk detection by 52.6% and OOD risk description by 64.8%.
Read the full abstract ▾
All authors Shaohan Yu, Lijun Li, Chenyang Si, Lu Sheng, Jing Shao
Computer Vision · Cross-listed from cs.CL
Instruction-Following Evaluation of Large Vision-Language Models
Daiki Shiono, Shumpei Miyawaki, Ryota Tanaka et al. (+1 more)
Following the initial flourishing of large
language models (LLMs), there has been a surge in proposed large
vision-
language models (LVLMs) that integrate LLMs with
vision capabilities. However, it has been observed that LVLMs, after tuning to visual instruction using commonly used training
datasets, often fail to exhibit the instruction-following ability that was present in the LLM before integration, leading to results in which they do not follow task instructions as expected. This study quantitatively demonstrates that LVLMs' instruction-following ability declines after fine-tuning and analyzes its underlying causes. In particular, we constructed new training
datasets highlighting whether the output format is specified. Then, we investigated how explicitly indicating the output format during fine-tuning affects LVLMs' instruction-following ability. Our quantitative evaluation confirmed that LVLMs' instruction-following ability declines after fine-tuning with commonly used
datasets. Furthermore, we found that LVLMs trained with
datasets, including instructions on output format, tend to follow instructions more accurately than models that do not. These findings suggest that including samples with instructions on output format during (visual) instruction tuning may help mitigate the decline in instruction-following abilities.
Read the full abstract ▾
All authors Daiki Shiono, Shumpei Miyawaki, Ryota Tanaka, Jun Suzuki
Computer Vision
Image Denoising Using Global and Local Circulant Representation
Zhaoming Kong, Xiaowei Yang and Jiahuan Zhang
The proliferation of imaging devices and countless image data generated every day impose an increasingly high demand on efficient and effective image denoising. In this paper, we establish a theoretical connection between principal component analysis (PCA) and the Haar transform under circulant representation, and present a computationally simple denoising
algorithm. The proposed method, termed Haar-tSVD, exploits a unified tensor singular value decomposition (t-SVD) projection combined with Haar transform to efficiently capture global and local patch correlations. Haar-tSVD operates as a one-step, parallelizable plug-and-play denoiser that eliminates the need for learning local bases, thereby striking a balance between denoising speed and performance. Besides, an adaptive noise estimation scheme is introduced to improve robustness according to
eigenvalue analysis of the circulant structure. To further enhance the performance under severe noise conditions, we integrate deep neural networks with Haar-tSVD based on the established Haar-PCA relationship. Experimental results on various denoising
datasets demonstrate the efficiency and effectiveness of proposed method for noise removal. Our code is publicly available at https://github.com/ZhaomingKong/Haar-tSVD.
Read the full abstract ▾
Computer Vision
ThinkGen: Generalized Thinking for Visual Generation
Siyu Jiao, Yiheng Lin, Yujie Zhong et al. (+9 more)
Recent progress in
Multimodal Large
Language Models (MLLMs) demonstrates that Chain-of-Thought (CoT)
reasoning enables systematic solutions to complex understanding tasks. However, its extension to generation tasks remains nascent and limited by scenario-specific mechanisms that hinder generalization and adaptation. In this work, we present ThinkGen, the first think-driven visual generation framework that explicitly leverages MLLM's CoT
reasoning in various generation scenarios. ThinkGen employs a decoupled architecture comprising a pretrained MLLM and a
Diffusion Transformer (DiT), wherein the MLLM generates tailored instructions based on user intent, and DiT produces high-quality images guided by these instructions. We further propose a separable GRPO-based training paradigm (SepGRPO), alternating reinforcement learning between the MLLM and DiT modules. This flexible design enables joint training across diverse
datasets, facilitating effective CoT
reasoning for a wide range of generative scenarios. Extensive experiments demonstrate that ThinkGen achieves robust, state-of-the-art performance across multiple generation
benchmarks. Code is available: https://github.com/jiaosiyuu/ThinkGen
Read the full abstract ▾
All authors Siyu Jiao, Yiheng Lin, Yujie Zhong, Qi She, Wei Zhou, Xiaohan Lan, Zilong Huang, Fei Yu, Yingchen Yu, Yunqing Zhao, Yao Zhao, Yunchao Wei
Computer Vision
RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from Scientific Literature
Hanzheng Li, Xi Fang, Yixuan Li et al. (+9 more)
The integration of
Multimodal Large
Language Models (MLLMs) into chemistry promises to revolutionize scientific discovery, yet their ability to comprehend the dense, graphical
language of reactions within authentic literature remains underexplored. Here, we introduce RxnBench, a multi-tiered
benchmark designed to rigorously evaluate MLLMs on chemical reaction understanding from scientific PDFs. RxnBench comprises two tasks: Single-Figure QA (SF-QA), which tests fine-grained visual perception and mechanistic
reasoning using 1,525 questions derived from 305 curated reaction schemes, and Full-Document QA (FD-QA), which challenges models to synthesize information from 108 articles, requiring cross-modal integration of text, schemes, and tables. Our evaluation of MLLMs reveals a critical capability gap: while models excel at extracting explicit text, they struggle with deep chemical logic and precise structural recognition. Notably, models with inference-time
reasoning significantly outperform standard architectures, yet none achieve 50% accuracy on FD-QA. These findings underscore the urgent need for domain-specific visual encoders and stronger
reasoning engines to advance autonomous AI chemists.
Read the full abstract ▾
All authors Hanzheng Li, Xi Fang, Yixuan Li, Chaozheng Huang, Junjie Wang, Xi Wang, Hongzhe Bai, Bojun Hao, Shenyu Lin, Huiqi Liang, Linfeng Zhang, Guolin Ke
Computer Vision
PurifyGen: A Risk-Discrimination and Semantic-Purification Model for Safe Text-to-Image Generation
Zongsheng Cao, Yangfan He, Anran Liu et al. (+3 more)
Recent advances in
diffusion models have notably enhanced text-to-image (T2I) generation quality, but they also raise the risk of generating unsafe content. Traditional safety methods like text blacklisting or harmful content classification have significant drawbacks: they can be easily circumvented or require extensive
datasets and extra training. To overcome these challenges, we introduce PurifyGen, a novel, training-free approach for safe T2I generation that retains the model's original weights. PurifyGen introduces a dual-stage strategy for
prompt purification. First, we evaluate the safety of each
token in a
prompt by computing its complementary semantic distance, which measures the semantic proximity between the
prompt tokens and concept
embeddings from predefined toxic and clean lists. This enables fine-grained
prompt classification without explicit keyword matching or retraining.
Tokens closer to toxic concepts are flagged as risky. Second, for risky
prompts, we apply a dual-space transformation: we project toxic-aligned
embeddings into the null space of the toxic concept
matrix, effectively removing harmful semantic components, and simultaneously align them into the range space of clean concepts. This dual
alignment purifies risky
prompts by both subtracting unsafe semantics and reinforcing safe ones, while retaining the original intent and coherence. We further define a
token-wise strategy to selectively replace only risky
token embeddings, ensuring minimal disruption to safe content. PurifyGen offers a plug-and-play solution with theoretical grounding and strong generalization to unseen
prompts and models. Extensive testing shows that PurifyGen surpasses current methods in reducing unsafe content across five
datasets and competes well with training-dependent approaches. The code can refer to https://github.com/AI-Researcher-Team/PurifyGen.
Read the full abstract ▾
All authors Zongsheng Cao, Yangfan He, Anran Liu, Jun Xie, Feng Chen, Zepeng Wang
Computer Vision
PathFound: An Agentic Multimodal Model Activating Evidence-seeking Pathological Diagnosis
Shengyi Hua, Jianfeng Wu, Tianle Shen et al. (+7 more)
Recent pathological foundation models have substantially advanced visual representation learning and
multimodal interaction. However, most models still rely on a static inference paradigm in which whole-slide images are processed once to produce predictions, without reassessment or targeted evidence acquisition under ambiguous diagnoses. This contrasts with clinical diagnostic workflows that refine hypotheses through repeated slide observations and further examination requests. We propose PathFound, an agentic
multimodal model designed to support evidence-seeking inference in pathological diagnosis. PathFound integrates the power of pathological visual foundation models,
vision-
language models, and
reasoning models trained with reinforcement learning to perform proactive information acquisition and diagnosis refinement by progressing through the initial diagnosis, evidence-seeking, and final decision stages. Across several large
multimodal models, adopting this strategy consistently improves diagnostic accuracy, indicating the effectiveness of evidence-seeking workflows in computational pathology. Among these models, PathFound achieves state-of-the-art diagnostic performance across diverse clinical scenarios and demonstrates strong potential to discover subtle details, such as
nuclear features and local invasions.
Read the full abstract ▾
All authors Shengyi Hua, Jianfeng Wu, Tianle Shen, Kangzhe Hu, Zhongzhen Huang, Shujuan Ni, Zhihong Zhang, Yuan Li, Zhe Wang, Xiaofan Zhang
Computer Vision
AnyMS: Bottom-up Attention Decoupling for Layout-guided and Training-free Multi-subject Customization
Binhe Yu, Zhen Wang, Kexin Li et al. (+6 more)
Multi-subject customization aims to synthesize multiple user-specified subjects into a coherent image. To address issues such as subjects missing or conflicts, recent works incorporate layout guidance to provide explicit spatial constraints. However, existing methods still struggle to balance three critical objectives: text
alignment, subject identity preservation, and layout control, while the reliance on additional training further limits their scalability and efficiency. In this paper, we present AnyMS, a novel training-free framework for layout-guided multi-subject customization. AnyMS leverages three input conditions: text
prompt, subject images, and layout constraints, and introduces a bottom-up dual-level
attention decoupling mechanism to harmonize their integration during generation. Specifically, global decoupling separates cross-
attention between textual and visual conditions to ensure text
alignment. Local decoupling confines each subject's
attention to its designated area, which prevents subject conflicts and thus guarantees identity preservation and layout control. Moreover, AnyMS employs pre-trained image adapters to extract subject-specific features aligned with the
diffusion model, removing the need for subject learning or adapter tuning. Extensive experiments demonstrate that AnyMS achieves state-of-the-art performance, supporting complex compositions and scaling to a larger number of subjects.
Read the full abstract ▾
All authors Binhe Yu, Zhen Wang, Kexin Li, Yuqian Yuan, Wenqiao Zhang, Long Chen, Juncheng Li, Jun Xiao, Yueting Zhuang
Computer Vision
Iterative Inference-time Scaling with Adaptive Frequency Steering for Image Super-Resolution
Hexin Zhang, Dong Li, Jie Huang et al. (+3 more)
Diffusion models have become a leading paradigm for image super-resolution (SR), but existing methods struggle to guarantee both the high-frequency perceptual quality and the low-frequency structural fidelity of generated images. Although inference-time scaling can theoretically improve this trade-off by allocating more computation, existing strategies remain suboptimal: reward-driven particle optimization often causes perceptual over-smoothing, while optimal-path search tends to lose structural consistency. To overcome these difficulties, we propose Iterative
Diffusion Inference-Time Scaling with Adaptive Frequency Steering (IAFS), a training-free framework that jointly leverages iterative refinement and frequency-aware particle fusion. IAFS addresses the challenge of balancing perceptual quality and structural fidelity by progressively refining the generated image through iterative correction of structural deviations. Simultaneously, it ensures effective frequency fusion by adaptively integrating high-frequency perceptual cues with low-frequency structural information, allowing for a more accurate and balanced reconstruction across different image details. Extensive experiments across multiple
diffusion-based SR models show that IAFS effectively resolves the perception-fidelity conflict, yielding consistently improved perceptual detail and structural accuracy, and outperforming existing inference-time scaling methods.
Read the full abstract ▾
All authors Hexin Zhang, Dong Li, Jie Huang, Bingzhou Wang, Xueyang Fu, Zhengjun Zha
Computer Vision
IdentityStory: Taming Your Identity-Preserving Generator for Human-Centric Story Generation
Donghao Zhou, Jingyu Lin, Guibao Shen et al. (+8 more)
Recent visual generative models enable story generation with consistent characters from text, but human-centric story generation faces additional challenges, such as maintaining detailed and diverse human face consistency and coordinating multiple characters across different images. This paper presents IdentityStory, a framework for human-centric story generation that ensures consistent character identity across multiple sequential images. By taming identity-preserving generators, the framework features two key components: Iterative Identity Discovery, which extracts cohesive character identities, and Re-denoising Identity Injection, which re-denoises images to inject identities while preserving desired context. Experiments on the ConsiStory-Human
benchmark demonstrate that IdentityStory outperforms existing methods, particularly in face consistency, and supports multi-character combinations. The framework also shows strong potential for applications such as infinite-length story generation and dynamic character composition.
Read the full abstract ▾
All authors Donghao Zhou, Jingyu Lin, Guibao Shen, Quande Liu, Jialin Gao, Lihao Liu, Lan Du, Cunjian Chen, Chi-Wing Fu, Xiaowei Hu, Pheng-Ann Heng