Computer Vision
Learning to Zoom Efficiently with a Contrastive Curriculum
Falko Helm and Iryna Gurevych
Using a zoom-in tool is an important foundational part of modern visual
agents, because it allows to efficiently handle tasks involving high-resolution images. Most previous methods need an extensive warm-start supervised fine-tuning phase for teaching models zoom-in. We show that this is not necessary by proposing a new intrinsic reward for learning tool use in MLLMs without the need for additional labels or warm-start SFT. Our InfoNCE-style reward uses a curriculum of increasingly hard negative tool calls as a contrastive training signal. Empirical experiments on $V^*$, HRBench and MME-RealWorld show that our approach is competitive while being more efficient. When used as a drop-in replacement for SFT, we even outperform all baselines. To directly measure the zoom-in ability of models, we further introduce the scalable synthetic Muffin&Chihuahua (M&C)
dataset. Each image consists of a grid with every
cell either showing a muffin or chihuahua. Leveraging the M&C
dataset's unique region of interest labels, we find that recall is the metric that most strongly correlates the zoom-in region with final task performance. Our model and code for reproduction is publicly available under https://github.com/UKPLab/emnlp2026-zoom-in
Read the full abstract ▾
Computer Vision
RoboTok: A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learning
Howard Qian, Yiting Chen, Yunfei Xie et al. (+6 more)
Robot learning increasingly depends on broad and diverse demonstrations, yet collecting
robot data remains expensive and difficult to scale across the wide range of real-world tasks. To address this bottleneck, we introduce RoboTok, a scalable data engine that uses a query human manipulation
video to retrieve manipulation-relevant internet demonstrations for training dexterous
robot policies. Specifically, we learn a latent motion space from 3D hand trajectories expressed in estimated actor-centered reference frames. This representation enables manipulation behaviors to be compared across variations in camera viewpoint, scene appearance, and actor occlusions, while remaining compact enough for efficient indexing and
retrieval over internet
video collections. We evaluate RoboTok against existing
robot-data
retrieval approaches using
retrieval metrics and downstream
robot policy performance, showing that RoboTok retrieves more manipulation-relevant demonstrations and improves downstream task success. In real-world
robot experiments, RoboTok-guided policies achieve a mean success improvement of 42.2 percentage points over the strongest baseline for each task, establishing hand-pose trajectory-aware
retrieval as a scalable way to leverage continuously growing web
video for
robot learning.
Read the full abstract ▾
All authors Howard Qian, Yiting Chen, Yunfei Xie, Kejia Ren, Podshara Chanrungmaneekul, Gaotian Wang, Bowen Wen, Chen Wei, Kaiyu Hang
Computer Vision · Cross-listed from physics.med-ph
Improving Clinical Target Volume Segmentation Accuracy using Anatomical Priors and Active Learning for the AGITG TOPGEAR Clinical Trial
Phillip Chlap, Mark Lee, Trevor Leong et al. (+11 more)
Training deep learning-based medical image
segmentation models is challenging with limited curated
datasets. For AGITG TOPGEAR, a gastric cancer trial, the Clinical Target Volume (CTV) is complex and defined by multiple anatomical landmarks, making upfront training data preparation difficult for an automated contour QA
segmentation model. We investigate anatomical priors, derived from surrounding organ
segmentations, to provide spatial context and improve TOPGEAR CTV
segmentation accuracy. We also evaluate active learning, iteratively expanding the training
dataset by selecting cases expected to improve performance. One hundred TOPGEAR CT scans were retrospectively analyzed. An initial set of 10 expert-contoured cases was used to train an nnU-Net model. TotalSegmentator generated a voxel-wise anatomical prior map from surrounding structures as an additional input channel. Active learning was simulated over four iterations, selecting cases by model uncertainty and
segmentation performance. All models used five-fold cross-validation for an ensemble uncertainty measure. Evaluation used a hold-out testing set of 50 cases. The anatomical prior improved CTV
segmentation accuracy, increasing mean Dice Similarity Coefficient (DSC) from 0.84 to 0.86. Active learning similarly improved performance to 0.86, with greatest benefit in the final round. Combining the anatomical prior with active learning achieved the highest accuracy, with a DSC of 0.87. Model uncertainty correlated with DSC, supporting its use in identifying suboptimal predictions and guiding active learning. Anatomical priors and active learning each improved CTV
segmentation accuracy and generalizability, with their combination achieving the best performance, supporting integration into
segmentation model development for automated contour QA in radiotherapy clinical trials.
Read the full abstract ▾
All authors Phillip Chlap, Mark Lee, Trevor Leong, Matthew Field, Jason Dowling, Hang Min, Julie Chu, Jennifer Tan, Phillip K. Tran, Tomas Kron, Annette Haworth, Martin A. Ebert, Shalini K. Vinod, Lois Holloway
Computer Vision · Cross-listed from cs.CL
Jina-OCR-v1: Efficient Document Parsing with Speculative Decoding and Dense Verifiable Rewards
Alejandro Barón García, Feng Wang, Emilia Garcia Casademont et al. (+1 more)
We present Jina-OCR-v1, an end-to-end document parsing model built to serve on low-budget GPUs. It combines the compressed-
vision encoder and the 3B mixture-of-experts decoder of DeepSeek-OCR, which activates about 570M parameters per
token, with a FastMTP speculative decoding head that shares a single draft block recursively across K=3 prediction steps. Greedy verification makes decoding lossless. Post-training combines instruction
alignment, robustness fine-tuning on difficult documents, and GRPO under dense verifiable rewards: deterministic formula, table, and structural checks that award partial credit. The training data mixes cleaned public corpora with targeted synthetic pages. At the default dynamic-resolution setting, Jina-OCR-v1 scores 91.14 on OmniDocBench v1.6 and 83.4 on olmOCR-Bench, and reaches the highest page throughput in our comparison at 2.57 pages per second. On a low-budget GPU such as the NVIDIA L4, FastMTP doubles decoding speed over greedy autoregressive decoding. The model is publicly available at https://huggingface.co/jinaai/jina-ocr-v1.
Read the full abstract ▾
All authors Alejandro Barón García, Feng Wang, Emilia Garcia Casademont, Han Xiao
Computer Vision
Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimization
Qingchan Zhu, Weihang You, Hanqi Jiang et al. (+3 more)
Visual
token pruning reduces the inference cost of
vision-
language models (VLMs), but most methods only ask which
tokens to keep. This retained-
token view can keep redundant high-scoring
tokens while leaving discarded evidence without a close representative. We propose CoverPruner, a training-free pruner that asks the complementary demand-side question: after a
token is removed, which surviving original
token represents it for the target VLM? CoverPruner formulates
pruning as Representational Coverage Maximization (RCM), covering the full projected visual-
token set with query-weighted demand. It instantiates RCM with projector-space coverage and a lightweight first-layer
attention probe. Across multiple VLM architectures and compression rates, CoverPruner achieves the best average accuracy among all compared methods, with the largest gains usually appearing under aggressive compression.
Read the full abstract ▾
All authors Qingchan Zhu, Weihang You, Hanqi Jiang, Changdi Yang, Tianming Liu, Geng Yuan
Computer Vision
VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement
Wenzhuo Xu, Yuchen Zhu, Chongjian Ge et al. (+8 more)
Visual fluency in generated
video does not imply physical reliability, and a scalar quality score alone is incapable of indicating the obligation a clip violates or the moment it fails. We present VeriPhy, an auditable physical-verification system in which a text-only planner compiles the
prompt into typed physical obligations and a statically validated execution plan before any frame is observed. During execution, observations gate and scope only declared calls to frozen low-level experts (e.g.,
segmentation and tracking, counting, eleven typed physical measurements over the resulting tracks, depth, OCR, and audio-event detection). Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state. Typed resolvers and fixed composition map usable records to a three-valued state (supported, contradicted, or unknown, surfaced as plausible, implausible, or abstain) with full provenance, so that every verdict is traceable to the evidence that produced it. We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in
prompt reference, space, and time. On a 149-clip core carrying 304 such records, VeriPhy accounts for 228, against 164 for a published question-decomposition evaluator given the same clips and the same claims. Recall alone does not separate it from prompting the same backbone monolithically, which reaches 222; what separates them is that each decision retains its evidence record and provenance, making the traces auditable one verdict at a time and usable as the interface through which a critic verdict could be written back into generation.
Read the full abstract ▾
All authors Wenzhuo Xu, Yuchen Zhu, Chongjian Ge, Xuan Shen, Jing Shi, Jason Kuen, Yongxin Chen, Molei Tao, Christopher McComb, Noelia Grande Gutiérrez, Jiuxiang Gu
Computer Vision · Cross-listed from cs.RO
Sensing Which Modality Matters: Evidence-Gated Regularization for Robust VLA Policies
Yue Yang, Diego Romeres, Chiori Hori et al. (+3 more)
Vision-
Language-Action (VLA) policies fuse
multimodal sensory inputs, but training on limited and homogeneous
robot demonstrations encourages spurious inter-sensor correlations rather than task-relevant signal, a failure we term modality
entanglement. Under real-world occlusions and distractors, this manifests as nuisance sensitivity to corruption of uninformative sensors and single-modality insufficiency when only one informative sensor remains intact. We propose Evidence-Gated Regularization (EGR), a modality-agnostic training objective that introduces zero inference-time overhead. EGR derives a per-frame and per-sensor task-relevance signal to gate two state-conditional consistency objectives: invariance on low-evidence sensors, and single-sensor sufficiency on high-evidence ones. We introduce a
benchmark based on BEHAVIOR-1K, comprising a fast inference-only diagnostic suite and 47 rollout-based skills targeting modality
entanglement. We validate EGR on this
benchmark and on two real-
robot setups with fundamentally different embodiments: a bi-manual setup with two Kinova arms and three RGB cameras, and a single-arm MELFA ASSISTA setup combining
vision and GelSight tactile sensors. EGR improves simulation success rates (SR) from 12.5% to 16.4% under full modalities (+31%), from 9.4% to 16.5% under uninformative-sensor corruption (+75%), and from 2.8% to 6.1% under single-sensor fallback (+120%). Under physical-object distractors, EGR boosts SR from 30% to 85% on the bi-manual setup (+183%) and from 55% to 70% on the tactile setup (+27%).
Read the full abstract ▾
All authors Yue Yang, Diego Romeres, Chiori Hori, Gedas Bertasius, Daniel Szafir, Siddarth Jain
Computer Vision
Beyond Small Patches: Black-Box Detection and Purification of Diverse Backdoor Triggers
Ahmed Abdelnaby and Mohamed Elmahallawy
Deep neural networks (DNNs) are increasingly deployed in real-world
vision systems, yet their predictions can be covertly manipulated by
backdoor attacks, in which malicious triggers cause targeted misclassification while preserving high clean accuracy. Existing defenses often rely on model internals, training data, or clean validation samples, making them difficult to deploy when only black-box access to a trained model is available. We propose TRIM (Trigger Removal by Identifying Manipulated Regions), a deployment-oriented black-box defense that detects and selectively removes
backdoor triggers at inference time without requiring model internals, training data, or clean samples. The key insight behind TRIM is to identify image regions that are responsible for anomalous model behavior and purify only those regions while preserving benign content. TRIM innovates via three key components: (i) region-based
segmentation with deep feature representations, (ii) adaptive trigger discovery through inpainting and
diffusion-based reconstruction to isolate regions responsible for misclassification—without assumptions about trigger type, shape, or location, and (iii) selective region purification that cleans poisoned regions while retaining benign content. To support practical deployment, TRIM further caches feature
embeddings of previously identified triggers, enabling efficient recognition and avoiding redundant detection and purification. Extensive experiments across diverse
datasets and
backdoor types, including blended, sparse, varying-size, and multiple triggers, show that TRIM consistently outperforms existing black-box defenses, reducing attack success rates (ASR) to as low as 1.16% while preserving clean accuracy of up to 87.87%. These results demonstrate that effective
backdoor mitigation is possible at inference time even when the defender has no access to any auxiliary data.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fields
Amir Mallak, Alaa Maalouf, Lior Wolf et al. (+2 more)
Neural fields (NFs) map continuous coordinates to signals such as color or density, but fast high-quality reconstruction from sparse observations remains difficult. Classical Neural Tangent Kernel (NTK) regression gives closed-form fits, yet it is fundamentally linear and cannot accumulate reusable task priors. We develop three
algorithms that address these gaps. NTK-KIP learns a distilled support set of coordinates (and optional labels) so that a finite NTK can inpaint large missing regions from little observed data, yielding a compact non-linear representation instead of a raw kernel solve. MetaQuill meta-learns a shared initialization for an INR so that new scenes can be adapted by updating only a small task-specific weight offset, which provides true feature learning and a reusable prior. Finally, MetaQuill-KIP fuses both ideas: it seeds the task with a KIP-style non-linear warm start, then refines only that small offset around the meta-learned initialization. MetaQuill-KIP achieves high-PSNR reconstructions and semantically plausible inpainting under very sparse observations, while requiring only lightweight per-instance adaptation, whereas
diffusion-style baselines typically depend on large pretrained generative priors and costly per-image tuning. This shows that NTK-driven neural fields can be made both non-linear and meta-learnable, narrowing the gap between analytic kernels and practical few-shot reconstruction.
Read the full abstract ▾
All authors Amir Mallak, Alaa Maalouf, Lior Wolf, Daniela Rus, Dan Rosenbaum
Computer Vision
SLIDEFORGE: An LLM Agent for Controllable Editing of Slides as Structured Artifacts
Haozhen Zheng, Fulin Wang, Tianhu Xiong et al. (+6 more)
Current AI
agents compellingly describe slides. However, AI-assisted slide editing requires more than understanding: the output must retain layout, style, component structure, and native editability. Towards, AI-assisted slide editing, existing
agents operate on screenshots or weak document representations and often fragment coherent visual units, rasterize editable content, or break layout. In contrast, for controllable slide editing, we introduce an agentic framework, SLIDEFORGE, which builds a Deck State Graph, an executable slide state that links visual decomposition, native pptx object structure, and perceptual organization. By recovering human-referable components while retaining fine-grained editable structure, SLIDEFORGE supports theme-preserving reconstruction through slide-native operations and rendered-state verification. We further introduce an evaluation paradigm for controllable slide transformation that jointly measures component recovery, preservation, restyling consistency, visual quality, and native editability. Experiments show that SLIDEFORGE outperforms direct prompting, screenshot-based
agents, and generic code-
agent baselines across these dimensions. Code is available at https://github.com/UIUC-MONET/SLIDEFORGE.
Read the full abstract ▾
All authors Haozhen Zheng, Fulin Wang, Tianhu Xiong, Yingjie Yu, Shengyi Qian, Hanchao Yu, Alex Schwing, Klara Nahrstedt, Mingyuan Wu
Computer Vision
WireSeg-32K: A Physics-Grounded Synthetic Dataset for Wire Instance Segmentation
Zilin Dai, Lehong Wang, Yi Yang et al. (+1 more)
Deformable linear objects such as wires and cables are difficult to segment because they are thin, highly deformable, and frequently self-occluded, while large-scale instance-level annotations are expensive to obtain in real scenes. Existing resources either focus on cable tracing or semantic
segmentation under constrained settings, or generate visually plausible images without physically grounded wire deformation. We present WireSeg-32k, a synthetic
dataset for wire instance
segmentation with 32,000 RGB images, instance masks, depth maps, and a complementary real-world test set with annotations. To generate this
dataset, we develop DeformX, a co-simulation pipeline that couples Cosserat-rod dynamics with photorealistic Isaac Sim rendering, enabling physically plausible, contact-consistent wire shapes, CAD-based wire assets, and diverse visually grounded scenes. As a simple baseline, LoRA fine-tuning SAM3 on WireSeg-32k alone improves real-world mAP@75 by 10.2% over the off-the-shelf model, showing that physically grounded synthetic data can transfer to real wire perception.
Read the full abstract ▾
All authors Zilin Dai, Lehong Wang, Yi Yang, Xiang Fei
Computer Vision · Cross-listed from eess.IV
Beyond Blur: A Semantic Tri-view Pipeline for Teledermatology Gradability via Skin Micro-relief
Robert Engel
Smartphone skin photographs are indispensable to teledermatology, yet assessing the diagnostic suitability of submitted cases (gradability) remains a critical bottleneck in mobile care workflows. Dermatologists routinely review multiple photographic views (regional, angled, and close-up) to identify consistent textural detail rather than relying on a single image. We present the Semantic Tri-view Pipeline, an interpretable architecture for automated teledermatology gradability screening that formalizes epidermal micro-relief as a computable biomarker of image quality. Using an expert-annotated subset of the public SCIN
dataset, we train a lightweight DeepLabV3+ model to segment micro-relief fidelity. These spatial masks are then aggregated across up to three case views with a logistic regression classifier, leveraging viewpoint redundancy to support robustness under uncontrolled smartphone acquisition. This approach learns context-aware, clinically intelligible heuristics, such as penalizing high-fidelity texture in regional distance views. Evaluated at a predefined 90% sensitivity operating point, the system's apparent errors largely reflect subjective clinical variance on borderline cases where clinicians rely on non-visual metadata. On SCIN, performance improves from an AUC of 0.81 (80.6% PPV) on variance-heavy majority-consensus cases to 0.96 (97.7% PPV) on optically unambiguous unanimous cases. Overall, this work delivers an interpretable,
privacy-by-design, edge-ready system that can provide real-time feedback during case submission to filter ungradable photo sets before review.
Read the full abstract ▾
Computer Vision
Solving the Needle-in-a-Haystack Problem in Mammography Vision-Language Model with Differentiable Subset Sampling
Young Seok Jeon, Beatrice Brown-Mulry, Rohan Satya Isaac et al. (+5 more)
There is growing interest in adopting CLIP-style
vision–
language model (VLM) pretraining for mammography. However, models that directly employ the standard CLIP architecture and training objective exhibit limited zero-shot performance in clinically important tasks such as cancer, finding-type, and BI-RADS predictions. We argue that this underwhelming performance is due to neglecting two characteristics of mammography data: (1) its high-res nature, and (2) homogeneity of radiology reports, largely driven by a predominance of negative/benign findings on examinations. We propose TopKSigLIP, a VLM designed to address these two limitations through a novel architecture and learning objectives. Instead of downscaling high-res mammography images to satisfy GPU
memory constraints, TopKSigLIP introduces TopK-Patch module that learns to sample a sparse set of high-res patches likely to contain lesions, sidestepping the resolution–batch size tradeoff of VLM training. The sampled patch locations additionally serve as a built-in localization tool. To address report homogeneity, we replace the contrastive loss, which falsely repels semantically similar pairs, with a Sup-sigmoid loss. Sup-sigmoid loss extends the sigmoid loss from SigLIP with soft labels derived from structured data. TopKSigLIP outperforms existing open-source mammography and general medical VLMs on both internal and external
benchmarks on density assessment, BI-RADS classification, finding subtyping, and cancer prediction under zero-shot evaluation. TopKSigLIP remains competitive under linear probing despite using a significantly smaller
vision encoder and smaller training batches than baselines. The TopK-Patch module additionally achieves superior lesion localization over post-hoc Grad-CAM. Code and weights are made public:https://github.com/Youngseok0001/TopKSigLIP.
Read the full abstract ▾
All authors Young Seok Jeon, Beatrice Brown-Mulry, Rohan Satya Isaac, Anjana Dissanayaka, Theo Dapamede, Mohammadreza Chavoshi, Judy Gichoya, Hari Trivedi
Computer Vision
Exemplar: Few-Shot Biomedical Segmentation That Refits Exactly in Seconds
Michal Průšek, Adam Novozámský and Filip Šroubek
Segmenting a new microscopy
dataset usually means training a deep network on its annotations, and training again for every added image. Exemplar solves a
segmentation readout in closed form and updates it exactly with each new mask in seconds, as accurately as networks trained for minutes. Its linear readout combines frozen DINOv3 features, read at a magnification chosen from the masks, with classical filter responses at native resolution. Its regularisation and its pixel weighting are chosen exactly by leaving out one annotated image at a time, so none is set aside for validation. On ten biomedical
datasets from
cells to vessels and filaments, Exemplar is more accurate than every few-shot method that only applies a pretrained model, and on average than nnU-Net trained on the same masks. A fine-tuned Cellpose-SAM, the strongest baseline, is as accurate on average, but each of its fits takes a median of 25 minutes against 8.5 seconds, and it cannot be updated exactly.
Read the full abstract ▾
Computer Vision
Position: Unlabeled IS NOT Equal to No Human Supervision in Visual Learning
Dong Lao
This position paper argues that the absence of labels does not imply the absence of human supervision in visual learning, and urges the research community to identify sources of supervision more explicitly. Many recent methods in computer
vision build upon representations learned from large-scale unlabeled data, and are therefore grouped under the same umbrella term “unsupervised.” However, different data curation schemes and training objectives embed substantially different human priors on which models rely, and we argue that one “unsupervised” umbrella term is no longer capturing these distinctions. This ambiguity makes it harder to compare unsupervised learning research conducted under different assumptions, coinciding with a sharp decline in papers titled with “unsupervised” in flagship computer
vision conferences since 2021, despite continued growth of the field. While we fully embrace pre-training as a strong foundation for modern computer
vision, we advocate for a community-level effort toward greater conceptual clarity: authors are encouraged to disclose priors in data selection and learning objectives, and to specify which components of a learning pipeline depend on which assumptions. Standardized disclosure practices can improve academic communication, ensure fairer comparisons, and preserve methodological diversity in unsupervised learning.
Read the full abstract ▾
Computer Vision
IDSPACE: A Novel Document Generator for Reliable Evaluation of Digital Identity Verification Systems [Extended Technical Report]
Lulu Xie, Yancheng Wang, Kanchan Chowdhury et al. (+3 more)
As services move online, trust institutions such as banks, lenders, and governments must verify the identity of remote users. Fraud detection tools are widely available, but evaluating and fine-tuning them remains difficult because identity documents are sensitive and therefore scarce. Synthetic data generation offers a path forward, and demand is clear: our prior work in this area has been downloaded over $11{,}000$ times (aggregated from eight parts). We introduce IDSpace, extending this line of research in three directions. First, we propose model-guided
Bayesian optimization, which tunes generation parameters to maximize both visual similarity and prediction consistency with target-domain models given only a few samples from a target domain. Second, we decouple user-specified metadata (demographics, fraud patterns, capture device) from automatically tuned control parameters (font styles, noise levels, image quality), allowing users to configure evaluations without low-level expertise. Third, we expand beyond template images to support scanned and mobile-captured documents. Experiments show IDSpace improves evaluation consistency by $15-45\%$ over baselines including CycleGAN,
diffusion inpainting, and non-guided optimization, using only a few real samples, while improving training accuracy by up to $9\%$ and SSIM similarity with the target domain by $10\%$. We also released a new
dataset consisting of $359{,}240$ high-quality synthetic documents across ten European ID types.
Read the full abstract ▾
All authors Lulu Xie, Yancheng Wang, Kanchan Chowdhury, Rolando Garcia, Yingzhen Yang, Jia Zou
Computer Vision · Cross-listed from eess.IV
Learning from Scarce Labels: Multi-View Echocardiography for Ejection Fraction Prediction
Zhiyuan Gao, Dominic Yurk and Yaser S. Abu-Mostafa
We present, to the best of our knowledge, the first publicly available resource for predicting left ventricular ejection fraction (EF) from parasternal long-axis (PLAX) echocardiography. Because no PLAX-EF
datasets previously existed, our work focuses on an innovative data generation strategy to overcome this scarcity. By leveraging a time-based correlation between clinical notes and echocardiographic
videos, combined with fine-tuning view classifiers and proxy labeling, we created a labeled
dataset of over 25,000 PLAX
videos. This enables us to train the first reproducible PLAX EF model, achieving a mean absolute error (MAE) of 6.86%. Given that apical four-chamber (A4C) methods, the clinical standard, report MAE values of 6%-7%, our results demonstrate that EF estimation from PLAX views is both feasible and clinically relevant. This surpasses the performance of existing methods and provides a clinically relevant solution for situations where apical views may not be feasible. Going further, we demonstrate that combining PLAX and A4C predictions via simple unweighted late fusion improves both single-view baselines to a 6.37% MAE, underscoring the value of multi-view integration. To promote continued research, we release the
dataset labels, trained models, and runnable demos on GitHub, Hugging Face, and Google Colab: https://github.com/Jeffrey4899/PLAX_EF_Labels_202509
Read the full abstract ▾
Computer Vision
SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models
Junchao Huang, Guian Fang, Shengju Qian et al. (+15 more)
We introduce SolarWM, a fully open foundation for building interactive
video world models from data preparation through long-horizon inference. Training across heterogeneous data sources and
video backbones is challenging:
datasets differ in temporal scale, camera geometry, visual quality, motion, and captioning styles, while
video generators use distinct representations and architectures. Naive data mixing and model-specific implementations therefore produce inconsistent supervision and make results difficult to reproduce and compare. SolarWM addresses this coupling with a reconfigurable multi-source data engine and a backbone-native adaptation framework. The engine converts 1.43 million canonical clips from 10
datasets into a unified, frame-aligned contract covering visual observations, metric camera geometry, captions, quality metadata, selection decisions, and provenance, while decoupling source processing from mixture construction. Under shared camera-conditioning, training, and inference interfaces, we instantiate four 5B–33B models based on Wan2.2, LTX-2.5, and MiniMax-H3 while preserving their native representations and objectives. A unified three-stage recipe combines bidirectional adaptation, teacher-forced autoregressive initialization, and distribution matching
distillation. The resulting causal models enable real-time interaction over rollouts ranging from minutes to hours after being trained on only 5s sequences. By releasing the resulting data, pipeline, recipes, weights, and framework, SolarWM provides a reproducible and extensible foundation for interactive world-model research.
Read the full abstract ▾
All authors Junchao Huang, Guian Fang, Shengju Qian, Xianghao Kong, Zhuoran Zhao, Wei Huang, Yihua Du, Zixin Zhang, Justin Cui, Yuchao Gu, Yukang Chen, Xinting Hu, Tianyu He, Shaoshuai Shi, Zhuotao Tian, Xin Wang, Mike Zheng Shou, Li Jiang
Computer Vision
Thinking in Pictures: A Systematic Benchmark for Reasoning-driven Image Generation
Yutong Liu, Nan Huang, Xu Cao et al. (+1 more)
Recent advancements in unified generative models (UGMs) and world simulators have achieved unprecedented results in visual perception and synthesis. However, these models primarily rely on surface-level event
alignment, leaving the capacity for high-level visual
reasoning underexplored. True visual generative intelligence demands "
Reasoning-to-Generation", an ability to infer latent rules from visual inputs and manifest solutions through precise, logically constrained visual outcomes. We introduce RIG-BENCH, a novel comprehensive
benchmark that systematically evaluates
Reasoning-driven Image Generation (RIG) across four cognitively demanding domains: Concept-based, Transformation-based, Pattern & Structure, and Scenario-based. Featuring 2000 curated samples, RIG-BENCH serves as a rigorous stress test for RIG. Our extensive evaluations of state-of-the-art UGMs and image/
video generation models reveal a significant
reasoning-generation gap, wherein models frequently produce locally plausible but globally illogical outputs. RIG-BENCH provides a vital diagnostic framework to guide the development of next-generation, logically grounded UGMs and world simulators.
Read the full abstract ▾
All authors Yutong Liu, Nan Huang, Xu Cao, James M. Rehg
Computer Vision
PlantC2USeg: Cross-Scale Consistent Pre-Training for Few-Shot Unified Plant Point Cloud Segmentation
Yu Tian, Xintong Jiang, Jan Franklin Adamowski et al. (+2 more)
Modern crop breeding demands precise organ-level analysis for trait quantification, making plant point cloud
segmentation (PPCS) increasingly important. However, conventional deep learning approaches rely heavily on densely annotated
datasets that are labor-intensive to acquire. Unified PPCS adaptation from distribution-shifted examples with minimal additional training remains challenging. To address this, we propose PlantC2USeg, a deep transfer learning framework featuring cross-scale consistency learning to explicitly align features across spatial scales and an information-restricted decoding strategy that prevents reconstruction shortcuts and promotes robust adaptation. The resulting pre-training enables stable few-shot generalization across species and sensing conditions, while unified fine-tuning with inherited thresholds further reduces adaptation overhead. Under full supervision on Soybean3D, PlantC2USeg achieves the highest semantic IoU and instance mWCov among compared methods, at 91.91% and 94.62%. With 20 labeled samples, it leads both metrics at 89.78% and 90.27%; with only 10 samples, it retains the highest mWCov of 83.23% while achieving 83.19% IoU. Across HR3D, 10-shot transfer to tobacco, tomato, and sorghum averages 78.41% IoU and 79.42% mWCov, while 22-shot transfer to SYAU-Maize achieves the highest IoU and mRec at 92.75% and 93.51%. Furthermore, a leading category-averaged mIoU of 85.0% on ShapeNet Part demonstrates the framework's capability to handle diverse shape variations beyond agricultural domains. These results demonstrate that PlantC2USeg reduces overall adaptation effort under distribution shifts, enabling scalable plant phenotyping and transferable 3D representation learning beyond agriculture.
Read the full abstract ▾
All authors Yu Tian, Xintong Jiang, Jan Franklin Adamowski, Shiv O. Prasher, Shangpeng Sun
Computer Vision
MuyBridge: Mobile Human Center-of-Mass Estimation from Monocular Video via Sparse Fusion
Aidan Bradshaw, Marco Giordano, David Rode et al. (+8 more)
The 3D center of mass (CoM) is a primary quantity in the biomechanical analysis of sport, rehabilitation, and clinical movement, yet existing 3D pose tracking, mesh recovery, and multi-view triangulation methods either optimize 3D keypoint accuracy without anatomical constraints or carry compute and capture infrastructure too heavy to deploy where CoM tracking is most useful. As a result, the metric CoM remains difficult for coaches and movement analysts to measure from a single camera where athletes train and compete. In this work, we introduce MuyBridge, an on-device system that estimates the athlete's segmental center of mass trajectory from a single phone camera
video stream. MuyBridge couples a compact 2D pose network and a distilled single-step
monocular depth network through an analytic metric fusion that uses anatomical and physical priors to anchor the metric CoM, requiring no 3D or task-specific supervision. Evaluated on the athletic movements of AthletePose3D (running, track and field, and figure skating), MuyBridge achieves 33-41 mm vertical CoM error and 2.3-6.6% absolute-relative range error (AbsRel) under a one-time calibration, and produces CoM estimates at the 63 FPS pose-estimation rate using asynchronous 2.86 Hz depth updates on iPhone 15. Code is available at: https://github.com/Abradshaw1/Muybridge
Read the full abstract ▾
All authors Aidan Bradshaw, Marco Giordano, David Rode, Andreas Habersack, Elif Basokur, Annika Kruse, Markus Tilp, Michele Magno, Peter Wolf, Luca Benini, Christoph Leitner
Computer Vision
RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation
Xiaolei Lang, Ze Kang, Zehao Huang et al. (+1 more)
Novel view synthesis from sparse inputs requires both
geometric grounding from the observed views and generative priors of unobserved regions, motivating recent hybrid methods that combine reconstruction and generation. However, existing methods bridge the two with rendered images or explicit 3D representations such as point maps or 3D Gaussians. Generation is thus conditioned on a lossy and imperfect projection of the scene, inheriting its errors, and reconstruction receives no signal from generation to correct them. We present RoGe, an end-to-end unified reconstruction and generation framework that removes this explicit bridge. It targets roaming within a scene anchored by sparse views: given a few posed images and a camera trajectory, it synthesizes a temporally coherent
video along that trajectory. From the sparse input views, RoGe builds an implicit scene representation with a feed-forward reconstruction model, and queries it with camera rays to obtain per-view
geometric features. These features are injected into a
video diffusion model as conditioning, without any explicit 3D intermediate. Both modules are trained jointly, so the generation objective directly shapes its own
geometric conditioning. We conduct extensive experiments, where RoGe outperforms reconstruction-based, generation-based, and hybrid baselines in terms of image-level quality and
video-level temporal and
geometric consistency. Ablations confirm that ray-queried implicit features outperform both raw reconstruction
tokens and rendered RGB as conditioning, and that joint training brings further gains. Our code will be released on https://jerry-locker.github.io/roge/.
Read the full abstract ▾
All authors Xiaolei Lang, Ze Kang, Zehao Huang, Naiyan Wang
Computer Vision
Efficient All-in-One Weather Restoration using Spectral Harmonization
Paula Garrido-Mellado, Daniel Feijoo, Yuning Cui et al. (+2 more)
Adverse weather conditions such as rain, haze, and snow significantly degrade image quality, posing challenges for both human perception and physical AI. Existing restoration methods require large computational budgets, struggling to process high-resolution images and handle different degradations. In this paper, we present Frequency Reconstruction via Spectral Harmonization, a novel lightweight all-in-one restoration method that explicitly decomposes feature representations into high- and low-frequency components at each scale of a hierarchical encoder-decoder architecture. By combining spectral decomposition with spatial processing through
Fourier-based skip connections, FReSH-IR captures complementary frequency information without sacrificing spatial detail. Our approach achieves similar restoration quality with 80% fewer parameters and operations than
transformer-based models. Extensive experiments demonstrate that our method offers a great efficiency-performance trade-off, highlighting its practical applications in constrained-resource systems.
Read the full abstract ▾
All authors Paula Garrido-Mellado, Daniel Feijoo, Yuning Cui, Alvaro Garcia, Marcos V. Conde
Computer Vision
Benchmarking RAW and RGB Restoration in Image Signal Processors
Zihao Lu, Radu Timofte and Marcos V. Conde
Modern cameras transform RAW sensor measurements into sRGB images through an image signal processor (ISP). We
benchmark two placements for blind restoration around a fixed ISP: (A) pre-ISP restoration in the RAW domain and (B) post-ISP restoration in the sRGB domain. The
benchmark covers four smartphone device groups, two learned ISPs, three degradation regimes–noise, blur, and joint noise and blur–, and several representative RAW and RGB restoration models. Our results show that placement alone does not determine performance. The RAW restoration strategy outperforms the best generic RGB restoration models. However, RGB restoration models trained considering the ISP transformations, achieve the best overall performance. Our novel
benchmark demonstrates that the image reconstruction performance strongly depends on the
alignment between the restoration model and the target imaging pipeline. We consequently recommend reporting restoration placement and ISP-aware supervision as key experimental factors. Our code is available at https://github.com/mv-lab/AISP
Read the full abstract ▾
Computer Vision
GDB-Reward: From Evaluation Metrics to Training Rewards for Graphic Design
Adrienne Deganutti, Purvanshi Mehta, Simon Hadfield et al. (+1 more)
Text-to-image models excel at natural image synthesis but struggle with graphic design, where success depends on satisfying precise constraints on typography, layout, color, and visual communication. While
prompt optimization offers an attractive alternative to expensive
diffusion model fine-tuning, learning
prompts for frozen image generators requires informative reward functions despite the entirely non-differentiable generation process. Reinforcement learning does not require differentiable objectives; it requires only scalar rewards capable of ranking candidate outputs. This raises a simple question: can design evaluation metrics themselves become reinforcement learning rewards? Our central contribution is GDB-Reward, a framework that systematically transforms heterogeneous graphic design evaluation metrics into a unified reinforcement learning reward. Experiments demonstrate that GDB-Reward provides an effective optimization objective, substantially improving adherence to the design specification in perceptual quality, rendering fidelity, and spatial accuracy while keeping the image generator entirely frozen. More broadly, our results demonstrate that heterogeneous, non-differentiable evaluation metrics can move beyond passive benchmarking to become effective optimization objectives for reinforcement learning in domains where differentiable supervision is unavailable.
Read the full abstract ▾
All authors Adrienne Deganutti, Purvanshi Mehta, Simon Hadfield, Andrew Gilbert
Computer Vision
AutoCompass: Accurate Visual Localization on Public Maps by Learning from Weak Labels
Javier Tirado-Garín, Alan Savio Paul, Shuai Chen et al. (+5 more)
Neural map matchers estimate an image's 3-DoF pose relative to a 2D map. These models are trained on large-scale
datasets of geo-referenced images, whose position and heading labels often contain noise that affects the trained models. To address this, we present AutoCompass, a supervision approach for training neural map matchers from inaccurate absolute pose labels. First, we show that heading labels are unnecessary: trained from raw GPS labels, models learn to predict accurate headings, automatically. Second, defining a tolerance region around raw GPS improves positional accuracy. Third, if available, our supervision uses relative poses between training images, obtained via SLAM or SfM, which provide a more accurate training signal. Across driving and
egocentric benchmarks, AutoCompass consistently outperforms counterparts trained with the usual strong reliance on absolute pose labels.
Read the full abstract ▾
All authors Javier Tirado-Garín, Alan Savio Paul, Shuai Chen, Axel Barroso-Laguna, Tommaso Cavallari, Daniyar Turmukhambetov, Victor Adrian Prisacariu, Eric Brachmann
Computer Vision
ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding
Jitai Hao, Ke Yang, Di Yan et al. (+3 more)
Streaming
video understanding is a critical capability for real-world applications, including
embodied intelligence, autonomous driving, industrial monitoring, surveillance and early warning, and wearable assistants. However, processing continuous
video streams with
multimodal large
language models (MLLMs) is computationally expensive. Existing efforts have explored reducing streaming overhead through visual
token pruning,
token merging, quantization, on-demand frame
retrieval, and context offloading. However, most existing methods overlook the dimension of model depth. Repeatedly executing full-depth MLLM prefill over incoming frames is prohibitively expensive, incurring substantial computational overhead and causing the KV cache to grow at a rate directly proportional to the prefill depth. To address these challenges, we propose ShallowStream, a novel framework that leverages the shallow layers of an MLLM to simultaneously perform frame encoding and
retrieval index building. During stream processing, ShallowStream maintains an always-on lightweight index using the KV cache of shallow layers. During query-time answering, we leverage the
attention scores generated by the shallow layers to score context frames and employ a diversity-aware selection strategy to retrieve precise and comprehensive evidence. ShallowStream achieves performance on par with the strongest existing streaming methods, while reducing per-frame prefill latency and 10-second end-to-end latency by up to 52.1x and 11.9x, respectively. Our code is available at https://github.com/CURRENTF/ShallowStream.
Read the full abstract ▾
All authors Jitai Hao, Ke Yang, Di Yan, Fan Liu, Qiang Huang, Jun Yu
Computer Vision
Video-Based Palm-Vein Authentication under Challenging Conditions
Xiaofeng Yan, Kechen Liu, Abhilash Venkatesh et al. (+3 more)
Palm-vein biometrics are increasingly used for secure, contactless authentication. Yet real-world deployment exposes them to surface noise (sweat, dirt), illumination and motion variation, and temperature-driven changes in vascular visibility, which remain underexplored for lack of data captured under such conditions. To study these effects, we introduce the Columbia University Palm-vein (CUP)
dataset, to our knowledge the first public
video-based palm-vein
dataset. CUP records every palm under four surface conditions (a clean baseline, warm, wet, and dirty) and pairs each subject with physiological and demographic metadata. On it we
benchmark twenty-one recognizers spanning static,
video, and multi-frame aggregation architectures. Models that verify reliably on clean palms lose most of their accuracy on dirty ones, and the mean equal error rate (EER) roughly quadruples. We recover much of that robustness along both axes of the capture. Temporally, a consensus over the few frames the sensor already returns cancels transient corruption; spatially, a test-time matcher that adds no learned parameters fuses the global cosine with a saliency-steered region-level optimal transport that routes the comparison around corrupted regions. The full design leads on every surface of CUP in EER, TAR@FAR=0.01, and Rank-1, at 4.3M parameters and 3.1 GFLOPs, a fraction of the
video models' cost. Attached to four frozen state-of-the-art backbones it cuts their mean EER by 29-37% without retraining, and on four public single-image
datasets the regional matching alone still helps. A preliminary audit across ten demographic and physiological traits finds two warm-condition gaps, along body water and gender, that survive multiple-comparison correction. CUP will be released for non-commercial research use at https://github.com/MobileX-CU/CUP_v1 upon publication.
Read the full abstract ▾
All authors Xiaofeng Yan, Kechen Liu, Abhilash Venkatesh, Cathy Zhang, Xia Zhou, Salvatore Stolfo
Computer Vision
Multi-Tool Image Editing Attribution in Facial Forgery
Sheng Liu, Qiang Sheng, Danding Wang et al. (+3 more)
As generative AI tools become increasingly powerful and easy to use, people can easily edit portrait images with a
prompt, necessitating the task of image editing attribution, which predicts the involved editing tools from the given image. Existing attribution methods hold the single-tool assumption and can only attribute a specific editing tool, but struggle to handle the more complex and increasingly common multi-tool editing scenarios, where artifacts left by different editing tools are composite and overlapped. To address this gap, we explore Multi-Tool Image Editing Attribution (MIEA), which aims to identify multiple editing tools involved in a multi-tool edited facial image. To simulate the real-life editing operations on facial images, we then construct a new
dataset, MultiEdit, which contains 500k+ edited facial images and covers six types of editing tools that support face swapping (Deepfake) and various facial enhancements. Inspired by the findings from data analysis, we design DPEC, a multi-tool attribution method that can capture distinguishable, locality-aware editing tool traces from both spatial and frequency domains with the support of an error-based curriculum learning strategy. Experiments show \Method\ outperforms nine methods for facial images edited in at most five steps.
Read the full abstract ▾
All authors Sheng Liu, Qiang Sheng, Danding Wang, Yu Li, Chenming Zhou, Juan Cao
Computer Vision
Balancing Frequencies and Pixels in Flow Matching
Lucas Degeorge, Paul Couairon, Arijit Ghosh et al. (+3 more)
Natural images follow a $1/f^2$ spectral distribution: most signal energy lies in the low spatial frequencies, while the perceptually important structures such as textures and edges occupy sparse high-frequency bands. Pixel-space reconstruction objectives, however, treat all spatial errors uniformly, causing low frequencies to dominate the optimization signal and delaying the learning of fine-scale details. In this work, we identify this objective-level spectral imbalance as a key inefficiency in training pixel-space flow models. To address it, we propose a Focal Log-Frequency Loss (f-loss), a spectrally balanced objective that equalizes the learning signal across frequencies, emphasizing high-frequency components that are otherwise underrepresented in pixel-space objectives. Building on this, we introduce a simple training strategy that combines frequency and pixel supervision: we first emphasize frequency-domain learning early to capture all frequencies, and then transition to standard pixel-space v-loss for spatial refinement. This balancing mitigates the low-frequency bias of pixel losses and aligns the training signal with the evolving needs of the model. Our approach is conceptually simple, requires no architectural changes, and acts as a drop-in replacement for flow matching losses. Across multiple model scales, it accelerates convergence by up to 40% while consistently improving FID and perceptual fidelity.
Read the full abstract ▾
All authors Lucas Degeorge, Paul Couairon, Arijit Ghosh, Alexei A. Efros, David Picard, Vicky Kalogeiton