Computer Vision
NIV: Neural Axis Variations for Variable Font Generation
Nadav Benedek, Ariel Shamir and Ohad Fried
Variable fonts enable continuous variation of glyph geometry along semantic design axes such as weight, width, slant, and optical size. However, constructing a variable font from a static font remains a labor-intensive process requiring expert typographic design and manual specification of glyph variation data. We introduce NIV (Neural Axis Variations), a method that automatically converts a static font into a fully functional variable font. Given glyph outlines and a set of desired design axes, NIV predicts per-point displacements. The model operates directly on vector glyph geometry and employs a novel Property
Embedding mechanism that captures interactions between multiple axes, enabling consistent multi-axis variation within a unified framework. We train NIV on a newly constructed
dataset derived from variable Google Fonts, comprising over one million variation tuples. The resulting model generalizes across unseen code points, unseen font styles, high-complexity CJK glyphs, and even out-of-distribution handwriting inputs. The generated outputs are standard variable font files supporting continuous interpolation via existing rendering engines. To facilitate research, we release the
dataset, the complete training and inference implementation, and trained models at https://github.com/ndvbd/NIV. Beyond typography, our approach demonstrates how structured
geometric objects with continuous parametric variation can be synthesized using neural deformations.
Read the full abstract ▾
Computer Vision
VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understanding
Lin Fu, Zheyuan Yang, Yang Wang et al. (+3 more)
We introduce VideoKR, the first large-scale training corpus specifically designed to strengthen knowledge- and
reasoning-intensive
video understanding. It comprises 315K
video reasoning examples over 145K newly collected, CC-licensed, expert-domain
videos. We develop a human-in-the-loop, skill-oriented example generation pipeline that targets progressively deeper
video reasoning capabilities while ensuring the difficulty, diversity, and reliability of both the examples and their CoT rationales. We also curate VideoKR-Eval, a new expert-annotated
benchmark where questions require genuine
video understanding and knowledge-intensive
reasoning rather than textual shortcuts. Our experiments show that, under a standard SFT$\rightarrow$GRPO pipeline, models post-trained on VideoKR outperform prior post-training approaches on knowledge-intensive
video reasoning while remaining competitive on general
video reasoning, highlighting data design as a key driver of progress in
video reasoning. We further conduct comprehensive ablations to isolate the contributions of VideoKR, providing actionable insights for future work.
Read the full abstract ▾
All authors Lin Fu, Zheyuan Yang, Yang Wang, Tingyu Song, Arman Cohan, Yilun Zhao
Computer Vision · Cross-listed from eess.IV
Oklch+: A Three-Parameter Extension of Oklab for Improved Color Difference Prediction
Naoyuki Uchida
Oklab and its cylindrical representation Oklch are widely adopted in interpolation and design workflows as perceptually motivated color spaces, but their color difference prediction accuracy falls short of CIEDE2000. We propose Oklch+, a three-parameter extension of Oklab comprising a power transformation on the L-axis and a Naka-Rushton compression on the C-axis, with Euclidean distance computed in the resulting transformed Oklab coordinates. The Naka-Rushton function is bounded in [0,1], reflecting the saturating nature of chroma sensitivity at high colorimetric values. Evaluated on COMBVD – 3,813 suprathreshold color difference pairs spanning six independent experimental
datasets – Oklch+ achieves STRESS = 29.09, closely matching CIEDE2000 (29.13; difference = 0.04), using only three parameters optimized against color difference data compared to approximately 17 for CIEDE2000. Cross-validation on a held-out BFD-P D65 subset (2,028 pairs) confirms generalization (STRESS = 26.14), with Oklch+ substantially outperforming Oklab (51.45) and achieving STRESS comparable to CIEDE2000 (24.12) on the held-out set. Improvement over Oklab (47.35) is confirmed across all six COMBVD sub-
datasets. Because Oklch+ defines a coordinate system in which Euclidean distance approximates perceptual distance, linear interpolation in the transformed space offers substantially improved perceptual uniformity relative to Oklab. Current evaluation is limited to the sRGB-centered COMBVD
dataset; validation in high-chroma regions with empirical observer-rated discrimination data remains future work.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Flash-WAM: Modality-Aware Distillation for World Action Models
Arman Akbari, Ci Zhang, Arash Akbari et al. (+6 more)
World-action models (WAMs) jointly generate future
video and
robot actions through iterative
diffusion, achieving strong performance on manipulation
benchmarks but requiring tens of denoising steps, a cost that precludes real-time control. Step
distillation has emerged as the natural remedy, but off-the-shelf methods break down in the joint
video-action setting because
video and action streams use different SNR-shifted noise schedules and reach training with substantially different marginal noise distributions, an asymmetry that single-modality
distillation methods cannot accommodate. We introduce Flash-WAM, a modality-aware step-
distillation framework inspired by consistency
distillation that selects the consistency function for each modality to match its noise regime: a linear-gradient-scaling parametrization for the action stream's low-noise regime, paired with a variance-preserving parametrization for the
video stream's high-noise regime, grounded in a structural analysis of the consistency-function family that characterizes the achievable gradient scaling under the consistency boundary condition. Instantiated on LingBot-VA, Flash-WAM compresses inference to a single step in each modality. On RoboTwin 2.0, this reduces per-chunk latency from $8.1$ seconds to $348$ ms on NVIDIA L40S, a $23{\times}$ speedup that enables real-time inference. Flash-WAM preserves task success on simulation
benchmarks ($85.5\%$ RoboTwin 2.0, $95.7\%$ LIBERO) and substantially recovers real-world performance ($60\%$ average on a Unitree G1
humanoid robot), while naive consistency
distillation drops to $24\%$ at the same step budget.
Read the full abstract ▾
All authors Arman Akbari, Ci Zhang, Arash Akbari, Lin Zhao, Yixiao Chen, Weiwei Chen, Xuan Zhang, Geng Yuan, Yanzhi Wang
Computer Vision
Controllable Dynamic 3D Shape Generation via 3D Trajectories and Text
Jaeyeong Kim, Ines Kim, Jahyeok Koo et al. (+1 more)
We introduce T2Mo, a feed-forward framework for controllable dynamic 3D shape generation conditioned on 3D trajectories and text. Due to the inherent ambiguity of
language, generating precisely intended motions using text alone remains challenging. To address this, we adopt 3D trajectories as controllable spatial guidance, specifying the exact paths along which selected points should move. By combining both, T2Mo generates object motions that spatially adhere to the given trajectories while globally reflecting the text semantics. To robustly handle trajectory inputs with arbitrary configurations, ranging from dense to sparse and unevenly distributed, we further propose a shape-grounded trajectory
embedding that maps an input trajectory set into a shape-aware
token set covering the entire object. We conduct extensive comparisons against text-based baselines and cascaded
video-based baselines that combine trajectory-guided
video generation with
video-to-dynamic mesh generation. Quantitative and qualitative evaluations, along with user studies, demonstrate that our approach produces motions that more faithfully follow the given
prompts with higher expressiveness while preserving motion quality.
Read the full abstract ▾
All authors Jaeyeong Kim, Ines Kim, Jahyeok Koo, Seungryong Kim
Computer Vision
An Open-Source Two-Stage Computer Vision Pipeline for Fine-Grained Vehicle Classification using Vision Transformers
Gandhimathi Padmanaban and Fred Feng
Vehicle body type is a significant determinant of cyclist injury severity in overtaking crashes, yet automated tools for classifying vehicles into injury-risk-relevant categories from naturalistic roadway
video do not exist in the open literature. Standard object detection
benchmarks provide only coarse vehicle labels (car, truck, bus, motorcycle), while existing fine-grained recognition systems are trained on controlled imagery and lack evaluation for deployment robustness across recording sites. This paper presents an open-source two-stage computer
vision pipeline combining a pre-trained RT-DETR detector for coarse vehicle localization with a fine-tuned
Vision Transformer (ViT-Base/16) for six-category body-type classification: passenger car, SUV, pickup truck, minivan, large van, and commercial truck. A confidence-based abstention mechanism withholds Stage 2 predictions when softmax output falls below 0.60, producing unknown labels rather than silent misclassifications. Evaluated on 3,805 annotated overtaking events from a bicycle-lane corridor in Ann Arbor, Michigan (in-distribution), the pipeline achieved 0.94 accuracy with per-class F1 scores from 0.91 (minivan) to 0.97 (SUV). On an independent out-of-distribution evaluation of 311 events from an open cycling
dataset without retraining, accuracy was 0.89. Three of four well-represented categories maintained F1 at or above 0.90 under domain shift. The largest degradation was observed for minivan (F1 = 0.72), driven by abstention rate rising from 2.4% to 25.0% rather than active misclassification, consistent with the mechanism propagating genuine model uncertainty. The full pipeline, including inference scripts, training code, evaluation utilities, and model weights, is released as open-source software to support reproducibility and reuse across roadside
video archives and cycling safety research.
Read the full abstract ▾
Computer Vision
GeM-NR: Geometry-Aware Multi-View Editing for Nonrigid Scene Changes
Josef Bengtson, Yaroslava Lochman and Fredrik Kahl
Recent developments in multi-view image editing with generative models have brought us a step closer toward general 3D content generation and customization. Most existing works focus on rigid or appearance-only edits by utilizing the geometry of the unedited scene. This naturally limits these methods to edits that preserve the underlying scene structure. Current nonrigid approaches are limited to object removal and insertion, reflecting the data they are trained on. General nonrigid edits, i.e., edits that substantially and arbitrarily change the scene geometry, remain challenging for existing methods. We propose GeM-NR, a fast and flexible training-free approach for general multi-view consistent image editing, including edits that drastically change the geometry and appearance of the scene. Given an anchor image edited with a chosen 2D editor and a query unedited image, GeM-NR edits the query image consistently with the anchor edit. The method incorporates multiple stages: (i) depth map estimation, where we propose a strategy to maximize the
alignment between the 3D point clouds of the edited and unedited scenes, (ii) projection onto a query viewpoint, and (iii) refinement of the obtained image conditioned on the unedited query. We demonstrate the ability of our method to handle edits with significant changes in geometry and appearance, something that existing methods struggle with. We perform an extensive evaluation showing that GeM-NR improves consistency for a wide variety of edit tasks, including generating 3D representations of the edited scene. Both quantitative and qualitative results indicate the state-of-the-art performance of our method in terms of edit quality as well as
geometric and photometric consistency across multiple views.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.GR
Geometry Gaussians: Decoupling Appearance and Geometry in Gaussian Splatting
Hongyu Zhou and Zorah Lähner
After the success of 3D Gaussian Splatting (3DGS) for novel view synthesis, many works have explored how to also use it for
geometric surface representation. However, extracting accurate
geometric information directly from 3DGS remains challenging and can often reduce the appearance rendering quality. In this work, we show that 3DGS in its default form is inheritedly unsuited to represent texture and geometry at the same time, by training with complete ground-truth texture and geometry information. We also propose a simple solution by applying a single additional geometry opacity parameter to each splat, together with an optional transparency-curated optimization pipeline. Our experiments, both with ground-truth and
vision foundation model
geometric input, show that this change leads to improved rendering and geometry performance on a wide variety of
dataset, and especially complex scenes with transparent objects benefit significantly from our method.
Read the full abstract ▾
Computer Vision
Continual Visual and Verbal Learning Through a Child's Egocentric Input
Xiaoyang Jiang, Yanlai Yang, Kenneth A. Norman et al. (+2 more)
Children learn the meanings of words from a continuous, temporally structured stream of
egocentric experience. Recent work shows that neural networks can also learn word-referent mappings from a child's
egocentric video recordings, but they cycle through the shuffled data for hundreds of epochs, contrasting with how children actually encounter their environment. We introduce BabyCL, a continual
multimodal learning framework that processes the SAYCam
dataset in a single chronological pass, combining streaming visual representation learning with an image-text contrastive objective. BabyCL combines a multi-stage temporal
segmentation of the stream with a dual replay buffer that independently manages visual and
multimodal histories, and it is jointly trained with three contrastive losses on a shared backbone. Under a matched optimization budget, BabyCL outperforms streaming learning baselines on the SAYCam Labeled-S 4AFC
benchmark, substantially narrowing the gap to an upper bound of offline training. Ablations show that the gains are robust to the length of the online temporal
segmentation window and the eviction rule of the replay buffer. Together, these results show that meaningful word-referent mappings can emerge under training conditions much closer to a child's actual experience.
Read the full abstract ▾
All authors Xiaoyang Jiang, Yanlai Yang, Kenneth A. Norman, Brenden Lake, Mengye Ren
Computer Vision
Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have
Elouan Gardès, Seung Eun Yi, Kartik Ahuja et al. (+6 more)
We propose a label-free approach to adapt powerful but generic
vision foundation models to specialized scientific domains. Standard supervised fine-tuning is often ill-suited to these settings: labels are scarce, and task-specific training can collapse the model's generality and hurt robustness. We instead leverage metadata to adapt representations to new domains in a self-supervised manner. Our method, FINO, combines a standard self-supervised objective with flexible metadata guidance that handles both highly granular discrete metadata and continuous metadata. It encourages the representation to preserve informative factors while suppressing spurious ones. Across subcellular fluorescence microscopy, Earth observation, wildlife monitoring, and medical imaging, FINO consistently outperforms standard unsupervised domain adaptation and fully supervised adaptation. It also exceeds highly-specialized domain-specific state of the art, while using no task labels for backbone adaptation and only lightweight probes for supervision.
Read the full abstract ▾
All authors Elouan Gardès, Seung Eun Yi, Kartik Ahuja, Théo Moutakanni, Huy V. Vo, Piotr Bojanowski, Wolfgang M. Pernice, Loïc Landrieu, Camille Couprie
Computer Vision · Cross-listed from cs.LG
Identifying Gems from Roman RAPIDly
Karan Gandhi, Ashish A. Mahabal, Jacob E. Jencson et al. (+6 more)
The Nancy Grace Roman Space
Telescope (Roman), set for launch as early as September 2026, will conduct wide-field infrared imaging surveys with unprecedented spatial resolution and cadence, enabling the discovery of millions of astronomical transients. Hence, it is necessary to have automated pipelines for generating alerts in place so that the
telescope can begin discovering reliable transients and variable objects soon after it is launched. However, no real Roman data currently exist, making the development of such pipelines difficult. In this work, we present a machine learning model $RuBR$ and a general methodology for distinguishing genuine transient and variable detections from spurious (bogus) detections within the RAPID pipeline. In particular, we present three models using this methodology: $RuBR_{comb}$ trained and tested on combined locally injected and OpenUniverse2024 transients, $RuBR_{loc}$ trained on locally injected transients and tested on OpenUniverse2024 transients, and $RuBR_{DA}$ that combines locally injected transients with a fraction of OpenUniverse2024 transients in domain-adaptation mode for training. This paves the way for strategies to adapt the $RuBR_{comb}$ model to real observations in the absence of any ground-truth labels during the early phases of the Roman mission. While the image differencing pipeline continues to be improved, our experimental results demonstrate the effectiveness of the proposed approach and its promise for robust real-bogus classification in the Roman era.
Read the full abstract ▾
All authors Karan Gandhi, Ashish A. Mahabal, Jacob E. Jencson, Russ R. Laher, Ben Rusholme, Lin Yan, Ryan M. Lau, Schuyler D. Van Dyk, Mansi M. Kasliwal
Computer Vision
ZipSplat: Fewer Gaussians, Better Splats
Alexander Veicht, Sunghwan Hong, Dániel Baráth et al. (+1 more)
Feed-forward 3D Gaussian Splatting reconstructs scenes from images in seconds. Pixel-aligned methods keep every Gaussian grounded in an observation, but produce redundant, potentially misaligned Gaussians across views.
Token-based methods instead aggregate all views into a fixed set of learned
tokens but lack explicit observation support and use a fixed reconstruction budget. We introduce ZipSplat, a method that predicts a compact, observation-grounded scene representation whose size is set at inference. It derives each scene
token from an image patch rather than a learned query, and decodes it into freely placed 3D Gaussians.
Tokens across views are merged into a compact set whose size is adjustable at inference. However, without
geometric supervision, decoded Gaussians can drift from their source patch, and merging can mix
tokens from different surfaces. We therefore introduce two
geometric losses to ground and shape this representation. A cone loss ties each
token's Gaussians to its source patch, and GeoCE aligns
token similarity with 3D proximity so that
tokens observing the same surface merge together. Building on these observation–geometry associations, we further introduce TokenBA, a
token-level bundle adjustment that refines scene
tokens and camera poses in a few seconds. ZipSplat requires neither camera poses nor intrinsics, yet outperforms pixel-aligned and
token-based baselines on DL3DV and RealEstate10K, even those given ground-truth cameras. It further generalizes zero-shot to Mip-NeRF360 and ScanNet++. Our project page is at https://veichta.com/zipsplat.
Read the full abstract ▾
All authors Alexander Veicht, Sunghwan Hong, Dániel Baráth, Marc Pollefeys
Computer Vision
InstantRetouch: Efficient and High-Fidelity Instruction-Guided Image Retouching with Bilateral Space
Jiarui Wu, Yujin Wang, Ruikang Li et al. (+3 more)
Language-guided photo retouching aims to adjust color and tone while preserving geometry and texture. Recently,
diffusion-based retouching shows a superior visual quality, but often struggles with both fidelity issues due to its generative nature and efficiency because of its iterative sampling process. In this work, we propose an efficient and fidelity-preserving retouching method using bilateral space manipulation, which is both compact and content-decoupled. Specifically, instead of directly editing pixels or image latents, our model predicts a low-resolution bilateral grid of affine transforms, which are sliced using a learned guidance map and then applied to the full-resolution image. This approach yields both high fidelity and improved efficiency. To retain strong priors of a pretrained generative model, we distill a multi-step
diffusion model into our bilateral grid framework using Variational Score
Distillation, complemented by a
prompt alignment loss to guide instruction-following behavior. Additionally, we introduce a new
benchmark and evaluate our method across multiple dimensions: fidelity, instruction following, and efficiency. Compared to the latest retouch methods, like Gemini-2.5-Flash (Nano-Banana), our method can avoid content drift, significantly improve latency, and generate visually pleasing edits, while maintaining a high level of fidelity. Project page: https://openimaginglab.github.io/InstantRetouch/.
Read the full abstract ▾
All authors Jiarui Wu, Yujin Wang, Ruikang Li, Fan Zhang, Mingde Yao, Tianfan Xue
Computer Vision
MaCo-GAN: Manifold-Contrastive Adversarial Learning for Single Image Super-Resolution
Daeyoung Han, Seongmin Hwang and Moongu Jeon
Conventional Generative
Adversarial Networks (GANs) for Single Image Super-Resolution (SISR) often struggle with hallucinated artifacts, largely because standard discriminators evaluate overall image naturalness rather than strict conditional realism. To address this, we propose MaCo-GAN, a novel
manifold-contrastive GAN framework that replaces the conventional
adversarial loss with a supervised contrastive objective. A core component of our method is a dynamic fake sample synthesizer that transforms ground truth (GT) data into a spectrum of challenging, perceptually plausible fake images that strictly maintain low-resolution (LR) correspondence. Utilizing these synthesized samples, we establish a robust contrastive minimax game: the generator is trained to attract its predictions toward on-
manifold fakes (low distortion) and repel them from off-
manifold fakes (high distortion), while the discriminator optimizes the exact opposite. By simply replacing the
adversarial loss of a baseline SR model with our proposed objective, we demonstrate consistent improvements in the perception-distortion trade-off across various
benchmarks. Extensive ablation studies validate the effectiveness of our framework and provide deep insights into the dynamics of this conditional contrastive game.
Read the full abstract ▾
Computer Vision
UniCAD: A Unified Benchmark and Universal Model for Multi-Modal Multi-Task CAD
Jingyuan Chen, Sheng Jin, Haopeng Sun et al. (+2 more)
Computer-Aided Design (CAD) underpins modern engineering and manufacturing by enabling the creation of precise, editable 3D models. However, CAD research typically studies tasks in isolation, and multi-modal, multi-task learning for CAD is hindered by the absence of a unified
benchmark. To address this gap, we introduce UniCAD, a comprehensive
benchmark for multi-modal CAD learning that covers point-to-CAD reconstruction, text/image-to-CAD generation, and CAD question answering across diverse input modalities. Alongside the
benchmark, we present UniCAD-MLLM, a universal multi-modal large
language model that ingests text, images, sketches, and point clouds and performs these heterogeneous tasks in an end-to-end fashion within a single framework. Extensive experiments on the UniCAD and Fusion360
benchmarks demonstrate that UniCAD-MLLM achieves state-of-the-art performance across all tasks, outperforming existing task-specific and multi-task baselines. We will release the
dataset, code, and pretrained models to accelerate future research.
Read the full abstract ▾
All authors Jingyuan Chen, Sheng Jin, Haopeng Sun, Wentao Liu, Chen Qian
Computer Vision
Anchor3R: Streaming 3D Reconstruction with Transient Anchors for Long-Horizon Visual Mapping
Peilin Tao, Chong Cheng, Yuansen Du et al. (+8 more)
Long-horizon online visual mapping requires continuous camera-motion and scene-geometry estimation under bounded computation. Recent feed-forward 3D reconstruction models provide strong
geometric priors, but streaming variants often predict poses in a fixed or historically maintained coordinate system, leading to train–test mismatch, early-anchor
attention bias, and accumulated drift. We propose Anchor3R, a current-centric streaming 3D reconstruction framework that predicts window-relative poses and local geometry in the current-frame coordinate system. Overlapping predictions form a dense relative-pose graph, supporting online pose updates and loop-aware motion averaging for global reconstruction. Experiments on indoor, outdoor, driving, and RGB-D
benchmarks demonstrate improved long-horizon pose accuracy and dense reconstruction quality over existing streaming baselines. Despite being trained only on 48-frame sequences, Anchor3R directly generalizes to streams exceeding 10,000 frames while maintaining bounded GPU
memory during online inference. Code is available at https://github.com/polar-explorer/Anchor3R.
Read the full abstract ▾
All authors Peilin Tao, Chong Cheng, Yuansen Du, Caiwei Song, Zhengqing Chen, Xiaoyang Guo, Wei Yin, Weiqiang Ren, Qian Zhang, Hainan Cui, Shuhan Shen
Computer Vision
MetaPoint: Unlocking Precise Spatial Control in Agentic Visual Generation
Dewei Zhou, Xinyu Huang, Xun Wang et al. (+6 more)
Generative visual models fundamentally struggle with precise spatial control. This arises from a core disconnect: models can process textual descriptions of space but cannot directly map numerical coordinates onto the 2D image canvas. We introduce MetaPoint, a method that bridges this gap by representing a continuous 2D coordinate as a single, special
token. Crucially, MetaPoint requires no new architectural components; it directly leverages the model's inherent positional encoding schemes to interpret these coordinates, treating our
token as a virtual point on the canvas. This lightweight approach enables pixel-level control of an object's position with one
token or its bounding box with two, all without requiring architectural changes or bespoke
attention masking. The MetaPoint
tokens are designed to be compositional, serving as spatial primitives. This allows a planner
agent to decompose a high-level user request into a structured sequence of primitives for the generator. By providing a simple, precise, and scalable building block for spatial control, MetaPoint unlocks more powerful compositional generative
agents and enables intuitive, interactive editing systems.
Read the full abstract ▾
All authors Dewei Zhou, Xinyu Huang, Xun Wang, Ji Xie, Yabo Zhang, Liang Li, Kunchang Li, Zongxin Yang, Yi Yang
Computer Vision
Handwriting Extraction and Analysis of Signature Lists in Swiss Popular Initiatives
Marco Peer, Thomas Gorges, Mathias Seuret et al. (+2 more)
Popular initiatives and referendums are central to Swiss democracy, yet the validation of handwritten signature lists remains a labor-intensive manual process. This paper investigates the potential of automated document analysis methods, including OCR and AI-based handwriting analysis, to support this task. We propose a pipeline combining template-based line
segmentation with text recognition and writer
retrieval techniques, evaluated on a
dataset of 443 handwritten entries from 418 writers. Results show that OCR struggles with out-of-vocabulary handwriting, with a CER of 29.6% for first names. In contrast, writer
retrieval performs more robustly, reaching an mAP of 50.6%. Furthermore, our experiments indicate that off-the-shelf OCR systems are not sufficiently reliable for transcription of handwritten signature data, particularly for short, out-of-vocabulary entries such as names or addresses. However, writer
retrieval methods can effectively identify visually similar entries across signature lists, making them a suitable tool for supporting the detection of potential duplicate submissions based on handwriting similarity.
Read the full abstract ▾
All authors Marco Peer, Thomas Gorges, Mathias Seuret, Vincent Christlein, Andreas Fischer
Computer Vision
CIPER: A Unified Framework for Cross-view Image-retrieval and Pose-estimation
Yurim Jeon, Dongseong Seo and Seung-Woo Seo
Cross-view geo-localization estimates the geographic location of a ground image by matching it against an aerial image database. Existing methods tackle this through either large-scale
retrieval or precise pose estimation, but not both:
retrieval-based methods enable wide-area search at the cost of localization accuracy, while pose estimation methods achieve high precision within only a narrow search space. Naively cascading these pipelines introduces error propagation and inconsistent feature representations. We formulate cross-view geo-localization as a unified problem requiring simultaneous city-scale
retrieval and precise 3-DoF pose estimation. We propose CIPER (Cross-view Image-
retrieval and Pose-estimation
transformER), a single architecture that jointly performs both tasks through mutually beneficial feature learning. CIPER uses a shared
transformer encoder with task-specific
tokens to disentangle global
retrieval features from spatial localization cues. To bridge the large domain gap between ground and aerial views, we introduce a two-way
transformer pose decoder that uses ground features as spatial queries for bidirectional cross-
attention. A set prediction strategy further enables stable 3-DoF regression under a unified multi-task objective. Experiments on VIGOR, KITTI, and Ford Multi-AV demonstrate competitive performance, especially under limited field-of-view and arbitrary orientation conditions. Code is available at https://github.com/yurimjeon1892/CIPER.
Read the full abstract ▾
Computer Vision
M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks
Jie Huang, Ruixun Liu, Sirui Sun et al. (+4 more)
As multi-modal models advance towards long-form
video understanding,
memory emerges as a critical capability. Despite substantial efforts in developing
video datasets and
benchmarks, existing works primarily focus on perception and
reasoning, without systematically evaluating
memory: what models retain, how faithfully information is preserved, and how robust
memory remains under interference. To address this gap, we introduce M$^3$Eval, the first comprehensive evaluation framework and
benchmark for probing different
memory dimensions in multi-modal models. Grounded in cognitive psychology, our design features carefully constructed tasks that isolate key aspects of
memory. Leveraging M$^3$Eval, we conduct extensive experiments across representative multi-modal models, revealing consistent weaknesses and distinctive behaviors. We find that models struggle to maintain disentangled representations when processing parallel
video streams, exhibit interference patterns differing substantially from those observed in human
memory, ground
memory sources more reliably in the spatial domain than the temporal domain, and demonstrate limited symbolic
memory. Collectively, our
benchmark provides a valuable resource for future research, while our findings highlight
memory as a fundamental yet underexplored capability and offer insights for designing more effective
memory mechanisms in multi-modal models. Our code and
dataset are available at https://pku-value-lab.github.io/m3eval-homepage.
Read the full abstract ▾
All authors Jie Huang, Ruixun Liu, Sirui Sun, Xinyi Yang, Yin Li, Yixin Zhu, Yiwu Zhong
Computer Vision
Multi-Camera AR Guidance System for Surgical Instrument Handling and Assembly: Investigating Workload and Efficiency
Shiyu Li, Julian Kreimeier, Hannah Schieber et al. (+4 more)
The handling and assembly of instruments during surgery imposes high cognitive demands on scrub nurses, particularly when instruments are unfamiliar. We present a supporting guidance system for
surgical instrumentation that combines multi-camera 6D pose estimation with augmented reality in-situ visualization on a head-mounted display without the requirement for additional markers. Pose estimation and consecutive camera calibration are achieved through known objects. The 6D pose estimation network is trained purely on synthetic data, aiming for better generalizability and real-world applicability. The AR guidance displays tooltip localization cues and step-wise assembly animations. Via gaze-based selection and a foot pedal, users can switch between assembly steps in intraoperative use. In a technical evaluation, our approach outperforms state-of-art 6D pose estimation. A user study with 29 scrub nurses was conducted in a
surgical simulation of knee arthroplasty, comparing the system against a paper manual. AR guidance significantly reduced the perceived workload compared. Objectively, AR guidance reduced task completion time by 21.3% (4.76 minutes). Specifically, scrub nurses less experienced with the instrument set benefited when using the system. Error frequencies were comparable between conditions. Qualitative feedback highlighted improved process clarity, reduced information overload, and perceived independence. To summarize, our marker-free multi-camera AR guidance approach for
surgical instruments can, subjectively and objectively, improve intraoperative instrumentation performance, particularly for untrained scrub nurses.
Read the full abstract ▾
All authors Shiyu Li, Julian Kreimeier, Hannah Schieber, Dirk Müller, Bernhard Kainz, Rüdiger von Eisenhart-Rothe, Daniel Roth
Computer Vision
Food-R1: A Unified Multi-Task Food Vision-Language Model with Reinforcement Learning
Yu Zhu, Yongkang Li, Wenjie Zhu et al. (+5 more)
Recent studies have explored
Vision-
Language Models (VLMs) for food analysis. However, most existing methods rely primarily on supervised fine-tuning (SFT), which often limits
reasoning and generalization capabilities. Moreover, high-quality large-scale nutritional annotations remain scarce. To address these issues, we introduce CalorieBench-80K, a large-scale
benchmark with curated calorie labels and dietary advice annotations. To the best of our knowledge, it is the first food image
benchmark to incorporate Chain-of-Thought (CoT) annotations for calorie
reasoning. We also propose Food-R1, a unified food VLM trained in a multi-task learning paradigm to equip the model with broad capabilities. Food-R1 undergoes CoT-based cold-start instruction tuning, followed by reinforcement fine-tuning (RFT) using Group Relative Policy Optimization (GRPO) to improve
reasoning and performance. Experiments on CalorieBench-80K and representative
benchmarks show that Food-R1 consistently outperforms strong baselines across food-related tasks. The code, model weights, and
benchmark annotations are available at the project repository.
Read the full abstract ▾
All authors Yu Zhu, Yongkang Li, Wenjie Zhu, Haoyi Jiang, Wenyu Liu, Wei Yang, Bin Li, Xinggang Wang
Computer Vision
Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance
Kaustav Kundu, Ritvik Shrivastava, Maxim Arap et al. (+14 more)
Proactive
multimodal assistants must decide when guidance is needed and what to say, while tracking progress through extended procedures and helping users recover when they depart from the expected plan. Progress is limited by
benchmarks that score isolated clips, cover narrow domains, or omit realistic deviations. We introduce EgoProactive, a smart-glasses
dataset of procedures across four domains in which users commit step-level deviations and recover from them, paired with reference guidance. We combine it with five re-annotated
egocentric and instructional
datasets to form Pro$^{2}$Bench, a cross-domain
benchmark that scores intervention timing, procedure completion, and response quality. Zero-shot, no frontier or open-weight model completes more than 26% of a procedure's interventions before its first error. We propose PWR (Plan, Watch, Recover), which keeps an explicit plan state in the model's context and revises it after every intervention, including recovery steps after a deviation. Across three open-weight backbones, training on Pro$^{2}$Bench without a plan barely changes performance, while adding the dynamic plan raises average procedure completion from 20% to 27% and improves turn-level decisions and response quality for every backbone. The gains hold on held-out
datasets and on deviations never seen in training.
Read the full abstract ▾
All authors Kaustav Kundu, Ritvik Shrivastava, Maxim Arap, Nanshu Wang, Xianhui Zhu, Quintin Fettes, Gautam Tiwari, Parth Suresh, Prateek Singhi, Théo Moutakanni, Alejandro Castillejo Munoz, Allen Bolourchi, Pascale Fung, Pinar Donmez, Babak Damavandi, Anuj Kumar, Seungwhan Moon
Computer Vision
Scene-Centric Unsupervised Video Panoptic Segmentation
Christoph Reich, Oliver Hahn, Nikita Araslanov et al. (+4 more)
Video panoptic
segmentation (VPS) aims to jointly detect, segment, and track all objects while partitioning the
video into semantically consistent regions. We introduce the task setting of unsupervised VPS, omitting any human supervision. Existing unsupervised scene understanding works mainly focused on image
segmentation tasks; the
video domain remains underexplored. We propose VideoCUPS, the first unsupervised VPS approach. VideoCUPS generates temporally consistent panoptic
video pseudo-labels from scene-centric
videos by exploiting unsupervised depth, motion, and visual cues. Training on these pseudo-labels using a novel
Video DropLoss yields an accurate, unsupervised VPS model. To
benchmark progress, we introduce a comprehensive evaluation protocol and four competitive baselines, extending state-of-the-art unsupervised panoptic image and instance
video segmentation models to VPS. VideoCUPS outperforms all baselines and demonstrates strong label-efficient learning. With VideoCUPS, our evaluation protocol, and baselines, we provide a strong foundation for future research on unsupervised VPS.
Read the full abstract ▾
All authors Christoph Reich, Oliver Hahn, Nikita Araslanov, Laura Leal-Taixé, Christian Rupprecht, Daniel Cremers, Stefan Roth
Computer Vision
Tran Dinh Tien and Zhiqiang Shen
Current
prompt-based and adapter-based tuning of
vision-
language models (VLMs) is attractive for medical imaging, where clinical data sensitivity favors frozen backbones and annotations are limited. However, these methods typically optimize only the ground-truth class, treating all other classes as equally incorrect, ignoring clinically meaningful class relations and yielding unstable decision boundaries in limited-supervision settings. We propose Omni-Geometry Knowledge
Distillation (OGKD), a new framework that injects class-relation structure into the teacher to produce directional targets that preserve the ground truth while respecting inter-class geometry. Using these targets, we develop two
distillation losses: Global Geometry-Aware
Distillation (GAD) operates on the global image
token, and Label-Guided Geometry
Distillation (LGD) applies the same geometry to attentive patch
tokens to improve fine-grained
alignment. Across comprehensive experiments and analyses on 11 widely-used medical
datasets for base-to-novel and few-shot evaluations, our OGKD achieves substantially better performance, consistently improving accuracy by an average absolute gain of 1.7%-2.8% over all prior state-of-the-art VLM adaptation counterparts. It also robustly generalizes to unseen classes and yields more reliable predictions than other approaches. Our code is available at https://github.com/tientrandinh/OGKD.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Toward Multi-Domain and Long-Tailed Quantization via Feature Alignment and Scaling
Ting-An Chen, Chin-Yuan Yeh and De-Nian Yang
Quantizing deep neural networks is essential for efficient inference on resource-constrained devices. However, most existing methods are designed for single-domain and class-balanced data, leaving practical settings with domain shifts or severe class imbalance underexplored. We address these challenges with Efficient Multi-Domain
Alignment Quantization (EmaQ), which aligns domain distributions through a CDF-based projection and uses sensitivity-aware weight aggregation to stabilize multi-domain quantization. We further extend EmaQ to EmaQ-LT for long-tailed quantization by introducing class-conditioned variance scaling and confidence-based logit adjustment to mitigate majority-class overconfidence. Theoretical analyses establish convergence guarantees and motivate the proposed sensitivity and scaling mechanisms. Experiments on standard, multi-domain (Office-31, Digits), and long-tailed (SynDigits-LT, CIFAR-10-LT, CIFAR-100-LT)
benchmarks show that EmaQ and EmaQ-LT achieve strong low-bit performance under domain shift and class imbalance.
Read the full abstract ▾
Computer Vision
BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine
Yang Liu, Jiajin Zhang, Danyang Tu et al. (+8 more)
Breast cancer remains a leading cause of cancer-related mortality among women. Its clinical management requires
multimodal reasoning across a clinical workflow that spans screening, diagnosis and treatment planning, where each stage involves distinct imaging modalities, task objectives, and
reasoning patterns. However, constrained by data scarcity and model versatility, existing medical MLLMs are typically evaluated on isolated modalities or narrow task families, limiting their ability to support workflow-level clinical
reasoning. In this work, we first introduce BreastStage, a workflow-aligned breast imaging instruction corpus comprising 1.86M instruction-following pairs curated from 17 sub-
datasets across 5 imaging modalities and 136 task templates. Its held-out split, BreastStage-Bench, provides a comprehensive
benchmark for evaluating
multimodal reasoning across the breast cancer care continuum. Building on this corpus, we propose BreastGPT, a unified MLLM equipped with a dual-branch visual encoder and concept-preserving
token compression to bridge the scale gap between standard radiology and gigapixel pathology. On BreastStage-Bench, BreastGPT achieves 75.66% closed-ended accuracy and 89.92% open-ended score, outperforming both general-purpose and medical-specific MLLMs across clinical stages and task formats. These results suggest that workflow-aligned data and cross-scale visual modeling are critical for clinically grounded medical MLLMs. All data, code, and model checkpoints are released at https://yangyy-liu.github.io/BreastGPT.io.
Read the full abstract ▾
All authors Yang Liu, Jiajin Zhang, Danyang Tu, Yaojun Hu, Jiao Qu, Jiuyu Zhang, Yu Shi, Wei Fang, Shi Gu, Ling Zhang, Yingda Xia
Computer Vision
CDPM-Align: Multi-Scale Guidance-Aligned Diffusion Pretraining for Robust Few-Shot Anatomical Landmark Detection
Roberto Di Via, Irina Voiculescu, Francesca Odone et al. (+1 more)
Anatomical landmark detection is a fundamental task in medical image analysis supporting a wide range of diagnostic and interventional workflows. Although recent methods have achieved sub-millimetric localisation, accuracy alone is not sufficient for clinical deployment, requiring reliability and robustness in prediction. Despite its clinical relevance, the impact of representation learning in this context is still underexplored. In this work, we introduce CDPM-align, a multi-scale guidance-aligned conditional
diffusion pre-training for anatomical landmark detection. Our experimental setup focuses on a few images and a few annotation regimes. Specifically, we employ three popular heterogeneous small-scale
benchmark datasets for representation learning via conditional generative pre-training. Furthermore, we consider low-annotation scenarios for the downstream task of landmark detection, with 10 and 25 annotated images, reflecting realistic trade-offs between clinical effort and resource constraints for annotations. Our results confirm that generative pre-training enables the model to learn a robust representation. This improves both accuracy and uncertainty on the downstream tasks, advancing towards safe and efficient clinical deployment.
Read the full abstract ▾
All authors Roberto Di Via, Irina Voiculescu, Francesca Odone, Vito Paolo Pastore
Computer Vision
Hierarchical Space Partition for Surface Reconstruction
Minjie Tang and Xiangfei Li
Generating compact polygonal models from point clouds is a key problem in 3D
vision and computer graphics. However, due to inherent limitations of
LiDAR scanning (e.g. range constraints and occlusions), critical scene information is often missing, leading to degraded reconstruction accuracy. To address this, we propose a plane assembling strategy that effectively recovers missing details while maintaining model compactness. We classify all the planes extracted from the scene into three categories: highly visible, barely visible, and invisible. The invisible planes, which are recovered by scene structure analysis, indicate the missing details. The three types of planes correspond to the three growth priorities. Each plane grows according to the priority level, and the space is partitioned progressively, namely, the hierarchical partition. Subsequently, we generate a watertight polygonal mesh from the partition via a min-cut-based optimization. Finally, comparisons on public
datasets show the effectiveness and superiority of our method against mainstream approaches. The project page is available at https://hsr-3dv.github.io/.
Read the full abstract ▾
Computer Vision
HD-DinoMoE: A Class-Aware Hierarchical Dual Mixture-of-Experts Network for Scleral Anomaly Segmentation in Complex Acquisition Scenarios
Yinxiang Yu, Maoxiang Chu, Qi Niu et al. (+7 more)
Traditional Chinese Medicine (TCM) ocular inspection provides empirical cues for assessing scleral surface anomalies, but its clinical use remains subjective and difficult to quantify. To support intelligent and quantifiable ocular inspection, this study presents the TCM-inspired Artificial Intelligence Ocular Auxiliary Diagnosis System (TAO) and focuses on pixel-level scleral surface anomaly
segmentation. For clinical and user-acquired images affected by multi-source distributional discrepancies, diverse anomaly morphologies, and scleral specular reflection (SSR), we propose HD-DinoMoE, a class-aware hierarchical dual mixture-of-experts network. HD-DinoMoE combines class-aware dual-stream DINOv3 feature fusion with class-specific multi-expert decoding to segment Vessels, Yellow and Black Spots, and Blood Spots. A three-stage backbone-frozen routing strategy stabilizes dual-backbone adaptation; Progressive Confidence Penalty (PCP) Loss reduces high-confidence false positives and
segmentation leakage in SSR regions; and Class-Aware Adaptive Sample Weighting (CA-ASW) balances sample- and class-level training contributions. We further construct the Multi-label Scleral Anomaly
Segmentation Dataset (ML-SASD), a new
benchmark with Clinical, Wild, and Mix settings and pixel-wise annotations for three anomaly categories. On ML-SASD-Mix, HD-DinoMoE achieves a mean Dice of 72.11% and a mean Intersection-over-Union of 58.44%, while maintaining favorable boundary localization and specular-region false-positive control. It also shows competitive generalization on the Vessels subset of the public SBVPI
dataset. These results indicate that HD-DinoMoE provides a feasible
segmentation solution for TAO under complex acquisition scenarios. The code and data access information are available at https://github.com/FX-CMX/HD-DinoMoE.
Read the full abstract ▾
All authors Yinxiang Yu, Maoxiang Chu, Qi Niu, Guanghu Liu, Wei Xu, Haotian Wang, Zhi Chen, Yutian Zhu, Yuelong Fan, Guanghao Liao