Computer Vision
DSSR-3D: Decoupled Reasoning for View-Dependent Referring in 3D Gaussians
Thanh-Khoi Nguyen, Thien-Phuc Tran and Minh-Triet Tran
Recent advances in 3D Gaussian Splatting have enabled open-vocabulary and referring
segmentation by distilling semantic knowledge from 2D foundation models into 3D representations. However, existing referring fields embed
language features in a globally view-invariant space, making them fundamentally unable to resolve observer-centric spatial relations (e.g., "to the left of") that depend on camera pose. We propose DSSR-3D, an inference-time framework for view-dependent referring
segmentation on continuous 3D Gaussian fields, formalized as two interfaces - pose-invariant semantic localization and pose-conditioned spatial
reasoning - such that any pair of functions satisfying these constraints yields a valid instantiation, requiring no retraining of the underlying semantic field and no reliance on discrete
geometric proxies such as bounding boxes. We instantiate the two interfaces with a temperature-sharpened softmax localization mechanism and a projection-based directional scoring function, fused via a lightweight, training-free step, and show they transfer zero-shot to structurally distinct semantic fields without adaptation. We further propose ViewRef-GS, a
benchmark isolating view-dependent
segmentation on 3D Gaussian fields, evaluated jointly with an augmented Ref-LERF to provide a comprehensive testbed for viewpoint-dependent spatial grounding. Experiments show consistent gains over existing 3DGS-based referring methods, with no additional training beyond the base semantic field
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models
Nicholas Sansoterra, Zishuo Zheng and Sachin Kumar
Instruction hierarchy (IH)
alignment teaches
language models to prioritize higher-level instructions when inputs conflict. While studied primarily in text-only settings,
vision-
language models (VLMs) introduce new challenges for IH: instructions may be embedded in images, split across modalities, visually transformed, or encountered during agentic tasks. Positing
multimodal IH
alignment as a
reasoning problem, we train VLMs using reinforcement learning with rule-based rewards, comparing text-only, image-only, and mixed-modality supervision. We find that text-only IH training partially transfers to
multimodal attacks, failing when models must decode, reconstruct, or reason over instructions across modalities. Image-based training improves robustness beyond text-only supervision, while mixed-modality training performs best overall. Importantly, the benefits generalize beyond the synthetic typographic training setting to real-image and web-
agent safety tasks, while largely preserving general
multimodal capability, showing that lightweight, verifiable supervision can meaningfully improve VLM robustness under
adversarial, cross-modal, and interactive instruction conflicts.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.GR
Cascaded Non-Line-of-Sight Imaging
Diego Royo, María Peña, Forrest B. Peterson et al. (+3 more)
Time-of-flight non-line-of-sight (NLOS) imaging recovers information from hidden objects by analyzing the time of flight of indirect
photons scattered on a visible (relay) wall. Most methods make the simplifying assumption that
photons travel exclusively three-bounce paths, thus ignoring other useful information encoded in higher-order
photons (with, e.g., four- or five-bounce paths). We present a novel cascaded NLOS imaging approach that leverages higher-order information and allows imaging a broader range of single- and multi-corner scenarios. We combine ultra-fast
laser scanning with recent time-gated 2D sensor arrays to capture the scene's impulse response on a visible relay wall. From the captured impulse response, our method computes an analogous virtual impulse response at any other hidden wall. This effectively allows us to concatenate a second, virtual NLOS imaging system that leverages higher-order illumination. We validate our cascaded imaging method both in simulation and with a real prototype, demonstrating NLOS imaging with fourth- and fifth-bounce illumination of objects in challenging orientations and hidden around two corners. We also analyze how wave-based NLOS imaging interacts with rough hidden walls, which explains and helps overcome existing visibility limitations. We further illustrate how to image hidden objects from different perspectives, thus observing previously unseen features, by relying on multiple hidden walls.
Read the full abstract ▾
All authors Diego Royo, María Peña, Forrest B. Peterson, Andreas Velten, Julio Marco, Diego Gutierrez
Computer Vision
ArtSociety: Multi-Agent Multimodal Collaboration for Art Emotion Understanding
Jian Li, Fanfan Ji, Jinxiang Lai et al. (+5 more)
The AffectiveArt Multidimensional Art
Emotion Understanding task asks to jointly predict an artwork's fine-grained
emotion (12 classes, 1549:1 head-to-tail ratio), binary valence/arousal, and five attribute-grounded descriptions – sub-tasks that exhibit strong empirical trade-offs, so the single-model solutions we tried do not jointly optimize all of them well. We present ArtSociety, a multi-
agent framework that assembles heterogeneous
multimodal experts – a DINOv2-
Giant vision agent (A1), a scene-grounded CoT fine-tuned MLLM (A2), and three closed-source
reasoning agents (A3-A5) – and coordinates them with two training-free controllers: (i) a rare-class-aware voting arbiter that lowers the agreement threshold for tail
emotions, exploiting decorrelated error patterns across
agent families; and (ii) a description-first
reasoning agent whose DESCRIBE-then-CLASSIFY chain of thought forces visual evidence before label commitment, yielding near-perfect grounded descriptions. A task-routing policy directs the hard
emotion task to the full five-
agent ensemble while assigning the near-saturated valence/arousal and generative description tasks to the single strongest
reasoning agent. On the official test set (1,000 artworks), ArtSociety achieves an Overall Score of 0.8870 (Classification 0.7789, Description 0.9952). An eleven-variant ablation study reveals that, once method and scale saturate at around 0.76, the decisive gains come from
agent collaboration and data-side supervision – a 30B MoE model trained on older data does not outperform an 8B model trained on better data. Code is available at https://github.com/swordlidev/ArtSociety
Read the full abstract ▾
All authors Jian Li, Fanfan Ji, Jinxiang Lai, Ying Tai, Jian Yang, Xiao-Tong Yuan, Chengjie Wang, Yabiao Wang
Computer Vision
Abstract-LoRA: Unlocking Single-Image Style Transfer through Targeted U-Net Block Training
Xinglin Hu
Diffusion models represent one of the most advanced paradigms in generative modeling. Leveraging their development, a growing number of style transfer methods based on
diffusion models have been proposed. However, among these methods, multi-image style transfer approaches that require at least five to ten style examples tend to achieve more satisfactory results. Single-image methods, by contrast, often struggle with either insufficient content preservation or inadequate style fidelity. This greatly limits style extraction from scarce artworks and undermines their artistic value. To address this, we propose Abstract-LoRA, a method that pushes the boundaries of single-image style transfer through lightweight LoRA training on specific U-Net blocks in
diffusion models. Specifically, our work is inspired by B-LoRA, a style transfer method that achieves basic style-content disentanglement by training specific U-Net blocks. However, it suffers from a critical limitation: the inability to capture complex backgrounds. Building upon B-LoRA, our method conducts a more refined analysis of U-Net blocks, employing additional U-Net blocks and clustering-based abstraction of style images to better disentangle and balance style and content. Extensive experiments demonstrate that our proposed method not only generates visually more harmonious and satisfying artistic images but also quantitatively improves the preservation of both style and content in the final outputs.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CE
LLM-Aided Design for Manufacturing: A Multi-Agent System for Intent-Preserving Redesign of CAD for Improved Manufacturability
Kojo Welbeck, Xiangyu Shi, Zahra Sadeghi et al. (+2 more)
We introduce autonomous, intent-preserving Design for Manufacturing (DFM) redesign of CAD parts: given an engineer's CAD model, the method returns a variant that is easier to manufacture without losing its design intent. Generating such a redesign in a single shot is unreliable, since CAD fidelity degrades as parts grow complex; we instead produce it as a sequence of individually verified design transitions. Our DFM-Redesign pipeline realizes this with two coupled
agent subsystems driven by a pretrained
multimodal LLM: a DFM Reviewer that inspects the current design and proposes one intent-preserving manufacturability improvement at a time, and a CAD Modifier that executes each proposal as an edit to the part's CadQuery program. The CAD Modifier closes a verification loop, compiling every candidate edit and visually checking it against the intended change from multi-view renderings, then re-generating or re-instructing until the edit is accepted or abandoned. Iterating review and verified modification compounds edits into parts more complex than one-shot generators reliably produce, preserves the original intent at each step, and requires no fine-tuning. On a 46-part
benchmark scored by chamfer distance to reference geometries, the CAD Modifier reproduces target parts more accurately on average than chain-of-thought single
agents given the same tools, and ablations isolate the contributions of the visual review loop and of captioning the design state before each edit. A centrifugal pump casing built from 32 chained transitions illustrates the complexity reachable by compounding verified edits. This is a preliminary report: evaluation of the full review-and-redesign loop, including manufacturability gain and an operational measure of intent preservation, is ongoing.
Read the full abstract ▾
All authors Kojo Welbeck, Xiangyu Shi, Zahra Sadeghi, Qi Zhu, Ping Guo
Computer Vision
Subject-Relative Micro-Motion and Sleep Dynamics for Near-Infrared Video Sleep Staging
Kunmin Jang, You Rim Choi, Hun Heo et al. (+5 more)
Near-infrared (NIR)
video is a promising modality for contactless sleep monitoring, but recent
video-based sleep staging methods often use it as a route to reconstructed respiratory/cardiac proxies or cross-modal physiological representations. We study
video-only sleep staging under labels defined by polysomnography (PSG), where the model infers sleep stages from NIR
video alone without explicit physiological proxy reconstruction or auxiliary physiological signal supervision. This tests whether NIR
video itself can provide informative sleep-stage evidence, rather than only serving as an input for recovering physiological proxies. We propose ViNUSS (
Video-Native Unmediated Sleep Staging), a framework that combines subject-relative micro-motion learning with full-night sleep dynamics modeling. Spatially anchored pre-spatial micro-motion encoding preserves localized temporal variation together with its spatial context. Within-subject stage contrast learns stage cues with respect to each subject's night-specific baseline. Two-scale sleep dynamics modeling captures within-epoch motion
evolution and organizes epoch-level evidence into a coherent full-night sleep-stage trajectory. On 475 overnight NIR recordings (~3,250 hours), ViNUSS achieves 0.80 accuracy and 0.78 macro-F1 for four-class sleep staging. Interpretability analysis suggests
attention to thoraco-abdominal periodic motion and gross body movements associated with arousals and position changes. These results support NIR
video as an independently informative and complementary modality for PSG-defined sleep-stage estimation
Read the full abstract ▾
All authors Kunmin Jang, You Rim Choi, Hun Heo, Heonjun Lee, Suahn Bae, Dongik Park, Hyun-Woo Shin, Hyung-Sin Kim
Computer Vision
DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models
John J. Han, Adam Schmidt, Muhammad Abdullah Jamal et al. (+2 more)
Vision foundation models (VFMs) are valuable in data-scarce domains such as surgery, where a single pretrained backbone can provide rich representations for many downstream tasks. Yet the dominant self-supervised pretraining paradigm uses only RGB images, leaving readily available complementary signals, such as depth maps, unused. This is a particular missed opportunity in surgery, where natural-image VFMs transfer poorly while the scene geometry is rich and informative. With strong off-the-shelf models now able to produce pseudo-labeled dense depth for any image corpus, we hypothesize that such signals can be folded into pretraining to learn better representations. We present DART, an RGB-D pretraining recipe that builds on DINOv2 with a simple modification: a pixel-space depth reconstruction objective applied to masked iBOT patches, supervised by pseudo-labeled depth. Depth is used only during pretraining, so fine-tuning and inference remain RGB-only. We find that this pixel-level reconstruction head improves representation quality rather than disrupting it. We further show that depth, which encodes scene geometry, is more effective as a target than alternative dense signals such as Canny edges, confirming that the gains stem from depth rather than added supervision alone. Across eight
surgical benchmarks spanning
segmentation, depth estimation, and image-level recognition, DART outperforms both natural-image and in-domain baselines, including a vanilla DINOv2 trained on identical data, improving dense prediction while also strengthening image-level understanding. More broadly, DART shows that freely available
geometric pseudo-labels can strengthen foundation model pretraining without extra labels or added inference cost, pointing toward stronger backbones for surgery.
Read the full abstract ▾
All authors John J. Han, Adam Schmidt, Muhammad Abdullah Jamal, Jie Ying Wu, Omid Mohareri
Computer Vision
VISTA: Dense Multi-Label Classroom Coding with Vision-Language Models
Andrew Franck, Brendan Ng, Ben Fitzgerald et al. (+3 more)
Video-
language benchmarks are usually constructed by the
dataset authors without published reliability statistics, leaving the noise floor of the construct unknown. We argue that
multimodal benchmarking benefits from methods taken from research communities that have already invested in strategies to ensure reliability. We illustrate the case with the Classroom Observation Protocol for Undergraduate STEM (COPUS): a 24-code multi-label observation instrument with a decade of peer-reviewed reliability literature. We recast COPUS as a
video benchmark for
multimodal foundation models, where it provides a dense set of structured labels (a 24-dimensional binary vector every 2 minutes across a 50-90 minute lecture), an externally validated vocabulary, and established literature that provides a per-code reliability target based on human evaluators. Annotations in our evaluation corpus are produced by a 5-person human-evaluator panel whose consensus
matrix is our reference. We propose VISTA, a baseline that runs MiniCPM-V-4.5 over a dense sliding window, refines its per-window outputs with a lightweight multi-layer perceptron (MLP) head trained on top of the frozen backbone, and max-pools the resulting predictions onto the 2-minute COPUS grid. On three held-out chemistry lectures, VISTA reaches 80.1% restricted macro accuracy versus 74.9% for the zero-shot variant, with the largest residual errors on visually similar instructor codes and on rare audio-dependent codes. We characterize three systematic failure modes (audio-partial observability, fine-grained group-work discrimination, long-tail recall) and release the
benchmark tooling,
prompts and baseline code at https://github.com/ajfranck/VISTA.
Read the full abstract ▾
All authors Andrew Franck, Brendan Ng, Ben Fitzgerald, Zane Derrod, Chris Cianci, Chris Craney
Computer Vision · Cross-listed from cs.RO
SocioGesture: Real-Time and Adaptive Social Gesture Perception for Human-Robot Interaction
Wenjin Fu, Li-Fan Wu, Jerin Peter et al. (+3 more)
Robots interacting with people must recognize not only explicit commands, but also social cues such as invitations, refusals, and unavailability. In real deployments, these cues must be inferred from noisy onboard perception under partial occlusion, changing viewpoints, and strict latency constraints. We present SocioGesture, a real-time adaptive social gesture perception system for human-
robot interaction (HRI). SocioGesture uses a compact confidence-aware body-hand skeleton representation and a lightweight dual-stream model that fuses body motion with hand articulation for low-latency onboard recognition. To improve deployment robustness, we train the model with occlusion-aware skeleton corruption, exposing it to missing hands, occluded arms, and temporally unstable keypoints without increasing the inference cost. On a social gesture
dataset collected in mixed indoor-outdoor HRI scenarios, SocioGesture achieves strong held-out-subject recognition, substantially improves robustness under structured joint occlusion, and runs in real time on a
robot-mounted edge device. During deployment, uncertain interaction segments are saved for offline labeling and adaptation, enabling SocioGesture to expand its gesture vocabulary while preserving performance in the original classes. These results demonstrate a practical path toward robust, efficient, and adaptive social perception for interactive
robots.
Read the full abstract ▾
All authors Wenjin Fu, Li-Fan Wu, Jerin Peter, Chip Huyen, Boyuan Chen, Jan Liphardt
Computer Vision · Cross-listed from eess.IV
Explainable Multimodal Deep Learning Integrating Imaging and Clinical Data for Oral Potentially Malignant Disorder Detection
Ruilin You, Yihan Wang, Jiabin Chen et al. (+4 more)
Oral potentially malignant disorders (OPMDs) are critical precursors to oral cancer, yet clinical detection remains challenging because of substantial phenotypic heterogeneity and overlap with benign conditions. Although image-based deep learning shows promise for automated screening, visual information alone may be insufficient in real-world settings, where diagnostic decisions also rely on patient-specific risk factors. We developed M2-OPMDNet, a
multimodal deep learning framework that integrates co-registered white-light and autofluorescence intraoral images with structured clinical information for OPMD detection. A customized questionnaire was designed to capture clinically relevant risk factors and symptoms in a standardized, reproducible format for integration with image-derived features. Multiple image encoders, including conventional convolutional neural networks and foundation model-based architectures, were evaluated using a prospectively collected
dataset reflecting real-world screening conditions. Model interpretability was assessed using SHapley Additive exPlanations (SHAP) to quantify feature- and modality-level contributions. M2-OPMDNet achieved an AUC of 0.952, outperforming unimodal approaches and showing improved performance for visually subtle lesions. SHAP analysis demonstrated that structured clinical variables contributed substantially to risk estimation and complemented imaging features. These results demonstrate that explainable
multimodal learning combining white-light and autofluorescence imaging with structured clinical data can provide accurate, transparent, and clinically grounded OPMD detection. M2-OPMDNet offers a scalable framework for real-world oral cancer screening and decision support.
Read the full abstract ▾
All authors Ruilin You, Yihan Wang, Jiabin Chen, Cherie Wink, Petra Wilder-Smith, Rongguang Liang, Bofan Song
Computer Vision · Cross-listed from eess.IV
Fractional-Order Adaptive Motion Magnification: Phase-Reliability Weighting for Noise-Constrained Video Amplification
Alejandro Garnung Menéndez
Eulerian
video amplification boosts sub-pixel motion by band-pass filtering per-pixel intensity traces and applying a uniform gain. That gain ignores local structure, so sensor noise is amplified together with the signal, especially in textureless regions where the monogenic phase is unreliable. We propose FrAM (Fractional-order Adaptive Motion Magnification), a pipeline developed first offline and then as a causal stream. It replaces the constant temporal gain with a Grünwald–Letnikov derivative of fractional order, giving continuous control over high-frequency emphasis, and replaces the uniform spatial gain with a per-pixel weight derived from the local amplitude of the monogenic signal. On a controlled synthetic sequence split into textured and flat halves, FrAM matches the amplification of the Eulerian baseline while keeping flat-region temporal noise at the input level. The reduction holds across an eightfold range of input noise levels. Real
videos show improved spatial selectivity and lower background noise in every case. The causal reformulation cuts the per-frame cost by two orders of magnitude, reaching 69\,fps at 640$\times$480.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.HC
EyeMakeYou: Identity-, Task-, and Subjective-State-Conditioned Diffusion for High-Frequency Gaze Synthesis
Kamrul Hasan, Mehedi Hasan Raju and Oleg V. Komogortsev
Eye movement biometrics (EMB) is an emerging behavioral modality for user authentication, particularly in virtual- and augmented-reality systems, where gaze dynamics contain distinctive subject-specific features. However, robust EMB systems require diverse, high-quality gaze recordings that are expensive to collect and often unavailable at the scale needed for model development. Generative models can mitigate data scarcity, but existing methods either synthesize generic gaze behavior or personalize signals primarily by identity, without jointly representing the user's task and subjective state. Consequently, generated signals may appear visually realistic while failing to retain the behavioral properties required for biometric applications. To address this limitation, we propose EyeMakeYou, a multi-conditional denoising
diffusion framework for subject-specific, high-frequency gaze synthesis. EyeMakeYou generates 5-s, 1000-Hz bivariate gaze-velocity sequences from an identity-removed reference trajectory and conditions the denoising process on an identity
embedding, a task
embedding, and self-reported ratings of overall difficulty, mental tiredness, and eye tiredness. Its objective combines
diffusion noise prediction and identity preservation with multi-resolution spectral, drift-consistency, and event-weighted local-smoothness losses. Experiments on GazeBase show that EyeMakeYou achieves higher median spatial accuracy and greater real–synthetic similarity in the
embedding feature space than the existing generative approaches, while retaining selected task-dependent associations between subjective reports and oculomotor features. These findings support conditional
diffusion as a practical approach for augmenting gaze
datasets for biometric and interactive applications.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.GR
STyMo: Fast and Controllable Few-Shot Motion Style Transfer
Jose Luis Ponton, Alexander Winkler, Ladislav Kavan et al. (+2 more)
Supporting a wide variety of motion styles is critical for creating diverse virtual characters, but current methods either require large stylized
datasets or pre-trained models that cannot generalize beyond their training distribution. We present STyMo, a few-shot approach that learns motion style from only seconds of paired data and trains in one to two minutes. Our key insight is to decompose style into two components: a static component capturing time-invariant posture, and a temporal component capturing frame-wise dynamics. This decomposition yields an interpretable system where posture intensity, temporal exaggeration, and per-body-region style can be adjusted at runtime. Furthermore, the reduction in required training data and computation time structurally permits an iterative authoring workflow. To ensure robustness on arbitrary inputs, we further introduce a stylizability gate that automatically prevents artifacts on out-of-distribution motions. We demonstrate results across diverse motion styles, from subtle emotional variations to exaggerated character archetypes, and release our processed paired
dataset to facilitate future research.
Read the full abstract ▾
All authors Jose Luis Ponton, Alexander Winkler, Ladislav Kavan, Yuting Ye, Petr Kadlecek
Computer Vision
Topology-Aware Training and Spatial Diagnostics for Fiber Bundle Segmentation in Tracer Histology
Joselyn Romero Avila, Kyriaki-Margarita Bintsi, Ermias Habte et al. (+3 more)
Anatomic tracer studies reveal how axon bundles project from an injection site, branch into smaller groups of axons, and course through the
brain to reach their destinations. Histological data from such studies provide anatomical reference information for validating
diffusion MRI tractography. However, manual annotation of the histological data is very labor-intensive, and although automated
segmentation methods have been proposed, they rely mainly on pixel-overlap losses such as BCE and Dice;
topology-aware loss functions have not been studied for this task. We compare BCE-Dice, clDice, Betti matching, and Topograph for fiber bundle
segmentation in macaque tracer histology using a frozen DINOv3 backbone. To our knowledge, this is the first exploration of foundation-model features for this task. BCE-Dice achieved the highest Dice, while clDice achieved the highest bundle recall but poor mask overlap. Topograph had similar Dice to BCE-Dice, the lowest $\beta_0$ error, and fewer false positives than BCE-Dice and Betti matching. Fiber bundle
segmentation methods are typically evaluated with a permissive rule that counts a bundle as detected given any overlap with the prediction. We show this rule does not capture oversegmentation, and that per-section TPR can be inflated by empty sections assigned perfect recall. To quantify this, we introduce Excess32, a spatial diagnostic measuring predicted pixels outside a 32-pixel tolerance band around annotated bundles. In validation, a Betti-Topograph union raises sparse-bundle TPR from 0.818 to 0.933, but worsens FDR from 0.296 to 0.509, Excess32 from 0.108 to 0.466, and area ratio from 0.94 to 3.34. These results show detection metrics alone are insufficient to characterize
segmentation quality.
Read the full abstract ▾
All authors Joselyn Romero Avila, Kyriaki-Margarita Bintsi, Ermias Habte, Julia F. Lehman, Suzanne N. Haber, Anastasia Yendiki
Computer Vision
Segmentation of the aorta in 4D flow MRI using 4D convolutional kernels and learning from sparse annotations
Hinrich Rahlfs, Julio Garcia, Chiara Manini et al. (+13 more)
Automated aortic
segmentation in 4D flow MRI is essential for reproducible hemodynamic assessment but is limited by scarce dense annotations and high computational demands. We developed a fully automated 4D (3D+time) U-Net for segmenting the ascending aorta, arch, and proximal descending aorta, using a parameter-efficient hybrid 4D kernel to capture temporal context and sparse 4D labels derived from existing 2D expert contours and centerlines, thereby avoiding the need for dense 4D annotations. Training comprised 268 scans from 8 centers and 2 vendors, with evaluation on an internal test set (32 scans) and an external post-contrast set (30 scans; different site, protocol, and annotator), compared against frame-wise 3D networks and two semi-automatic references. Against time-resolved annotations, the 4D U-Net achieved Dice scores of 0.927 (internal) and 0.911 (external), versus 0.919/0.847 for the 3D U-Net, 0.893 for static PC-MRA, and 0.808 for registration-based propagation; differences were small in systole but pronounced in diastole. Agreement with expert contours for peak velocity, net flow, axial and circumferential wall shear stress, and diameters was excellent (ICC >=0.954 internal, >=0.980 external), while semi-automatic references performed worse. The method thus provides reproducible, time-resolved aortic
segmentation for automated hemodynamic analysis and generalizes across multicenter, multivendor, and independent post-contrast data. The model is publicly available.
Read the full abstract ▾
All authors Hinrich Rahlfs, Julio Garcia, Chiara Manini, Markus Hüllebrand, Sebastian Schmitter, Sarah Nordmeyer, Titus Kühne, Heiko Stern, Christian Meierhofer, Andreas Harloff, Sebastian Kelle, Alexander Lenz, Peter Bannas, Jeanette Schulz-Menger, Ralf F Trauzeddel, Anja Hennemuth
Computer Vision
Shidu Ren, Yunze Liu, Xing Liu et al. (+3 more)
Long-horizon
multimodal agents should remember not only what happened but also who participated. This capability depends on linking recurring faces, voices, names, person-associated objects, events, and social relations to consistent identities over time. Existing long-
video and
multimodal-
agent benchmarks measure broad
memory question answering, but they do not isolate the ability to maintain recurring person identities and reason over their cross-time relations. We introduce ICM-Bench (Identity-Centric
Memory Benchmark), which, to the best of our knowledge, is the first
benchmark specifically designed to evaluate identity-centric
reasoning over long
video memories in
multimodal agents. The
benchmark contains 839 synthetic clips spanning 141 minutes and 1,217 open-ended questions about six recurring adults in a one-year life album. A theme-configurable pipeline generates the
video collection and associates each question with its target identities and traceable supporting evidence. We compare direct caption-
memory baselines,
memory-augmented
agents, and graph-
retrieval systems. Gemini 3.1 Pro achieves the highest overall accuracy of 74.0%, yet its score falls to 60.3% on questions that require long-term identity profiles. The results show that current systems recover many event-level
memories but remain less reliable when evidence must be accumulated around a stable person.
Read the full abstract ▾
All authors Shidu Ren, Yunze Liu, Xing Liu, Chi-Hao Wu, Enmin Zhou, Junxiao Shen
Computer Vision · Cross-listed from cs.ET
STEMPix: A Phase-Transition-Material-Based Pixel Sensor for Resolving Edge-Movement Direction
Md Rahatul Islam Udoy, Sumeet Kumar Gupta, Deep Jariwala et al. (+1 more)
This paper proposes a spatio-temporal edge-movement direction pixel (STEMPix) for generating compact direction-aware edge movement information inside a CMOS-compatible image sensor array. The proposed design targets specialized sensing applications where local boundary movement is more important than full-frame intensity reconstruction. Instead of transferring full multi-bit frames for external processing, STEMPix generates a 3-bit local edge direction code (LEDC) by combining pixel-level temporal change information with neighboring-pixel spatial edge information. We design the architecture using a two-tier organization, where the photodiode layer is separated from the computation layer to preserve light-collection area while accommodating the additional in-array processing circuitry. The proposed
circuit is evaluated through HSPICE transient simulations. The estimated implementation achieves a horizontal pitch of 1.73 {\mu}m, a vertical pitch of 2.36 {\mu}m, and a
geometric fill factor of 95.47%. The average active switching energy is 0.465 fJ per LEDC operation across representative edge-movement cases. The proposed STEMPix operation also supports global-shutter capture and dynamic thresholding. These results indicate that STEMPix can provide a compact and scalable front-end representation for edge-movement-aware sensing systems.
Read the full abstract ▾
All authors Md Rahatul Islam Udoy, Sumeet Kumar Gupta, Deep Jariwala, Ahmedullah Aziz
Computer Vision · Cross-listed from cs.RO
AquaBEV: Monocular Underwater BEV Occupancy with 3D Sonar Supervision
Trung Tien Dong, Shengji Jin, Chen Chen et al. (+2 more)
Autonomous
underwater robots are widely used for exploration, monitoring, and inspection, where safe
navigation depends on understanding the surrounding free and occupied space. Bird's eye view (BEV) occupancy provides such a representation, but predicting it from a single
underwater RGB image is difficult due to limited, unreliable
geometric cues from appearance alone. 3D imaging sonar offers complementary
geometric measurements to supervise this task. We introduce AquaBEV, a
monocular underwater occupancy model that predicts local BEV occupancy from a single RGB image, using paired 3D imaging sonar as
geometric supervision during training. AquaBEV maps visual features into a calibration free polar representation and applies causal decoding along the range dimension before reconstructing the prediction in Cartesian BEV coordinates. A controlled
underwater occupancy
benchmark was established, adapting representative occupancy methods to the same RGB to sonar task under a unified protocol. AquaBEV achieves 31.4 Visible IoU and 38.6 Observed IoU, 4.0% and 4.3% relative improvements over the strongest transferred baseline.
Read the full abstract ▾
All authors Trung Tien Dong, Shengji Jin, Chen Chen, Yi Sheng, Xiaomin Lin
Computer Vision
FAVE: Foveated Adaptive Visual Encoding for Efficient Fine-Grained Visual Understanding
Amitangshu Mukherjee and Kaushik Roy
Fine-grained visual understanding depends on local detail, yet visual encoders face a trade-off between costly full-image high-resolution processing and compact global encoding that can weaken such evidence. Inspired by human active
vision, we separate where to look from what to encode. We focus on the latter and introduce FAVE (Foveated Adaptive Visual Encoding), a lightweight variable-resolution ViT that encodes externally selected regions at high acuity while preserving native geometry. We first isolate this encoding problem using oracle ground-truth crops in a controlled small-object regime. On ImageNet objects with a native maximum side of 96 pixels, FAVE improves Top-1 by 9.4 points over a fixed-resolution ViT on the same crop window with 12.7 times lower FLOPs. Increasing global resolution or backbone capacity does not recover the same operating point. We then integrate FAVE as a complementary local branch in FastVLM. Its local
tokens are combined with FastVLM's global visual
tokens, while the original global pathway and
language model remain frozen. With at most 16 additional local
tokens, FAVE improves TextVQA by 1.60 points and achieves a 3.3 times controlled TTFT speedup over SmolVLM2-2.2B. On GQA attribute questions, it improves FastVLM-1.5B by 1.31 points, extending the benefit beyond text while narrowing the gap to FastVLM-7B. Together, these results show that selectively allocating high-acuity local capacity provides an efficient complement to broader global representations and model scaling for fine-grained understanding of small objects, text, and attributes.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Development and Evaluation of Ultrasound Image Learning Pipelines for MASLD Risk Stratification
Guangyi Zhang, Xiaohong Wang, Eugene Cheah et al. (+4 more)
Metabolic dysfunction-associated steatotic liver disease (MASLD) affects approximately 30% of the general population. Ultrasound-based imaging, including B-mode imaging and shear wave elastography (SWE), is widely used for noninvasive fibrosis assessment; however, the role of deep learning-based ultrasound image learning for MASLD risk stratification remains insufficiently characterized. In this study, we developed and evaluated ultrasound image learning pipelines using B-mode and SWE images for fibrosis staging and identification of patients with at-risk metabolic dysfunction-associated steatohepatitis (MASH). A total of 250 ultrasound examinations, one exam per subject, were included. Model performance was evaluated using 3-fold cross-validation with area under the receiver operating characteristic curve (AUROC). End-to-end SWE image learning achieved performance comparable to operator-guided SWE across fibrosis stages. Overall, SWE-based learning consistently outperformed B-mode image learning in fibrosis staging, with AUROC improvements from 0.64 (95%CI: [0.56, 0.72]) to 0.72 (95% CI: [0.65, 0.79]) for F>=2 (significant fibrosis, p=0.11), from 0.67 (95%CI: [0.58, 0.75]) to 0.78 (95% CI:[0.72, 0.85]) for F>=3 (advanced fibrosis, p=0.02), and from 0.69 (95%CI: [0.56, 0.82]) to 0.80 (95%CI: [0.72, 0.89]) for F4 (cirrhosis, p=0.10). These findings highlight the potential of SWE image learning for MASLD risk stratification.
Read the full abstract ▾
All authors Guangyi Zhang, Xiaohong Wang, Eugene Cheah, Peng Guo, Brian A. Telfer, Theodore T. Pierce, Anthony E. Samir
Computer Vision
What Moves? Localized Motion Representations for Compositional Scene Control
Frank Fundel, Malek Ben Alaya, Thomas Ressler-Antal et al. (+2 more)
Real-world dynamics are inherently compositional: multiple entities move simultaneously within a shared scene, each exhibiting distinct motion patterns. Yet current motion representation models entangle the dynamics of different entities, without explicitly capturing localized motion for each individually. Crucially, motion is defined relative to a global reference frame, including camera motion and scene layout. However, localized
embeddings are often computed from cropped images or obtained by masking features after encoding, discarding the context needed to interpret motion. To address this, we introduce a promptable localized motion representation that produces persistent
embeddings for user-specified regions defined by spatial masks. Rather than cropping the input or masking features, our model processes the full
video and conditions motion encoding directly on the queried region. This yields temporally consistent, region-addressable
embeddings that isolate local dynamics while retaining the global context required for disambiguation. We demonstrate object-level motion transfer, enabling controlled composition of dynamic scenes. Beyond generative control, our
embeddings support localized action classification in multi-actor
videos. Across both tasks, our approach improves controllability and outperforms global representations localized through cropping or post-hoc masking.
Read the full abstract ▾
All authors Frank Fundel, Malek Ben Alaya, Thomas Ressler-Antal, Stefan Andreas Baumann, Björn Ommer
Computer Vision
What Do Scan-Derived Class Prototypes Add? Disentangling Supervision, Prototype Content and Query Protocol in Recognition over Frozen Foundation Features
Chenxi Tao, Hong-In Won and Seung-Kyum Choi
A scan supplies labeled images and a
geometric reference. We separate their contributions in a recognizer whose scan-derived prototype
matrix acts as a supervised head's fixed output layer. On T-LESS, HOPE and 18 self-collected industrial parts, we test real, random and exactly permuted prototypes, matched geometry-free classifiers, stronger appearance rules and paired background protocols. Across DINOv2-
giant and MetaCLIP-H with real-background queries, the largest fused-accuracy advantage of the real prototypes over either control is one percentage point; larger differences favor controls, by up to 2.8 points in arm means. On HOPE with DINOv2-
giant the head alone is 2.8 points above exact permutations (95% interval: 0.8-4.7); this advantage does not reach fusion and is not observed on MetaCLIP-H. On DINOv2-
giant, matched logistic regression comes within 0.5 points of fusion on T-LESS and exceeds it on HOPE and the self-collected parts. Against white cutouts, real HOPE query backgrounds lower image-prototype accuracy by 43 points on DINOv2-
giant and 13 on MetaCLIP-H. The audit separates prototype content, label supervision and query protocol.
Read the full abstract ▾
Computer Vision
AdaptVPR: Route-Aware Hard Positive Generation for Robust Visual Place Recognition
Shunpeng Chen, Jingyi Zhang, Changwei Wang et al. (+6 more)
Visual Place Recognition (VPR) localizes a query image by retrieving database images of the same or nearby place, yet its robustness is often degraded by domain shifts arising from illumination, weather, seasonal changes, and dynamic occlusions. One contributing factor is the limited appearance diversity of the same place in existing training data. To address this issue, we propose AdaptVPR, a route-aware generative augmentation framework that constructs same-place hard positives for robust VPR training. AdaptVPR first uses a
vision language model to parse scene attributes and estimate editing feasibility, while a rule-based scheduler determines the generation route according to editability scores and risk constraints. The generation process is decomposed into three complementary routes: the Global Appearance Route introduces global scene changes in weather, illumination, and time of day; the Local Occlusion Route inserts plausible dynamic occluders; and the Dual Route combines both types of perturbations to produce more challenging appearance shifts. Each generated candidate is evaluated using a VPR-oriented verification scheme based on
geometric consistency and appearance diversity, reducing the risk of structural drift while ensuring sufficient appearance variation. Global candidates are generated once and rejected if verification fails, while Local Occlusion and Dual candidates use verification feedback for limited
prompt refinement and regeneration. Using this framework, we construct AdaptCities, containing 160K verified synthetic same-place hard positives. Experiments across multiple VPR baselines and
vision foundation backbones show consistent gains on standard
benchmarks and substantial improvements under challenging domain shifts, with R@1 gains of up to 9.2%. The source code and data resources are publicly available at https://github.com/chenshunpeng/AdaptVPR.
Read the full abstract ▾
All authors Shunpeng Chen, Jingyi Zhang, Changwei Wang, Shengpeng Xu, Yukun Song, Xingtian Pei, Jinzhou Lin, Li Guo, Shibiao Xu
Computer Vision · Cross-listed from eess.IV
Ultrasound-Based Prediction of Cirrhosis Decompensation Using Large-Scale Computer Vision Models
Guangyi Zhang, Peiyun Ni, Eugene Cheah et al. (+4 more)
Decompensation represents a critical transition in the course of cirrhosis, yet clinicians have limited non-invasive tools to reliably predict its onset. In this study, we propose a novel imaging-based approach that leverages large-scale computer
vision models to analyze routine abdominal ultrasound images and extract predictive features beyond those captured by traditional laboratory-based risk scores. Ultrasound is widely available, low cost, and suitable for longitudinal surveillance, making it an attractive modality for scalable risk stratification and long-term follow-up. Our framework integrates automated ultrasound data processing with modern deep learning architectures to identify patients at high risk of decompensation prior to the occurrence of clinical deterioration. This non-invasive strategy offers a practical complement to existing clinical scoring systems and may enable earlier, more proactive management of patients with compensated cirrhosis.
Read the full abstract ▾
All authors Guangyi Zhang, Peiyun Ni, Eugene Cheah, Rajat Chandra, Peng Guo, Raymond T. Chung, Anthony E. Samir
Computer Vision · Cross-listed from eess.IV
Cross-modal triage network: a multimodal deep learning framework for severity-based triage and visual explainability in chest radiographs
Zinah Ghulam, Richa Mittal and Eranga Ukwatta
Purpose: Increased number of chest radiograph (CXR) scans create a triage bottleneck, queueing urgent examinations behind routine ones. Existing AI tools are predominantly unimodal binary classifiers lacking severity awareness, and
multimodal systems are rarely benchmarked against expert radiologists. To this end, we developed a
multimodal deep learning framework for joint severity triage, pathology detection, and native visual explanation. Approach: We propose the cross-modal triage network (CMTN), fusing a Swin
Transformer V2 visual encoder with a PubMedBERT text encoder via gated cross-
attention. The CMTN was trained on 34,639 image-text pairs (12,489 patients) from MIMIC-CXR-JPG, optimizing an ordinal focal loss for four-tier severity triage and binary cross-
entropy for 14 pathologies. Beyond quantitative benchmarking,
attention heatmaps were evaluated against a blinded expert radiologist in a two-phase clinical audit comparing model triage output to expert severity assessment (100 cases) and grading spatial-semantic concordance (116 heatmaps). Results: The CMTN achieved strong ordinal agreement with reference labels (quadratic weighted kappa [QWK] = 0.9341, 95% CI: 0.9219 to 0.9449) and macro-AUROC of 0.9970 across 14 pathologies, with 34~ms latency, outperforming the state-of-the-art BioViL
multimodal baseline (QWK = 0.7679). However, the blinded Phase I clinical audit revealed substantially lower agreement with genuine radiologist judgment (QWK = 0.1399). Phase II found 54.3% of heatmaps achieved clinically acceptable spatial localization. Conclusions: The CMTN demonstrated an efficient
multimodal architecture for CXR triage. The divergence between algorithmic and radiologist agreement demonstrates that
benchmark performance against NLP-derived labels is insufficient, highlighting the need for radiologist-labeled ground truth before clinical deployment.
Read the full abstract ▾