Computer Vision
EgoSIS: From Factorized Visual Ego-Transitions to Motion-Canonical Spatial Evidence for UAV Reasoning
Jingpu Yang, Fengxian Ji, Mingxuan Cui et al. (+4 more)
UAV video question answering requires separating camera motion from changes in the scene, but RGB-only
multimodal models receive no explicit, stable reference for that separation. We present EgoSIS, a pose-free adapter that converts RGB-derived bidirectional flow into motion-canonical visual evidence in three stages. Factorized Visual Ego-Transitions (FVET) fits a robust image-plane transition and exposes motion, residual-support, and reliability factors. Reliability-Gated Ego-Transition
Memory (ReTEM) uses reliability-weighted updates for a bounded history and re-anchors it at cuts or sustained uncertainty. Ego-Aligned Spatial Evidence (EASE) warps supported visual features into each segment's local anchor and injects four spatial evidence
tokens per visual slice through zero-initialized residuals, without changing Qwen's visual-
token count. On SIS-Bench, EgoSIS-8B obtains 89.9% perception, 82.5% perception-plus-
memory, and 76.2% overall accuracy, with the largest gains concentrated in self-awareness perception and
memory. The adapter thus provides an interpretable interface between optical flow and spatial
reasoning.
Read the full abstract ▾
All authors Jingpu Yang, Fengxian Ji, Mingxuan Cui, Yilin Sun, Hang Zhang, Jianhua Zhu, Yufeng Wang
Computer Vision
Data-Efficient Crosswalk Segmentation from Overhead CCTV via Confidence- and Geometry-Guided Pseudo-Labeling
Abdirashid Omar and Jonghyuk Park
Pixel-level annotation of fixed traffic-camera imagery is expensive, while crosswalk models trained from street-level imagery face a substantial viewpoint and appearance shift when applied to elevated CCTV. We investigate a data-efficient target-domain pipeline using 241 manually annotated CCTV images and 5,926 unlabeled CCTV frames. A source-domain experiment trains a 31.0M-parameter custom U-Net on 3,300 first-person-view (FPV) images and obtains 93.05% IoU on its 330-image FPV test split. This result is a source baseline, not transferred performance: the released CCTV notebook instantiates a 42.0M-parameter DeepLabV3-ResNet50 from torchvision weights, and no compatible mapping from the U-Net checkpoint is implemented. Training on 201 manual CCTV images and selecting on 40 held-out manual masks yields 88.91% IoU. The model then predicts all unlabeled frames; image-level certainty and a largest-component area prior rank the candidates, and the top 1,000 attain mean certainty 0.976 and mean combined score 0.988. A repository audit shows that the reported second-stage 98.52% IoU was measured on a 150-image split containing only teacher-generated pseudo-masks. Because of a directory-layout mismatch, the executed combined-data loader found zero manual samples and split 1,000 pseudo-labeled samples into 850 training and 150 evaluation samples. We therefore report 98.52% as internal pseudo-label agreement rather than human-ground-truth accuracy. The defensible target-domain result is 88.91% IoU on the 40 manual validation images. Batch-one FP32 inference at 512 x 512 requires 12.98 ms, corresponding to 77.03 FPS, on an NVIDIA RTX A6000 48 GB GPU. These findings support the practicality of confidence-and-geometry filtering while also showing why pseudo-label evaluation must remain isolated from the labels used for self-training.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representation
Abdullah Al Shafi
The rapid growth of
multimodal data has intensified the need for question answering (QA) systems capable of
reasoning across heterogeneous sources such as text, tables, and images. In this paper, we present a comprehensive methodological comparison of three influential frameworks, namely
Multimodal Adaptive Extraction (MAE),
Solar, and UniMMQA, tracing the
evolution of
multimodal question answering from modality-adaptive pipelines to fully unified architectures. We examine how each approach models cross-modal interactions, transforms heterogeneous inputs, and performs
reasoning, highlighting key design differences in modality representation,
reasoning, and answer generation. Our analysis demonstrates a clear shift from explicit modality-specific processing toward unified text-centric formulations enabled by pre-trained
language models (PLMs). Empirical comparisons across
benchmark datasets show that this transition leads to substantial improvements in both Exact Match (EM) and F1-Scores, with UniMMQA achieving the most consistent and scalable performance. Despite these advances, we identify persistent challenges, including information loss during modality transformation, error propagation in multi-stage pipelines, and limitations in capturing fine-grained cross-modal dependencies. Overall, this study provides a deeper understanding of current design trends and offers insights into the future direction of unified
multimodal reasoning systems.
Read the full abstract ▾
Computer Vision
FRAME: Factored Retrieval via Attribute Readouts for Object-Centric Scene Memory
Woosang Jeon, Sanghyeok Choi, Minwoo Kim et al. (+2 more)
Language-guided
robots need persistent scene
memories to follow instructions, revisit objects, and resolve references to objects encountered over time. While much of
language-guided scene-
memory retrieval has emphasized spatial or relational references, many everyday object references specify objects by multiple persistent attributes, such as category, material, size, or surface appearance. We formalize this problem as attribute-compositional
retrieval, where a fixed object-centric scene
memory is queried with natural
language to retrieve the object satisfying the requested attributes. To investigate this capability directly, we introduce a controlled evaluation protocol with fixed scene
memories and attribute-defined targets, separating
retrieval from perception and annotation ambiguities. We then propose FRAME, which turns
language into query-relevant attribute weights, uses learned readouts to estimate per-attribute evidence from object
embeddings, and ranks objects by aggregating this evidence according to the query. Across held-out scenes and object assets, FRAME outperforms representative scene-
memory retrieval baselines while reducing post-decomposition object scoring to lightweight
matrix-vector computation. These results position attribute-compositional
retrieval as a complementary scene-
memory capability for
language-guided
robots, showing that persistent object attributes can be exposed as composable evidence for accurate and efficient multi-attribute
retrieval.
Read the full abstract ▾
All authors Woosang Jeon, Sanghyeok Choi, Minwoo Kim, Taehyun Jung, Taehyeong Kim
Computer Vision
Medical AI Encodes a "Feeling of Error": Verifying Cancer Segmentation via Internal Concepts
Mengmeng Ma, Yunxiang Peng, Tang Li et al. (+4 more)
Cancer
segmentation models can fail silently, generating plausible but incorrect masks that risk missed findings or unnecessary biopsies. A critical question arises: Do AI models "know" when they are wrong, and if so, can we use the signal to predict their own failures? Humans do have a "Feeling of Error" (FOE): a spontaneous sense of unease that flags a potential error during thinking. We investigate whether cancer
segmentation models exhibit an analogous internal signal. Unlike output-level cues (e.g., prediction confidence or uncertainty), which offer no insight into why a failure occurs and suffer from a sensitivity-quality tradeoff where high detection sensitivity could degrade overall
segmentation quality. We instead propose to capture the model's FOE from its inner workings. Using mechanistic interpretability tools, specifically Sparse Autoencoders, we decompose internal neural activations into a dictionary of human-interpretable concepts and show that failure cases exhibit a distinct latent signature: fewer active concepts with lower activation magnitudes compared to successful
segmentation. By training a classifier on these concept activations, we achieve accurate failure detection along with explanations for the model's mistakes. Experiments on prostate, pancreatic, and
brain cancer
segmentation demonstrate that our approach outperforms output-based methods in failure detection while preserving
segmentation quality.
Read the full abstract ▾
All authors Mengmeng Ma, Yunxiang Peng, Tang Li, Lu Lin, Binsheng Zhao, Oguz Akin, Xi Peng
Computer Vision
Linnan Zhao, Xu Liu, Lingling Li et al. (+3 more)
Reasoning segmentation converts an implicit linguistic conclusion into a precise mask, requiring both semantic identification and spatial grounding. Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context, although they receive different supervision and fail differently. This coupling obscures whether a failure arises from target interpretation or from localization. We present SeGDeP, an explicit what-where interface. A semantic
prompt branch and an independent
geometric projection path transform resolved MLLM states into semantic features and a DETR-predicted box, which jointly condition a SAM 3 mask decoder. Training first aligns this executable interface, then uses group reward-decoupled policy optimization (GDPO) to balance format, box-IoU, and mask-IoU feedback. SeGDeP-4B reaches 82.7 average cIoU over eight RefCOCO-family splits and 66.0/59.6 gIoU on ReasonSeg val/test while adapting only 0.38% of Qwen3-VL parameters through LoRA. Controlled stage-wise ablations, gradient diagnostics, and
prompt interventions further show that the two paths develop complementary semantic and
geometric specialization rather than duplicating the same evidence.
Read the full abstract ▾
All authors Linnan Zhao, Xu Liu, Lingling Li, Licheng Jiao, Fang Liu, Wenping Ma
Computer Vision
DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding
Zhuo Cao, Bingqing Zhang, Sen Wang et al. (+1 more)
Text-guided
Video Temporal Grounding (VTG) aims to localize the relevant segments in an untrimmed
video based on text queries, yet collecting dense temporal annotations and training task-specific models remain costly and brittle under distribution shift. Recent training-free VTG approaches mitigate this issue by directly matching pretrained
vision-
language representations, but they still face two fundamental information bottlenecks: frame-wise visual encoding overlooks temporal dynamics, while fixed query
embeddings cannot resolve query ambiguity. To address these issues, we propose DSE-VTG, a \underline{D}ual-\underline{S}ide \underline{E}nhancement framework that addresses both without any task-specific training. On the visual side, Multi-scale Similarity Fusion (MSF) combines frame- and clip-level similarities into a unified, temporally aware similarity profile. On the textual side, Query-level Test-Time Adaptation (Q-TTA) optimizes a lightweight additive offset to adapt the query
embedding to the
video at test time, without finetuning the backbone or calling external large
language models. Extensive experiments on three standard and two OOD
benchmarks show that DSE-VTG achieves state-of-the-art performance among training-free methods. On Charades-STA, it improves mIoU over the strongest prior training-free method by 5.61 points. Under distribution shift, DSE-VTG reaches 50.86 mIoU on Charades-CG Novel-Word, surpassing the strongest supervised baseline by 2.76 mIoU. Our code will be released upon acceptance.
Read the full abstract ▾
All authors Zhuo Cao, Bingqing Zhang, Sen Wang, Xue Li
Computer Vision
FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute
Hongchi Xia, Tianhang Cheng, Wei-Chiu Ma et al. (+1 more)
We present FIRE3D, a unified framework that transforms a single RGB image or casual
video into interactable 3D scene assets for games and interactive applications in under a minute for up to twelve textured instances including preprocessing. At the core of FIRE3D is a feed-forward inference pipeline that predicts a compositional scene representation from posed RGB-D observations estimated from the RGB capture, including the 6-DoF pose, bounding box, mesh, and texture for detected objects. By modeling the scene as a collection of discrete entities, FIRE3D produces amodal object assets that can be independently edited and used in interactive applications. Our framework requires no test-time optimization, runs substantially faster than the compared reconstruction systems, and provides object-level completeness beyond existing feed-forward 3D approaches. We demonstrate leading detection accuracy, strong geometry reconstruction, and competitive rendering quality on the evaluated
benchmarks, with substantial runtime gains.
Read the full abstract ▾
All authors Hongchi Xia, Tianhang Cheng, Wei-Chiu Ma, Shenlong Wang
Computer Vision
Leveraging Visual and Geometric Priors for Metric-scale and Complete Vehicle Gaussian Reconstruction from Limited Views
Jinyu Miao, Jiusi Li, Yifei He et al. (+4 more)
High-fidelity vehicle assets are essential for controllable traffic scene generation, particularly for synthesizing rare and safety-critical long-tail scenarios. However, reconstructing a reusable vehicle representation from in-the-wild onboard images remains challenging for two reasons. First, image-to-3D generation methods generally produce models without reliable metric scale. Second, onboard cameras usually observe only one side of a target vehicle, making conventional multi-view reconstruction incomplete on unobserved regions. To solve these problems, we propose a feed-forward vehicle asset reconstruction method, which leverages two complementary priors to reconstruct 3D Gaussian representations for vehicles using sparse one-sided observations. To achieve metric-scale reconstruction, a visual foundation model is first utilized to serve as a visual prior for Gaussian initialization. The Gaussian attributes are then estimated by a learnable encoder-decoder module. A
symmetry-aware cloning strategy is presented to complete the unobserved side directly in Gaussian space, which exploits the bilateral structure of vehicles as a
geometric prior. Experiments on the public
dataset demonstrate that the proposed method significantly outperforms existing approaches in both vehicle asset completeness and
geometric accuracy.
Read the full abstract ▾
All authors Jinyu Miao, Jiusi Li, Yifei He, Miao Long, Kun Jiang, Mengmeng Yang, Diange Yang
Computer Vision
Beyond Gait: Person Identification from Millimeter-Wave Point Clouds Across Activities of Daily Living
Xilai Wang, Zixiong Han, Saad Rhanmouni et al. (+3 more)
Person identification from millimeter-wave (mmWave) point clouds has mainly relied on gait. Indoor walking, however, is often brief and interrupted, while other activities of daily living (ADLs) may provide complementary identity information. We investigate identification across seven ADLs using mm-ADL, a new point-cloud
dataset collected from 11 subjects under a controlled protocol. This extension introduces heterogeneous states and transitions whose spatial and temporal characteristics vary with activity. We therefore study whether activity can provide useful context for learning identity representations. We propose an activity-conditioned framework in which a human activity recognition router dispatches each clip to an activity-specific identity expert. The framework is implemented as a supervised mixture of experts, using a dual-stream static-dynamic PointNet (DS-SDPNet) to combine time-aggregated spatial structure with frame-to-frame information. We evaluate closed-set identification (ID) and subject-disjoint re-identification (ReID). With learned hard routing, ID accuracy increases from 62.1% to 68.0%. In a two-occupant ReID setting, hard routing increases mAP from 57.2% to 75.4% and Rank-1 accuracy from 59.1% to 82.1%. Under a matched gallery partition, activity-specific experts also outperform a shared
embedding, showing that the gain extends beyond restricting the gallery. These results support the feasibility of using ADLs beyond gait for identification and the value of activity conditioning under controlled indoor conditions.
Read the full abstract ▾
All authors Xilai Wang, Zixiong Han, Saad Rhanmouni, Chenzhe Zhao, Yunze Lu, Miodrag Bolic
Computer Vision
ArmPoser: Real-Time, Calibration-Free Arm Pose Estimation from Smartwatch IMU
Bishnu Dev, Vasco Xu, Xi-Aan Loh et al. (+3 more)
Arm pose estimation enables applications in fitness, extended reality input, rehabilitation, and life logging. Prior smartwatch-based approaches rely on calibration poses and preprocessing pipelines that transform raw IMU measurements into standardized training formats. These steps hinder deployment in everyday settings and introduce errors due to imperfect calibration and sensor drift. We present ArmPoser, a calibration-free arm pose estimation system using a single smartwatch IMU. Our central contribution is training models directly in the reference frame native to consumer smartwatches, aligning learning with how IMU data is produced by deployed devices. By operating on device-native axes, ArmPoser removes the need for coordinate transformations, explicit
alignment, and bone-offset calibration used in prior work. We further augment training with physically grounded variations in watch placement and arm morphology to account for user-specific variability. ArmPoser also includes a wear-configuration module that infers anterior or posterior forearm placement and crown orientation. We evaluate pose estimation on public
benchmarks and on a 10-participant, 30-activity study using watchOS and Android smartwatches, where ArmPoser matches or exceeds calibrated baselines without any user calibration.
Read the full abstract ▾
All authors Bishnu Dev, Vasco Xu, Xi-Aan Loh, Chenfeng Gao, Henry Hoffmann, Karan Ahuja
Computer Vision
Evaluation Principles for MRI-MRA Registration in Trigeminal Neuralgia: An ROI-Centered Neurovascular Benchmark
Xupeng Zhang, Xihang Wang, Michael Xie et al. (+6 more)
Preoperative evaluation of trigeminal neuralgia (TN) often requires joint interpretation of structural MRI, which depicts the trigeminal nerve and surrounding cisternal anatomy, and time-of-flight MRA, which highlights vascular structures. Although MRI-MRA fusion is clinically attractive for visualizing neurovascular compression, this task is poorly captured by conventional whole-
brain registration evaluation because the clinically relevant target is a small trigeminal ROI, vessel annotations are partial and clinically focused, local TOF-MRA contrast is variable, and field-of-view mismatch can limit deformable
alignment. We formulate TN MRI-MRA fusion as an ROI-centered neurovascular registration-evaluation problem and construct a
benchmark from 149 patients with clinician-annotated bilateral trigeminal ROIs. Six representative registration pipelines were evaluated using local image-based metrics,
segmentation-derived vessel-localization metrics, prediction-volume analysis, and contrast- and FOV-stratified comparisons. Conventional evaluation summaries were often misleading: local image similarity, vessel-background separability, and downstream vessel localization did not co-rank methods; one-sided vessel distances were strongly affected by predicted vessel extent under partial annotations; and local MRA contrast determined when vessel-separability metrics were informative. Deformable refinement provided only a small, FOV-dependent benefit over affine
alignment, while reader review showed that locally favorable vessel distances could coexist with globally implausible registrations. These findings indicate that TN MRI-MRA registration should be evaluated as a local, vessel-aware, contrast-sensitive, and FOV-aware visualization task rather than as generic
multimodal brain registration. Our code is publicly available at https://github.com/jhuldr/TN-Reg-
Benchmark.
Read the full abstract ▾
All authors Xupeng Zhang, Xihang Wang, Michael Xie, Haoyuan Liang, Hau Ern Lien, Oishika Das, James Feghali, Risheng Xu, Peirong Liu
Computer Vision
Hi-FLoop: Hierarchical State-Feedback Loops for Multi-Timescale World Modeling
Rx Fan and Z Han
Multi-
agent traffic simulation seeks diverse, coordinated, and physically realistic futures from maps and observed history. Long-horizon closed-loop generation must reconcile multiple decision time scales while its context evolves with generated states. Existing methods often unfold long futures from an initial scene and resolve intent, interaction, and motion monolithically, weakening cross-scale consistency and adaptation.
Multimodal rollout poses a further consistency problem: independently reselecting modes across
agents or commits can stitch together incompatible futures instead of preserving a coherent joint branch. We present Hi-FLoop, a branch-consistent multi-timescale state-feedback framework. Eight scene-level Worlds represent joint hypotheses; all
agents share one selected World identity throughout all 16 commits of an 8-second rollout, while Goal, Preview, and Control states adapt within that branch. An 8-second Goal anchors intent, a 2-second Preview coordinates interactions, and 1-second Control produces physical motion. Every 0.5-second commit feeds back only its executed prefix as new facts, while unexecuted hypotheses never enter factual
memory. Joint Preview Interaction induces a sparse directed future graph and uses conflict probabilities and signed arrival-time differences to refine interaction-aware motion. For generated-state recovery, a prefix-frozen A-to-B cascade transfers typed physical state and the branch index–but no latent state–from a frozen prefix model to an independently parameterized recovery model. On the full H-D public-validation split of 955 scenarios, the S2.1 cascade obtains an 8-second scene-joint ADE-at-joint-minFDE@8/joint-minFDE@8 of 2.048/6.384 m when one World must explain all evaluated
agents.
Agent-centric oracle-minADE@8 is 0.526 m at 6 seconds and 0.875 m at 8 seconds.
Read the full abstract ▾
Computer Vision
Interpretable Hyperspectral Unmixing Framework with Fixed Endmember Prior and Structured Residual Refinement
Ziyi Guan, Jianping Zhang and Qian Liu
Hyperspectral unmixing decomposes mixed pixels into material endmembers and their abundances from contiguous spectral observations. In modular sensing pipelines, endmembers are often first identified and then treated as fixed during abundance estimation. When this fixed endmember prior is inaccurate, spatially structured mismatch arising from illumination changes, sensor artifacts, or material boundaries may be incorrectly captured by the abundance variables, leading to unstable decompositions. This study presents an interpretable stage-wise hyperspectral unmixing framework (I-HyperSU) under fixed endmember priors, which is explicitly decomposed into a fixed endmember
matrix $\mathbf{A}$, an abundance block $\mathbf{X}$, and a structural residual refinement block $\mathbf{S}$. The X-block estimates abundances using FISTA with nonnegativity and sparsity enhancement, and a soft penalty that approximately enforces sum-to-one constraints. The S-block jointly applies low-rank SVD structural regularization and a lightweight deep image prior (DIP) to refine structured residuals. This staged design makes the interaction between abundance and residual components transparent and interpretable. Experiments on Samson, Urban, and Jasper Ridge
datasets demonstrate that, under fixed and imperfect endmember priors, soft abundance relaxation consistently outperforms hard simplex projection. Under the default N-FINDR endmember prior, the proposed framework reduces the joint reconstruction error by 61.7%–69.5% compared with a fixed-$\mathbf{A}$ UCLS baseline, while keeping the abundance RMSE nearly unchanged, indicating that the residual refinement branch accounts for structured model mismatch without degrading the abundance estimates. For example, on Urban, the reconstruction SAM decreases from $5.99^\circ$ for the X-only model to $1.92^\circ$ for the full model.
Read the full abstract ▾
Computer Vision
AXS-Net: Interpretable Deep Unfolding for Hyperspectral Image Denoising via Spectral Basis Unmixing and Structured Noise Refinement
Ziyi Guan, Jianping Zhang and Zheng Yang
Hyperspectral images (HSIs) are often degraded by mixed noise, including band-dependent Gaussian perturbations and structured artifacts such as stripes, dead-lines, and impulse noise. Most deep denoisers regress the clean image directly, entangling signal and structured noise. We instead model HSI denoising as $\Y=\A\X+\Snoise+\Nnoise$, where $\A\X$ is a low-rank spectral-subspace (unmixing) reconstruction, $\Snoise$ is structured sparse noise and $\Nnoise$ is residual Gaussian noise. The resulting regularized optimization problem is unrolled into AXS-Net, a $K$-stage alternating proximal-point framework. Each stage combines an analytic spectral-basis gradient step, an SSX-Block proximal operator for abundance coefficients, and an SBlock proximal operator for the structured residual with column-consistent and sparse priors. This optimization correspondence exposes interpretable endmembers, abundance maps, and structured-noise estimates. Across ICVL, CAVE, and Harvard
datasets and five noise configurations, the proposed AXS-Net achieves strong in-domain accuracy and competitive zero-shot transfer, with consistent gains across all five noise regimes on ICVL and Harvard. The recovered structured-noise closely follows the synthetic reference, and the recovered spectral basis is smooth and band-ordered rather than an arbitrary set of latent channels.
Read the full abstract ▾
Computer Vision
Compensating for Scarce Historical Images in Cross-Domain Cultural Heritage Retrieval Using Synthetic Aging
Marcin Iwanowski, Adam Mazgaj, Ferdynand Gorski et al. (+1 more)
Cultural heritage collections often contain contemporary and historical visual records of the same physical object. Linking these records is difficult because corresponding images may differ in viewpoint, acquisition conditions, color reproduction, framing, resolution, and degradation, while genuine historical images are frequently scarce. This study investigates whether synthetically aged contemporary images can replace or complement missing historical training data in bidirectional instance-level
retrieval. Synthetic old-domain images are generated using degradation-oriented transformations. An EfficientNetV2-M model is evaluated on identity-disjoint training, validation, and test sets across three
dataset partitions and three training seeds. Mixed real-synthetic training is compared with real-only baselines using proportionally scaled and fixed 300-batch-per-epoch schedules. Complete replacement of genuine historical images reduced bidirectional mean R@1 from 86.56% to 81.27%, showing that synthetic aging does not reproduce the full genuine old-domain variability. Increasing the number of independently generated synthetic variants provided no consistent improvement. Under controlled scarcity, however, synthetic completion improved mean R@1 by 3.69 percentage points at 25% genuine historical coverage and by 2.92 points at 50%, relative to the proportionally scaled real-only baselines. At 75%, the gain decreased to 2.00 points, while performance remained comparable to the complete-real-data reference. Fixed-schedule real-only controls did not reproduce these improvements. The results indicate that genuine and synthetic observations are complementary. Synthetic completion primarily benefits
retrieval by extending cross-domain identity coverage rather than by increasing training exposure, with its contribution gradually decreasing as genuine historical coverage increases.
Read the full abstract ▾
All authors Marcin Iwanowski, Adam Mazgaj, Ferdynand Gorski, Sabina Szymoniak
Computer Vision
Kairos: A Dataset for Fine-Grained Video-Language Modeling over Space, Time, and Dynamics
Ruibo Ming, Lei Sun, Deheng Zhang et al. (+8 more)
Many emerging
video language modeling tasks require systems to move beyond clip-level abstraction and model visual content as it unfolds over extended time horizons. However, most existing
video datasets rely on coarse or sparsely aligned supervision, which compresses temporal variation and limits the ability of models to learn reusable representations of continuous visual dynamics. We introduce Kairos, a
video dataset for
video-
language modeling with time-resolved annotations. Kairos consists of long-duration
videos, ranging from ten minutes to half an hour, annotated with fine-grained temporal
alignment. The annotations capture ongoing actions, entity appearances and attributes, interactions, and evolving contextual cues along the
video timeline. This time-resolved structure supports fine-grained evaluation, long-range modeling and
reasoning, instruction data construction, representation learning, and
video generation. Kairos provides a general-purpose foundation for modeling visual experiences over time.
Read the full abstract ▾
All authors Ruibo Ming, Lei Sun, Deheng Zhang, He Zhang, Jialu Li, Jian Wang, Zhendong Li, Mengshun Hu, Danda Pani Paudel, Luc Van Gool, Jinjin Gu
Computer Vision
CVT-GS: Learning to Simplify 3D Gaussian Splatting with Centroidal Voronoi Tessellation
Bingxian Li, Yilong Li, Jingliang Peng et al. (+6 more)
While 3D Gaussian Splatting (3DGS) has emerged as a powerful representation for real-time novel view synthesis, rendering high-fidelity scenes often relies on a massive number of Gaussian primitives, incurring substantial storage and computational overhead. Existing simplification techniques are largely intrusive, requiring training-time
pruning, architectural modifications, or computationally expensive per-scene fine-tuning. These drawbacks limit their deployment on off-the-shelf pretrained models. In this paper, we propose CVT-GS, a novel optimization-free post-hoc simplification framework that directly compresses trained 3DGS scenes without sacrificing visual fidelity. Our approach first constructs spatially coherent
cells over Gaussian centers via a geometry-aware Centroidal Voronoi Tessellation (CVT). Subsequently, a lightweight neural
cell merger predicts the geometry and appearance of a single, highly representative Gaussian primitive for each
cell under differentiable rendering supervision. By formulating simplification as a rendering-aware many-to-one merging process rather than naive primitive
pruning, CVT-GS outputs a standard 3DGS scene that is seamlessly compatible with existing renderers. Experiments on various
datasets demonstrate the superiority of our method. Notably, when achieving a 100-fold reduction in Gaussian points, our method operates 12 times faster than state-of-the-art methods while improving the PSNR by 1.3 dB.
Read the full abstract ▾
All authors Bingxian Li, Yilong Li, Jingliang Peng, Peng-Shuai Wang, Fei Zhu, Guozheng Li, Chi Harold Liu, Guoping Wang, Bo Pang
Computer Vision · Cross-listed from cs.GR
TBR: Transport-Based Rendering with Deposition Strokes for Inverse Graphics
Tianqi Liu, Yushan Han and Hang Liu
We present a stroke design in which strokes are transport-coupled: each stroke deposits material of its own area and moves every earlier mark without changing its area, so later strokes deform earlier ones. We then solve the inverse problem under this design: given a target image, we optimise an ordered program of such strokes whose replay approximates it, with digital marbling as the motivating medium. The stroke is a capsule that continuously joins circular drops to drawn deposits; its transport is exactly area-preserving, with a closed-form inverse outside the deposit, and a variant with the same inverse differs from line-source potential flow by 8% of the mean displacement. A replay adjoint regenerates intermediate states instead of storing them and uses 8.7x less
memory than checkpointed automatic differentiation; a fused implementation fits a 2000-stroke program at 1024x1024 in about four minutes on one GPU. On five marbled sheets the recovered programs are level with a published stroke-based fitter as rasters, replay across a fourfold resolution range, and support edits in program order and palette space that stay valid under transport.
Read the full abstract ▾
Computer Vision
Enhancing Table Structure Recognition via Bounding Box Guidance
Lei Hu and Shuangping Huang
Table Structure Recognition (TSR) aims to extract the bounding boxes of
cells and table structure (e.g., HTML) from table images. Although current approaches have made significant progress, the latest image-to-sequence methods overlook the explicit utilization of the bounding box information when predicting HTML sequences, leading to error predictions in complex scenes. In this paper, we introduce a novel framework BGTR (Bounding Box-Guided Table Recognizer). To more effectively utilize bounding box information, we first predict the bounding boxes of
cells and then use this information to guide the generation of HTML sequences. While utilizing bounding box information can enhance the accuracy of HTML sequences, for natural scene tables, the data volume is too small to allow for sufficient training of bbox-guided HTML generation. In response, we adopt a progressive training method for natural scene tables and introduce SNSTab, a synthetically generated natural scene table
dataset. Our experiments on five
benchmark datasets demonstrate SOTA performance.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.MA
MorphoOrgaAgent: A Foundation-Model-Based Multi-Agent System for Autonomous Organoid Analysis
Hanyi Zhang, Maximilian Hoermann, Lion J. Gleiter et al. (+5 more)
Organoids are three-dimensional tissue models whose morphology provides important insights into tumor development, disease progression, and drug testing. Extracting these morphological features relies heavily on manual
segmentation, which is time-consuming and labor-intensive. Furthermore, performing quantitative statistical analysis typically requires custom coding skills and a mathematical background, presenting a major barrier for experimental biologists. To address these challenges, we introduce MorphoOrgaAgent, a multi-
agent framework that achieves zero-shot organoid
segmentation, automated data analysis, and report generation based on natural
language input. The framework consists mainly of three core components: a TaskUnderstandingAgent that identifies requested measurements and visualization types; a hybrid
segmentation module that combines Cellpose-derived
geometric prompts with text
prompts to guide SAM3 for zero-shot organoid instance
segmentation; and a ReportAgent that computes quantitative metrics and compiles them alongside generated visualizations into a structured report. We further introduce MorphoOrgaVQA, a
benchmark designed for quantitative evaluation of
agent systems in organoid morphology analysis. Experimental results demonstrate that MorphoOrgaAgent handles both explicit and descriptive user requests, produces measurements closely matching ground truth, and generates complete analysis reports without requiring manual programming. The complete source code and MorphoOrgaVQA
benchmark are publicly available at https://github.com/peng-lab/MorphoOrgaAgent.
Read the full abstract ▾
All authors Hanyi Zhang, Maximilian Hoermann, Lion J. Gleiter, Yiling Xu, Bettina Katalin Budai, Hans-Ulrich Kauczor, Carsten Marr, Tingying Peng
Computer Vision
CausalChapter: Improving Long-Video Chaptering with Interventional Dependency Modeling
Xinran Duan, Guozhang Li, Yaoyao Zhong et al. (+3 more)
Long-form instructional
videos require automatic chaptering to support browsing,
navigation, and knowledge access. Recent long-context
language models can perform chaptering from textualized
video inputs, but they remain costly and brittle for content-dense lecture
videos with long transcripts, smooth topic transitions, and detailed chapter outputs. A scalable segment-then-caption paradigm reduces this cost, but introduces two new challenges: boundary error propagation and fragmented cross-chapter context. We propose CausalChapter, an intervention-inspired framework for long-
video chaptering that estimates prediction-level influence through lightweight masking and removal interventions. For boundary localization, our Local Dependency Shift module detects drops in predictive dependency between adjacent temporal windows; for chapter description generation, our Cross-Segment Support Selection module reranks historical contexts according to their support for the current prediction. Experiments on long-
video chaptering
benchmarks show that CausalChapter improves boundary localization, chapter description quality, and cross-chapter coherence.
Read the full abstract ▾
All authors Xinran Duan, Guozhang Li, Yaoyao Zhong, Mei Wang, Lizhi Wang, Hua Huang
Computer Vision
CoordFormer: Give Me Any Coordinates and I Will Give You Labels
Iacopo Curti, Pierluigi Zama Ramirez, Alioscia Petrelli et al. (+1 more)
Semantic
segmentation on very-high-resolution images remains challenging due to the high computational cost and the difficulty of capturing fine-grained details. We propose CoordFormer, a novel coordinate-based architecture for semantic
segmentation that predicts labels at arbitrary spatial locations through a Coordinate Decoder equipped with a Localized Cross-
Attention mechanism. The decoder combines coordinate
embeddings with high-resolution local patch features and interacts with global
tokens extracted from a downsampled image processed by a ViT foundation encoder, enabling rich semantic context while preserving pixel-level precision. This design enables flexible inference at arbitrary resolutions while keeping
memory low on very-high-resolution inputs, and supports an efficient semantic-edge-focused strategy that concentrates computation along boundaries, maintaining fine-grained accuracy while reducing latency and computational cost. CoordFormer achieves state-of-the-art performance on MaSS13K and outperforms comparably sized and higher-parameter methods on DIS5K and KPIs, demonstrating its effectiveness for high-quality, very-high-resolution semantic
segmentation.
Read the full abstract ▾
All authors Iacopo Curti, Pierluigi Zama Ramirez, Alioscia Petrelli, Luigi Di Stefano
Computer Vision
TriCCOT: Tri-part Convolutional Conformal Transformer for Onboard Space Object Detection
Adrien Dorise, Marjorie Bellizzi, Julia Cohen et al. (+1 more)
Onboard object detection in Earth observation is constrained by limited computational resources and the absence of fully corrected imagery. While convolutional detectors are hardware-efficient, they often struggle to extract robust representations from raw and noisy data. Conversely,
transformer-based models provide stronger global
reasoning capabilities but remain difficult to deploy on FPGA accelerators due to quadratic
attention complexity and non-compatible operations. We introduce TriCCOT, a tri-part architecture for robust and deployable onboard object detection. TriCCOT combines a convolutional region proposal network, a conformal prediction stage, and Aper-GATES, our hardware-friendly
attention-based classifier. The region proposal network generates candidate bounding boxes, which are subsequently enlarged via conformal prediction, providing a distribution-free probabilistic coverage guarantee. The resulting crops are processed by Aper-GATES, which reformulates self-
attention through convolutional projections, global channel statistics, and hardware-friendly gating operations, avoiding standard
transformer operations that are poorly suited to CNN-oriented accelerators. Experiments on the DIOR and VDVRaw
datasets demonstrate competitive detection performance and improved robustness to spatial blur and signal-dependent noise when compared to FPGA-compatible architectures. Finally, we report full deployment on a Xilinx Versal VCK190 FPGA without modifying the underlying DPU architecture, enabling unified CNN-
Transformer inference for spaceborne embedded applications.
Read the full abstract ▾
All authors Adrien Dorise, Marjorie Bellizzi, Julia Cohen, Stéphane May
Computer Vision
Charts Are Beyond Pixels: Probing for Layer-Wise Chart Understanding and Editing
Xiaochuan Zhong, Yifan Hou, Chenxi Pang et al. (+1 more)
Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This structural view motivates evaluating whether models can understand and manipulate charts at the layer level. Existing chart
benchmarks, however, primarily assess the correctness or fidelity of final outputs and do not directly evaluate these layer-wise behaviors. We present LayerWiseBench, a
benchmark organized around three core concepts, layer attribution, layer binding, and visibility ordering, that structure its chart-understanding and chart-editing evaluations. Generated from executable chart programs, LayerWiseBench pairs each rendered chart with spatially aligned per-layer RGBA assets and construction-derived labels for functional roles, semantic bindings, and visibility relations. From this layer-wise representation, we derive controlled understanding questions, editing targets, reference images, and evaluation regions. It contains 2,800 source charts across 14 chart paradigms, from which we derive 7,329 layer-wise understanding questions and 53,791 instruction-guided editing variants. Among the evaluated VLMs, Qwen3.5-27B, which achieves the highest QA macro-average, obtains 93.04% accuracy on layer attribution and 97.46% on layer binding, but only 61.46% on visibility ordering. Across the four evaluated image editors, overall mIoU ranges from 1.49% to 4.93%, and visibility-constrained edits have the lowest mIoU for every editor, ranging from 0.37% to 2.00%. Taken together, these results identify tasks involving front-to-back relations between overlapping components as a recurring challenge across understanding and editing, motivating more explicit modeling of component identity and visibility relations.
Read the full abstract ▾
All authors Xiaochuan Zhong, Yifan Hou, Chenxi Pang, Shaobo Cui
Computer Vision
Qiaohui Chu, Haoyu Zhang, Meng Liu et al. (+3 more)
Egocentric 4D interaction
forecasting aims to anticipate both where future interactions will occur in 3D and how the human body will move to realize them, providing an important capability for assistive robotics and human-computer interaction. Existing methods struggle to translate semantic understanding into precise continuous 3D localization and to balance motion diversity with structural consistency in pose
forecasting. More fundamentally, these tasks are often modeled separately, leaving the continuous
geometric and temporal correspondence between interaction locations and body motion insufficiently captured. To address these challenges, we introduce Coherent4D, a large-scale
egocentric dataset for continuous 4D interaction
forecasting, comprising approximately 233K samples across three domains. Each sample pairs a sequence of future 3D interaction locations with corresponding full-body poses, aligned in time and expressed in a shared coordinate system. We also provide evaluation metrics in continuous space. Building on this formulation, we propose HIGFlow, a Hand Interaction Guided Residual Flow framework that models
forecasting as a cascaded where-to-how process. HIGFlow first forecasts continuous future interaction locations by combining semantic grounding with short-horizon visual dynamics, and then uses the predicted location sequence to condition a deterministic motion anchor and residual Flow Matching for diverse yet structurally consistent full-body motion
forecasting. Extensive experiments across all three domains demonstrate consistent improvements over representative baselines on both location and pose
forecasting, while ablations validate the contributions of the proposed components. The project page is available at https://corrineqiu.github.io/from-where-to-how/.
Read the full abstract ▾
All authors Qiaohui Chu, Haoyu Zhang, Meng Liu, Haoxiang Shi, Dongmei Jiang, Liqiang Nie
Computer Vision
SynthRCT: Scalable Conditional Deformation Synthesis for Synthetic Repeat CT Generation
Tomas Guija-Valiente, Blanca Rodriguez-Gonzalez and Norberto Malpica
In
proton therapy, plans are typically optimized on a single planning CT, making robustness evaluation essential under anatomical changes. However, current scenarios often rely on simplified perturbations that poorly capture complex, patient-specific variability. We propose SynthRCT, a scalable conditional generative framework for 3D anatomical deformation synthesis. Based on a conditional variational autoencoder, SynthRCT learns a latent deformation space and decodes sampled latent codes into local stationary velocity fields conditioned on an input anatomy. Local fields are assembled into coherent full-volume transformations, enabling
memory-scalable generation for large field-of-view CT data. We validate the approach on respiratory 4DCT data with multiple breathing-phase anatomies per subject. SynthRCT enables patient-specific sampling of plausible anatomical transformations beyond predefined robustness scenarios. Code available at: https://github.com/TomasGuija/SynthRCT.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
MFVINS: Multiple Fisheye Camera-Based Visual Inertial System
Eunseong Jang, YuJin Chung, Sang Jun Lee et al. (+2 more)
A simultaneous localization and mapping (SLAM) method using a
monocular camera and a low-cost inertial measurement unit (IMU) sensor is an effective way to fulfill a low-cost sensor configuration. Using this sensor configuration, visual-inertial system (VINS) focuses on fusing data from a camera and an IMU sensor to estimate the six degrees-of-freedom (DOF) of the sensor pose. Typically, VINS uses only a single camera as visual input, which lead to problems such as error accumulation due to occlusion, various illumination, and textureless environments. In this paper, we propose a new multiple fisheye camera-based visual-inertial system called MFVINS. We present an IMU-aided FAST feature tracker for multiple cameras that enables efficient extraction and robust matching of local features. Then, the proposed method filters out outliers caused by fisheye distortion on the normalized image plane. Subsequently, a new reprojection error with physical validity constraints is proposed for bundle adjustment using learning-based depth estimation. The proposed method is applied to various scenarios, and its effectiveness is demonstrated by comparing previous VINS methods. In particular, MFVINS is implemented in real-time process to leverage the advantages of using multiple cameras – robustness against occlusion and textureless regions – while reducing the computational burden.
Read the full abstract ▾
All authors Eunseong Jang, YuJin Chung, Sang Jun Lee, Jihyun Yoon, HyungGi Jo
Computer Vision
GOLF: Global Observation with Local Focus for Calibration-Aware Stereo Interaction Field Estimation
Minqiang Zou, Riqiang Jin, Zhi Lv et al. (+7 more)
We present GOLF, the first-place solution to the SHOW3D Interaction Field Estimation Challenge at HANDS@ECCV 2026. Given synchronized
egocentric stereo views, the task is to predict a 3D vector from each of 21 hand joints to the closest point on the manipulated object. GOLF combines dense global context, locally sampled hand/object evidence, and common-frame Plücker-ray geometry. We adapt DINOv3 ViT-H+/16 with LoRA and trainable LayerNorm parameters, then jointly decode both interaction fields. Our primary model achieves an official score of 27.61 and a mean ADE of 27.96 mm on the hidden test set. An equal-weight ensemble with a complementary directly fine-tuned variant improves these results to an official score of 27.47 and a mean ADE of 27.82 mm, securing first place.
Read the full abstract ▾
All authors Minqiang Zou, Riqiang Jin, Zhi Lv, Dong Luo, Lianghai Tian, Zhenyu Zhao, Qi Xu, Tong Wu, Mochen Yu, Yao Tang
Computer Vision
Effects of model architecture and learning strategies on deep learning-based recognition of activated sludge microscopic images and comparison with quantitative image analysis
Suguru Hakoshima, Tomohiro Tobino and Fumiyuki Nakajima
Microscopic image analysis has long been recognized as a promising approach for monitoring activated sludge. In recent years, deep learning-based image analysis has been increasingly adopted in this field because of its high performance. However, previous studies on microscopic image analysis of activated sludge have rarely explored
transformer-based models or self-supervised foundation models and have instead relied on CNNs and supervised ImageNet pretraining. In addition, previous studies often downsampled image sizes, but the effects of downsampling have not been sufficiently investigated, and the relationship between downsampling strategies and image analysis performance remains unclear. Furthermore, no study has quantitatively compared deep learning performance with quantitative image analysis (QIA), which was widely used before the emergence of deep learning. In this study, to examine how model architecture and learning strategies affect performance in microscopic image analysis of activated sludge and to quantitatively determine whether deep learning outperforms QIA, we prepared three types of activated sludge samples, classified their microscopic images, and evaluated classification accuracy. Our results showed that
transformer-based architectures and alternative pretraining methods were effective in terms of classification accuracy. Our downsampling analysis showed that using overly small images reduced accuracy, but increasing image size beyond a certain point did not improve it further. In addition, the analysis indicated that, to achieve high classification accuracy, maintaining the field of view was a more effective downsampling strategy than maintaining resolution. Finally, our comparison between deep learning and QIA showed that deep learning outperformed QIA in terms of accuracy.
Read the full abstract ▾