Artificial Intelligence
Context-dependent agent evaluation with orthogonal equilibrium learning
Haorui Ma, Zehua Zang, Jiangmeng Li et al. (+3 more)
Many applications require to evaluate
agents under contextual information (e.g., a
prompt, task, or user group). We study how to perform such context-dependent
agent evaluation from offline feedback. Existing score-based models for this purpose (e.g., Bradley-Terry) impose a transitive preference ordering, which fails to reflect collective preferences when human judgements are heterogeneous. Inspired by social choice theory, we frame evaluation as a contextual game between two players, each selecting a distribution over
agents as the strategy to receive greater collective preference than the other. Then, the support of the Nash
equilibrium defines a context-specific set of winners. However, learning context-specific
equilibria from offline logs is difficult because each context reveals human feedback on only a subset of
agents, and, hence, a naive plug-in
estimator can therefore be biased. To address these challenges, we propose NashEval, a general framework for robust contextual
equilibrium learning. NashEval first constructs debiased estimates of the contextual payoff
matrix that characterizes the game. NashEval then learns the context-to-
equilibrium mapping with a tailored orthogonal loss, which avoids the need to solve a separate game for each context. We show theoretically that errors in estimating the nuisance functions underlying the payoff
matrix affect the risk of the learned
equilibrium (i.e., exploitability) only through higher-order terms. Across various experiments, NashEval improves robustness of
equilibrium learning and consistently identifies the set of top-performing
agents across contexts.
Read the full abstract ▾
All authors Haorui Ma, Zehua Zang, Jiangmeng Li, Yi Li, Fanjing Xu, Stefan Feuerriegel
Artificial Intelligence · Cross-listed from cs.CV
Auditing Quality Filters for Long-Tail Human Data Curation
Rishav Agarwal, Nirshal Chandra Sekar and Anirudh Vemula
Robots on construction sites must detect workers who are kneeling or bending, which we call low poses. These workers can be lost from training
datasets during automatic labeling. We study a pipeline that detects people, estimates their body joints using NLF, and groups similar poses. Low poses account for only about 2 percent of the retained examples. This low share may partly reflect the pipeline's quality filter, which rejects examples with low detection confidence or uncertain joint estimates. We examine this filtering using four alternative pose clues: bounding-box shape, vertical body span, pose grouping aligned to the scene's vertical direction, and image appearance. All four suggest that low poses are rejected by the filter more often. Separately, controlled simulated scenes show that a person detector fine-tuned on a public construction
dataset misses more workers in these poses even when we correct their bounding box height is matched to that of standing workers. These findings suggest that low poses are scarce and hard to find, and we cannot rely on bounding boxes or poses for long-tail human data curation.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
CyberWorld: World Models for Sample-Efficient Autonomous Cyber Defense
Ryozo Masukawa, Sanggeon Yun, Raheeb Hassan et al. (+3 more)
Deep reinforcement learning has become a prominent approach to autonomous cyber defense. Existing methods are predominantly model-free and consequently require extensive environment interaction. World models provide an alternative by learning predictive dynamics and optimizing policies through imagined trajectories, yielding substantial gains in sample efficiency in robotics and
embodied control. Extending this paradigm to cybersecurity raises a fundamental question: what should constitute the "world" in a cyber world model? We introduce CyberWorld, a Dreamer-style world modeling framework that learns latent cyber dynamics from vector, graph, textual, and
multimodal representations of the defended network. Across all four scoreable CyberWheel attack strategies, the graph-based CyberWorld variant exceeds a strategy-agnostic control after 3.6k-15.8k environment steps, compared with millions of steps required by model-free PPO. Across representation choices, graph structure provides greater robustness under
topology-dependent attacks, while simpler representations remain competitive in overall performance. Among successful runs, the number of episodes required to reach the control remains approximately constant as network size increases from 15 to 100 hosts. These results establish learned cyber dynamics as a sample-efficient and scalable basis for autonomous defense, and identify world representation as a central design axis for robustness and scalability.
Read the full abstract ▾
All authors Ryozo Masukawa, Sanggeon Yun, Raheeb Hassan, Hyunwoo Oh, SungHeon Jeong, Mohsen Imani
Artificial Intelligence · Cross-listed from cs.SD
NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speech
Qiaolin Wang, Pedro Sandoval-Segura, Anunaya Joshi et al. (+2 more)
While modern text-to-
speech (TTS) systems generate highly natural
speech and support inline non-verbal vocalization (NVV) tags, accurate control over these events remains challenging. A key gap is the lack of established post-training methods for non-verbal control in continuous autoregressive flow-matching TTS. To this end, we present NVAlign, a direct-gradient post-training framework for NVV tag-following in this architecture. We first perform supervised fine-tuning (SFT) of TTS models and an NVV-aware automatic
speech recognition (NV-ASR) model on NVV-annotated
speech, then freeze the NV-ASR model to serve as the reward model for post-training. A two-step gradient surrogate enables efficient reward backpropagation through the flow-matching sampler to jointly update the autoregressive backbone and acoustic flow head. Fidelity penalties and reference-velocity regularization help preserve speaker similarity and
speech quality. Results from NVV-SuperBench and human listening evaluations show that NVAlign improves tag-following accuracy over SFT and Flow-GRPO baselines. These findings demonstrate that direct reward-gradient optimization can improve non-verbal control in continuous autoregressive flow-matching TTS. Audio samples are available at https://nvalign.github.io/.
Read the full abstract ▾
All authors Qiaolin Wang, Pedro Sandoval-Segura, Anunaya Joshi, Edvardas Jurkonis, Jake Downie
Artificial Intelligence · Cross-listed from cs.LG
FARE: Deep Reinforcement Learning For Fair Exposure Constrained Uncertainty Aware Financial Content Personalization
Arundeep Chinta, Lucas Vinh Tran and Jay Katukuri
Content personalization systems in financial services must ensure fair exposure across diverse offerings-a requirement driven by contractual obligations and the need to prevent "rich-get-richer" dynamics where content with high click-through rate (CTR) dominates while other relevant products receive minimal visibility. Share of Voice (SOV) constraints, which guarantee each content category a target fraction of top-position exposure, address this by promoting product diversity and balanced user discovery. While re-ranking layers atop CTR models are common in practice, we propose two key novelties: (1) framing SOV-constrained ranking as a deep reinforcement learning problem analogous to constrained trade execution in algorithmic finance, and (2) explicitly incorporating CTR prediction uncertainty into the
agent's state space and policy design-enabling larger ranking adjustments for high-uncertainty predictions where deviation from CTR-optimal ordering is less costly. We introduce FARE (Fair Ranking Executor), a modular uncertainty-aware execution layer that translates any black-box CTR model's predictions into SOV-fair rankings without retraining the underlying model. Our uncertainty-weighted proportional control policy (FARE-PC) and learned neural policies (FARE-ES, FARE-PPO) demonstrate that uncertainty-aware approaches can substantially reduce SOV deviation from
fairness targets while minimizing engagement loss, with gradient-free
evolution strategies outperforming policy gradient methods on synthetic data and the ordering reversing on KuaiRand-Pure.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
DOHF: Online Diffusion Fine-tuning with Doob's $h$-transform Guidance
Zhengyi Guo, Jiayuan Sheng, Wenpin Tang et al. (+1 more)
Reward-based
diffusion fine-tuning faces practical challenges when desirable outcomes are rare or conditioning corrections are costly to estimate. In this work, we propose
Diffusion Online $h$-guidance Fine-tuning (DOHF), which turns Doob's $h$-transform into a practical online training
algorithm. DOHF assigns optimality weights to generated samples, estimates the normalized local correction $\nabla\log h$ under the current rollout policy, and distills it directly into the generative model. Theoretically, we characterize the population-optimal DiffusionNFT update as well as the various classfier free guidance methods through a unified $h$-transform perspective. Methodologically, our framework accommodates black-box and non-differentiable rewards without additional network evaluations. We further show improved
alignments under three empirical scenarios. Our work demonstrates how adapting probabilistic conditioning through inexpensive estimation and iterative
distillation can improve generative learning across statistical sampling and visual generation.
Read the full abstract ▾
All authors Zhengyi Guo, Jiayuan Sheng, Wenpin Tang, David D. Yao
Artificial Intelligence · Cross-listed from cs.CR
A Large-Scale Benchmark and Risk Assessment of Traffic Analysis Attacks on Cloud LLM Services
Shahrooz Pouryousef, Jesus Lopez, Saeefa Rubaiyat Nowmi et al. (+4 more)
Cloud-based Large
language model (LLM) services create a network-level traffic side channel that can expose model,
prompt, and task behavior despite encryption. From packet sizes, directions, timing, and burst structure alone, a passive local observer can infer the serving model, the user's
prompt category, and the task executed by a collaborative multi-
agent system. Yet current evidence is fragmented across separate
datasets and settings, limiting reproducibility and comparison. We present, to our knowledge, the first unified measurement study and public
benchmark of encrypted LLM traffic across both user–LLM and multi-
agent executions. The large-scale
benchmark contains 60,000 user–LLM interactions across 10 models and 6
prompt categories, plus 2,838 multi-
agent executions covering 10 task categories and two coordination
topologies. Using only encrypted packet metadata, we assess the risk of traffic analysis attack by characterizing traffic signatures, identifying the features most associated with leakage, and testing robustness under
prompt reformulation, decoding-temperature changes, larger candidate model sets, and partial traffic observation. Model fingerprinting achieves 97.7% balanced accuracy,
prompt-category fingerprinting reaches 76.7% mean accuracy, and multi-
agent task fingerprinting achieves up to 90.7% accuracy.
Prompt reformulation weakens but does not remove model-specific leakage, and task fingerprints remain detectable even from a single
agent's traffic.
Read the full abstract ▾
All authors Shahrooz Pouryousef, Jesus Lopez, Saeefa Rubaiyat Nowmi, Md Mahmuduzzaman Kamol, Moinul Hossain, Muoi Tran, Mohammad Saidur Rahman
Artificial Intelligence
COUNTERMEM: World-Model Verified Counter-Factual Memory for Language Agents
Hongji Pu, Ruixiang Tang and Yongfeng Zhang
Existing
agent memory frameworks mainly create
memory through an
agent's interaction with the factual world, e.g., remembering feedback from actions taken to improve performance on future tasks. However, these frameworks seldom ask the "what if" question during
memory construction: what if a different action had been taken, would the feedback have changed, and how could this feedback become useful
memory? Obtaining such feedback directly in an active environment can be expensive and can alter the state needed for comparison. In this work, we introduce COUNTERMEM, a reinforcement-learning framework for constructing and using verified counterfactual
memory across tasks. After a failed action, COUNTERMEM evaluates local alternatives from a copy or reset of the original state using executable world models, such as tests, proof checkers, and solvers. It stores improvements with the original and corrected actions, checked outcomes, and conditions for reuse. A learned
memory-use policy selects a retrieved record or skips
memory to balance task success and interaction cost, while the base LLM remains fixed. Both
memory and policy are frozen during held-out evaluation. We evaluate COUNTERMEM on 12
benchmark settings across six domains. With gpt-oss-120b, COUNTERMEM improves both ReAct and Reflexion on all 12
benchmarks across six domains, averaging a gain of 12.6 percentage points over their unaugmented versions. In the four-domain comparison across two backbones, task-run
tokens decrease by 7.7-42.0%, excluding offline selector-training costs. Further analyses show that removing verification or persistent storage weakens the gains, while applying verified corrections to unsuitable decisions can reverse them. Code will be released upon acceptance.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CV
CueKFS: Agentic Cue-Driven Keyframe Selection for Long Video Understanding
Weitai Kang, Hanieh Deilamsalehy, Yumo Xu et al. (+3 more)
Keyframe selection (KFS) has long produced compact
video summaries for browsing and
retrieval, and representative frames for thumbnails. More recently, when conditioned on a question, KFS provides an alternative to uniform sampling for long-
video question answering by selecting frames that are more relevant to the question. Most methods rank frames by similarity to the question. Yet a relevant frame may score poorly when the question combines subjects or moments that no single frame shows, or requires implicit information absent from its wording. Other methods try to break down the question into subqueries, but suffer from inaccurate decomposition due to their static initial context. Therefore, we propose CueKFS, a training-free method that reformulates question–frame matching as comparing frames against a set of dynamically generated visual cues. From an initial set of salient frames, we decompose the question into cues. Each cue concurrently probes the
video to navigate to its own evidence. A
reasoning VLM then agentically revises the cue set against its evidence to re-explore the
video. CueKFS then allocates the budget across the surviving cues. Across three
benchmarks, CueKFS establishes state-of-the-art results in all 27 evaluated settings with available prior results, achieving budget-averaged gains of up to +4.54% over the previous baseline and a median of only two VLM calls. We further provide a detailed behavioral analysis of CueKFS, showing that agentic cue refinement drives active re-exploration of the
video, yielding relative similarity gains of up to 92% over the initial context.
Read the full abstract ▾
All authors Weitai Kang, Hanieh Deilamsalehy, Yumo Xu, Dewang Sultania, Serdar Cellat, Yan Yan
Artificial Intelligence
IndustryLLM: Failure-Driven LLM Training for Industrial Procurement
Liang Ding, Zhiang Xu, Yuyang Sheng et al. (+13 more)
Industrial procurement requires
language models to bridge informal buyer jargon, sparse marketplace attributes, and authoritative engineering standards under strict safety tolerances. We present IndustryLLM, an open-weight industrial
language model trained from Qwen3.5-35B-A3B-Base (35B total parameters with ~3B activated per
token, with the
vision encoder frozen). Rather than relying on generic text scaling, we introduce a failure-driven adaptation recipe spanning continued pre-training (CPT) and supervised fine-tuning (SFT). CPT leverages a curated ~100B-
token corpus integrating 5B
tokens of national standards (e.g., GB/T) and technical archives, 10B
tokens of de-identified real-world industrial transaction and inquiry records, and 60B
tokens of general replay. To overcome register mismatch and factual brittleness, we systematically reconstruct an estimated 20B-
token domain subset via multi-register rewriting across 10 genres and 8 writing styles, confidence-routed minimal factual editing, and error-targeted QA synthesis (resolving colloquial typos like '42-luo-mu' -> 42CrMo, expanding ambiguous codes like '16674' -> GB/T 16674, and clarifying conflicting dimensional specs). For downstream deployment, we formalize an evidence-gated constraint-evaluation interface enforcing three-valued logic where unverified product evidence remains unknown rather than satisfied. Offline evaluations demonstrate consistent gains on procurement-query structuring (+2.97 percentage points in exact match, 95% CI [2.11, 3.86] in No-Think mode), while randomized online A/B experiments in production yield substantial improvements (+4.25% GMV, +8.3% satisfied inquiries) alongside a latency reduction from 6-7 s to 1.5 s. Model weights and configs are released at https://huggingface.co/alibaba-
multimodal-industrial-ai/IndustryLLM.
Read the full abstract ▾
All authors Liang Ding, Zhiang Xu, Yuyang Sheng, Bin Chen, Songlin Bai, Run Zhu, Dingjun Wu, Hui Xu, Yandi Wang, Fulin Shi, Leilei Gan, Linlin Yu, Qihuang Zhong, Keqin Peng, Yalong Li, Chengfu Huo
Artificial Intelligence · Cross-listed from cs.LG
Deep Reinforcement Learning for Equity Trading: Benchmarking Actor-Critic Methods with Forward Retraining
Bicheng Wang and Xinyi Zhang
Consistently profitable trading is difficult because equity
markets are noisy, non-stationary, and only partially predictable from historical data. We
benchmark five deep reinforcement learning (DRL) actor-critic methods: A2C, PPO, DDPG, TD3, and SAC, that learn trading actions end-to-end from
market states, and compare them with a supervised price-
forecasting baseline. Using daily data for 20 large-capitalization S&P 500 stocks from 2000 to 2020, enriched with trend-following technical indicators and log min-max scaling, we train on 2000-2018 and backtest on 2019-2020. Each
agent is evaluated both when trained once and under forward retraining, in which it is retrained on all data available before each successive test window. DDPG achieves the highest annual return (55.5%), Sharpe ratio (1.38), and alpha (0.22), but also the highest
market beta (1.24). TD3 and SAC offer a better risk-return balance, with Sharpe ratios of 1.37 and 1.33 and maximum drawdowns of about 25%. Forward retraining improves A2C, PPO, and SAC, leaves TD3 essentially unchanged, and reduces DDPG's annual return from 55.5% to 29.8%, consistent with TD3's greater robustness to hyperparameters. The
forecasting baseline has the smallest maximum drawdown (9.6%) and the lowest beta (0.31), underscoring a trade-off between the higher returns of end-to-end DRL and the lower risk of forecast-driven strategies.
Read the full abstract ▾
Artificial Intelligence
Metro-WM: Long-Horizon Latent Planning with Realisable Sub-Goals
Royson Lee, Fady Rezk, Titouan Parcollet et al. (+2 more)
Model-predictive control with Joint-
Embedding Predictive Architectures (JEPAs) provides a strong zero-shot goal-reaching planner, but it is only effective over short planning horizons. Hierarchical extensions attempt to bridge this gap by learning a macro planner to predict intermediate latent sub-goals to guide the micro planner. In this work, we demonstrate that unconstrained latent sub-goal prediction is fundamentally flawed. A rigorous evaluation reveals that a leading state-of-the-art macro planner routinely emits physically unrealisable sub-goals. To resolve this, we introduce Metro-WM, a hierarchical framework that issues sub-goals by retrieving genuine states from prior experience rather than generating ungrounded latent vectors. Specifically, Metro-WM constructs a graph whose vertices are observed frames from offline expert demonstrations or random-action trajectories, allowing frames from different episodes to be connected and stitched into routes to the goal. Planning over the full graph also makes the system highly robust to execution errors: if the micro planner drifts off course, Metro-WM instantly finds a new optimal path from the current state. Our experiments show that Metro-WM achieves superior long-horizon success rates of up to 37.33 percentage points over the next best hierarchical approach while being up to 10.9 times faster, requiring both 13-56 times less offline compute and fewer tuned hyperparameters. Additional analysis reveals that Metro-WM finds shorter paths than the offline demonstrations, outperforms an oracle relying on the query's own demonstration, and maintains robust performance under extremely sparse
dataset conditions.
Read the full abstract ▾
All authors Royson Lee, Fady Rezk, Titouan Parcollet, Timothy Hospedales, Cristina Cornelio
Artificial Intelligence · Cross-listed from cs.NI
AirLog: Store-Level Indoor Life Logging Made Easy
Zihui Yun, Jiaying Du, Yue Yu et al. (+4 more)
This paper presents AirLog, a smartphone-based life journaling system that automatically reconstructs users' store visits in shopping malls and summarizes them into human-readable journals. Unlike conventional indoor localization systems, AirLog avoids labor-intensive
radio-map construction and dedicated wireless localization infrastructure and
algorithm calibrations. Instead, it repurposes two cues already available in commercial spaces: semantic information exposed by ambient Wi-Fi SSIDs and indoor directory images. AirLog converts directory images into spatial maps and fuses Wi-Fi semantic anchors with inertial dead reckoning to recover store-level trajectories, which are then summarized into journals by an LLM. Such store-level life logs can support applications such as personal
memory recall, activity reflection, and automated diary generation without requiring users to manually record where they have been. We implement AirLog on commodity smartphones and evaluate it on both a large-scale public
dataset and a self-collected
dataset. The results demonstrate that AirLog substantially improves store-level region recovery, semantic matching, trajectory reconstruction, and journal quality over existing baselines. A human evaluation further shows that the generated journals are coherent and faithful to users' visits.
Read the full abstract ▾
All authors Zihui Yun, Jiaying Du, Yue Yu, Zhewei Liu, Zhen Xiang, Longfei Shangguan, Zhenlin An
Artificial Intelligence
LLM Judge Validation Under Sparse Overlap: From Inference to Design
Junxuan Li, Arko Mukherjee and Soumyabrata Pal
Validating an LLM-as-a-judge requires estimating its agreement with humans, yet annotation budgets rarely allow every item to be multiply labeled. We prove that this overlap sparsity is the first-order determinant of wrong deployment decisions: at 5% pairwise overlap, wrong-decision rates reach 25% and the probability of selecting the wrong best judge among ten candidates is 65%. The two actionable levers are overlap quantity and allocation. For quantity, we derive a minimum-overlap formula showing $\rho \geq 0.25$ suffices for non-borderline judges while borderline cases remain fundamentally hard. For allocation, a zero-cost stratified scheme halves false-rejection rates relative to random sampling when strata are informative. We validate on 10 LLM judges across four evaluation
matrices spanning visual assessment, causal
reasoning, and summarization.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.RO
PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning
Hang Yu, James Staley, Cheng Xi Tsou et al. (+8 more)
Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated
robot tasks, with additional exploration using a fine-tuned
vision-
language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.
Read the full abstract ▾
All authors Hang Yu, James Staley, Cheng Xi Tsou, Xiujin Liu, Wenchang Gao, Jindan Huang, Shijie Fang, Zhegong Shangguan, Angelo Cangelosi, Reuben Aronson, Elaine Short
Artificial Intelligence · Cross-listed from cs.CV
SynDORBench: Evaluating LVLM Perceptual Robustness Under Physically Constrained Visibility Conditions
Jeremy Stephen Gabriel Yee, Zhengkui Wang, Zhiyuan Zhang et al. (+5 more)
Large
vision-
language models (LVLMs) have demonstrated remarkable performance on
multimodal reasoning benchmarks, yet their perceptual reliability under physically constrained imaging conditions remains poorly understood. Existing evaluations predominantly assume ideal visual inputs and therefore fail to characterize how camera distance, illumination, viewpoint, and pixel density fundamentally affect semantic recoverability. We introduce SynDORBench, the first physically grounded
benchmark for evaluating LVLM perceptual robustness under DORI-calibrated conditions aligned with human visual capability standards. SynDORBench comprises over 54k question–answer pairs generated through a controllable synthetic pipeline that systematically varies viewing distance, lighting, camera geometry, and action pose according to physically interpretable pixel-density regimes. To support scalable low-visibility supervision, we further propose a discernibility annotation framework that propagates human perceptual labels using mask-conditioned statistical features and ensemble learning. We evaluate 16 open-source LVLMs, a commercial LVLM baseline, and YOLO11x across human-presence classification and action recognition tasks under progressively degraded visibility conditions. Our results reveal that perceptual failure in LVLMs is strongly governed by pixel density and physical imaging constraints rather than model scale alone. Surprisingly, several compact open-source LVLMs outperform larger commercial baselines and substantially exceed YOLO11x robustness under long-range and low-light conditions. SynDORBench establishes a new
benchmark paradigm for physically grounded
multimodal evaluation, enabling systematic analysis of LVLM reliability under real-world perceptual constraints and direct comparison against human visibility thresholds.
Read the full abstract ▾
All authors Jeremy Stephen Gabriel Yee, Zhengkui Wang, Zhiyuan Zhang, Avinash Anand, Timothy Liu, Benedict Chan, Aik Beng Ng, Simon See
Artificial Intelligence · Cross-listed from cs.CV
A Surgical Foundation Model Reveals Task-Dependent Label Efficiency
Florian Philipp Stilz, Lorenzo Arboit, Vinkle Srivastav et al. (+6 more)
Developing label-efficient models is a central challenge in
surgical AI due to the high cost and scarcity of expert annotation. While self-supervised foundation models adapt well to new tasks with minimal data, how label efficiency varies across different
surgical tasks remains largely unexplored. Here, we introduce SURGE, a
surgical foundation model trained on SurgSpectrum-30M+, the largest pretraining
dataset comprising over 30 million frames, with checkpoints released to enable further research. We systematically evaluate label efficiency across 5 task categories and 15
benchmarks. These range from temporal and spatial scene understanding to fine-grained
reasoning tied to instrument-anatomy interactions and safety-critical maneuvers. SURGE outperforms prior state-of-the-art on all
benchmarks, even surpassing task-specific models on complex
reasoning tasks. Crucially, we reveal a task-dependent scaling behavior: while scene understanding tasks saturate with minimal supervision, fine-grained
reasoning tasks continue improving with substantially larger annotation budgets, providing a blueprint for allocating expert effort in complex domains. Code: https://github.com/CAMMA-public/SURGE
Read the full abstract ▾
All authors Florian Philipp Stilz, Lorenzo Arboit, Vinkle Srivastav, CAMMA International Surgical Partners, Jacques Marescaux, Sergio Alfieri, Pietro Mascagni, Nassir Navab, Nicolas Padoy
Artificial Intelligence
DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution
Chengkai Xu, Jiaqi Liu, Yicheng Guo et al. (+2 more)
Evaluating VLM-based autonomous driving remains difficult because driving competence is composite, where a capable system must ground traffic participants and hazards, integrate context across views and time, reason about future
evolution, and act appropriately under closed-loop interaction. Existing
benchmarks usually assess either open-loop understanding or closed-loop driving but provide limited structure for explaining how these abilities are organized, how they relate, and how they may inform model diagnosis and improvement. We present DriveHierarchy, a hierarchical
benchmark that organizes VLM-based autonomous driving into four ranks, spanning perceptual grounding, contextual
memory, mental
reasoning, and closed-loop execution. To instantiate this hierarchy, we integrate multiple open-source autonomous-driving
datasets into a unified open-loop
benchmark with 76,798 question-answer pairs over 84,279 frames and develop a closed-loop simulation platform with interactive scenario construction on a real-world road network, from which 100 driving scenarios are curated for
embodied evaluation. Experiments on 15 VLMs show that DriveHierarchy captures structured but non-redundant capability variation, relates open-loop understanding to closed-loop driving, and provides a practical basis for diagnosis and
benchmark-guided optimization. DriveHierarchy therefore serves as a unified framework for evaluating and improving VLM-based autonomous driving systems. An anonymized project has been released on https://github.com/PerfectXu88/DriveHierarchy
Read the full abstract ▾
All authors Chengkai Xu, Jiaqi Liu, Yicheng Guo, Peng Hang, Jian Sun
Artificial Intelligence · Cross-listed from cs.SD
Video-to-Music Generation for Gameplay Videos
Felipe Marra and Lucas N. Ferreira
Video-to-music models have advanced considerably in the last few years, particularly in film and music
video applications. In this paper, we investigate this problem in the
video game domain, which introduces new challenges for these models:
video frames are rendered graphics, music is mostly synthetic audio, and soundtracks loop across entire levels rather than following on-screen events. We introduce a new
dataset of 217.6 hours of Super Nintendo (SNES) gameplay
video paired with 485 hours of clean soundtracks, free of sound effects and voice-overs, matched to gameplay audio via audio fingerprinting. With this
dataset, we train a simple encoder-decoder
transformer that passes
video features directly to a MusicGen decoder, comparing different encoding strategies: textual descriptions (T5), independent frames (ViT), or spatiotemporal patches (ViViT). Each encoder is tested both frozen and fine-tuned, while the decoder is always fine-tuned. Frozen encoders match or outperform their fine-tuned counterparts on every metric, and the frozen ViViT achieves the best overall results. We compare this model with state-of-the-art baselines using both objective metrics and a listening study (N = 96). Despite having up to 18% fewer parameters, our model outperforms all baselines on objective metrics, surpasses GVMGen in the listening study, and performs comparably to OSSL.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?
Alizishaan Khatri
Activation probes are increasingly used to monitor LLMs in deployment. A probe is typically trained under one inference configuration, then used under whatever batch size and numerical precision the serving stack uses. Because common GPU kernels are not batch-invariant and floating-point formats round differently, the activations seen at deployment are not the ones the probe was trained on. We measure what that costs for Llama-3.1-8B, Qwen3-8B and Gemma-3-4B across batch sizes 4, 8 and 16 and float32, bfloat16 and float16, training 768 probes on one configuration, evaluating each on every other, and comparing verdicts example by example. Probes are stable, but aggregate accuracy is the wrong instrument for showing it: it understates how many verdicts change by a factor of two to nine. At the
prompt, accuracy never moves by more than 0.47 percentage points across 1,392 transfers and only 0.076% of verdicts change; under float32 with only the batch size varied, none of 201,960 verdicts change. During decoding the flip rate rises to 2.8%, but rows whose realised
tokens matched flip in only 0.12-0.15% of cases, while rows whose
tokens diverged flip in 12.9%: the cause is the text, not the arithmetic. A bfloat16 batch-size change flips the first generated
token for 2.1% of rows and leaves 25% on different
tokens by
token 20. Flips are symmetric, Cohen's kappa stays above 0.94, and AUROC moves by at most 0.05 points. Underneath, activations move about as much as the format's rounding: a bfloat16 batch-size change perturbs them by a median relative L2 of 1e-2, roughly 8x the float16 figure. Probes absorb this; the model's own next-
token argmax does not. Robustness evaluations of activation monitors should report per-example agreement rather than aggregate accuracy, separate representational noise from input change, and state the serving configuration.
Read the full abstract ▾
Artificial Intelligence
Working with AI: A Design Framework for Human-AI Collaboration
Yuqian Lu, Regina Lee, Rui Zhou et al. (+3 more)
Artificial Intelligence (AI), particularly GenAI, is becoming an increasingly important part of modern work. In industrial settings, AI can support decision-making, automate routine activities, assist humans, and improve productivity. However, successful AI adoption depends on more than what the technology can do. It also depends on how people experience and work with it. This raises an important question: how should human-AI collaboration be designed so that it works well for both people and organisations? This white paper addresses that question by presenting a practical framework for designing human-AI collaboration. The framework considers the human, the AI system, the task, the organisation, and the wider societal environment. It explains what effective collaboration looks like, what conditions influence it, what requirements should be met, and what design decisions organisations should consider. The report also includes a human-AI collaborative assembly system with cobot use case to demonstrate how the framework can be applied in practice. The use case shows how design requirements can be translated into specific collaboration features and evaluated through a case study. The aim of this white paper is to provide a clear and practical guide for designing human-AI collaboration that is effective, human-centred, and responsible.
Read the full abstract ▾
All authors Yuqian Lu, Regina Lee, Rui Zhou, Lixin Jiang, Andrew McDaid, Amy Lawrence
Artificial Intelligence
ConflictVLA-Bench: Benchmarking Behavioral Responses of Vision-Language-Action Models to Premise Conflicts
Liyu Hou, Yuan Wu and Yi Chang
While
Vision-
Language-Action (VLA) models perform strongly on manipulation tasks, their responses to invalid task premises remain underexplored. Existing evaluations of premise conflicts often focus on terminal task outcomes, yet task failure alone cannot distinguish behavioral disengagement from continued pursuit followed by an execution error. We call the latter pattern Failed Persistence. To study this phenomenon, we introduce ConflictVLA-Bench, which pairs conflict rollouts with premise-consistent reference rollouts and evaluates both outcomes and execution processes. Built on LIBERO, the
benchmark contains 2,826
prompt-conditioned conflict tasks spanning four conflict families, four structural configurations, and two
prompt conditions. Across all eight VLAs, invalid premises reduce original goal completion by at least 17.3 percentage points, with the reduction reaching 56.2 percentage points for OpenVLA. Crucially, even when models succeed on premise-consistent tasks and fail on their matched conflict tasks, they often continue to approach the original targets, retain early trajectory structure, and show limited action magnitude suppression. Failed Persistence therefore recurs across the evaluated models. Explicit premise checking does not consistently produce selective and coordinated behavioral changes. These findings show that terminal failure alone establishes neither behavioral disengagement nor refusal and that outcomes alone are insufficient for VLA evaluation. Experimental data and additional details are available on the project page: https://github.com/EmbodiedAISurvey/ConflictVLA-Bench
Read the full abstract ▾
Artificial Intelligence
CP-Agent: A Harness-Engineered Agent for Crystal Plasticity Simulation Workflows
Samuel Onimpa Alfred, Abhishek Kumar and Veera Sundararaghavan
Crystal plasticity (CP) simulations predict the mechanical behavior of polycrystalline metals, yet their routine use is hindered by the manual effort of configuring heterogeneous tools, orchestrating multi-step data pipelines, and calibrating constitutive parameters against experiments. These bottlenecks impede productivity in systematic parameter studies, motivating interest in automated workflows. This study presents CP-
Agent, a harness-engineered LLM-based
agent that autonomously executes complete CP modeling workflows from natural-
language tasks. Operating under the ReAct paradigm, the
agent reasons about tool selection and sequencing while delegating numerical search to established optimizers. The harness comprises a minimal system
prompt, typed tool definitions, a dispatcher, and a safety-bounded iteration loop, encoding domain knowledge through tool schemas rather than hard-coded logic. CP-
Agent is demonstrated on four case studies: calibrating four slip parameters of additively manufactured stainless steel 316L against tensile data; validating the workflow against published copper
benchmarks, reproducing stress-strain and texture
evolution; recovering the initial crystallographic texture of copper, where the
agent correctly identifies a diffuse initial texture; and reproducing the multi-pass rolling texture
evolution of a Mg-Zn-Ca
alloy, where the
agent chains five deformation passes and recovers the experimentally observed weakened, split basal texture. In all cases, the
agent inferred the correct execution sequence from the task statement, robustly across repeated runs, and delivered physically interpretable results. This work establishes harness engineering as a systematic approach to automating CP modeling workflows while maintaining physical interpretability and auditability through visible
reasoning traces.
Read the full abstract ▾
Artificial Intelligence
Learning to Stop without Learning to Stop: Self-Supervised Confidence Training Improves Reasoning Efficiency
Parsa Hosseini, Akasha Tigalappanavara, Sumit Nawathe et al. (+6 more)
Reasoning models often generate very long
reasoning traces, making inference computationally expensive. Existing approaches typically improve efficiency either through inference-time early-stopping mechanisms or by explicitly encouraging shorter
reasoning during training, for example through reinforcement learning with length penalties. We show that substantial efficiency gains can instead emerge from a different kind of supervision: confidence. Using a self-supervised procedure, we fine-tune
reasoning models to predict their confidence in the answer at intermediate points along their own
reasoning trajectories using only 600 training problems. Confidence is used only as a training target: the loss contains no objective for
reasoning length, efficiency, or stopping. At inference, the fine-tuned models use the standard generation procedure, with no confidence elicitation or early-stopping mechanism. Despite this, self-supervised confidence fine-tuning makes
reasoning more efficient, reducing generated
tokens by up to 25% at matched accuracy across Gemma, Qwen, Nemotron, and GPT-OSS models on mathematical, scientific, and coding
reasoning benchmarks, with efficiency gains comparable to methods that explicitly optimize for shorter
reasoning. Analysis of
reasoning episodes further shows that confidence supervision largely preserves the base models' high-level
reasoning composition rather than selectively suppressing particular behaviors. Our results suggest that efficient
reasoning may emerge as a downstream consequence of learning metacognitive signals, without being directly optimized.
Read the full abstract ▾
All authors Parsa Hosseini, Akasha Tigalappanavara, Sumit Nawathe, Chenrui Fan, Sourya Basu, Genta Indra Winata, Anirban Das, Soheil Feizi, Nima Chitsazan
Artificial Intelligence · Cross-listed from stat.ME
Statistical attribute alignment for black-box generative AI via output post-processing
Kevin Jiang, Morgane Austern, Edgar Dobriban et al. (+1 more)
Generative AI systems are increasingly used, but aligning their outputs with user requirements poses a continuing challenge. Here, we aim to ensure that the distribution of an attribute of an AI-generated output aligns with a user-specified target. This is motivated by examples such as
fairness, where we want to ensure that a protected attribute (e.g., gender, race, or age categories) follows a desired distribution, and synthetic data generation, where we want the generated data to be representative of a target distribution. We study the practically important black-box access setting, where a user can repeatedly query a generative AI model. The goal is to return $m\ge 1$ outputs whose joint attribute distribution is as close as possible to this target. For both exact and approximate
alignment, we develop
algorithms that minimize the expected number of queries to the generator, and we further demonstrate their optimality as the number of requested outputs $m \rightarrow \infty$. Experiments on text-to-image generation and geocoded persona generation tasks show that our post-processing
algorithms improve statistical attribute
alignment, complementing prompting-based interventions.
Read the full abstract ▾
All authors Kevin Jiang, Morgane Austern, Edgar Dobriban, Jason M. Klusowski
Artificial Intelligence · Cross-listed from cs.SE
Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer
Md Shohel Arman and Igor Molybog
We investigate whether natural-
language documentation helps coding
agents resolve software issues, and we build the tools to construct and evaluate it. We introduce a roundtrip
benchmark that scores code descriptions by whether code regenerated from them passes the original tests, and show that completeness, not length, drives a description's fidelity. Using the
benchmark as an optimization signal, we discover a description-writing
prompt that reaches full fidelity and generalizes to unseen files. We then test the hypothesis that motivated the work: that better documentation helps an
agent resolve real repository issues. Across two model families and ten repositories, and against a positive control confirming that our evaluation can detect a genuine improvement, we find that it does not. When the source is present, neither static compact documentation nor retrieved context beats the issue alone. We report this negative result together with the
benchmark and the optimizer, and we characterize the boundary at which documentation helps.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CV
OC-GS: Gaussian Splatting for Irregular Turntable Capture
Jae Joong Lee and Bedrich Benes
Uneven rotation and dropped frames make equal-angle assumptions unreliable for turntable reconstruction. We present OC-GS, an object-centric Gaussian splatting that refines each image's angle while maintaining a shared camera, rotation axis, and pivot. This orbit-consistent refinement jointly optimizes image-derived geometry and angles to reconstruct objects from sparse, irregular captures. On rendered objects with 12, 8, and 6 irregularly spaced views, OC-GS achieves mean foreground PSNR scores of 21.26, 19.36, and 15.83dB, respectively, exceeding all four evaluated pose-free Gaussian splatting baselines in each condition. Under a shared trainer, refining image-estimated angles improves mean foreground PSNR by 7.88dB over keeping those estimates fixed. An ablation study shows that both image-derived angle initialization and the shared motion model contribute to the improvement. On real captures, OC-GS's refinement increases mean foreground PSNR by 0.70dB. Results show that refining uncertain angles within a shared motion model improves reconstruction from sparse, irregular turntable captures.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.HC
Adapting for AI: How elementary teachers adjust their practices for an AI-integrated curriculum
Fasika Melese, Ruiyang Wu, Xinyue Cui et al. (+4 more)
Conversational AI tools are entering children's everyday experiences, and schools are interested in adopting them. However, successful classroom integration depends not only on the technology but also on the work teachers do to make it usable and appropriate for their students and classroom context. There is little known about how elementary teachers work as they implement conversational AI tools in real classrooms. In this study, we examine three teachers' experiences implementing an AI literacy and English
Language Arts (ELA) curriculum built around ToyTalk, a conversational AI toy development platform, over 13 instructional days, a three-week summer camp. Drawing on daily individual reflections, group reflections, and post-camp interviews, we find that teachers' adaptive practices of repair, differentiation, translation, and balancing sit at the intersection of three tensions (technology, learner, and instruction). Teachers' understanding of AI and their role evolved over the camp experiences. From these findings, we contribute design implications and considerations for deploying conversational AI within elementary classrooms.
Read the full abstract ▾
All authors Fasika Melese, Ruiyang Wu, Xinyue Cui, Joanna Perkins, Xiaoyi Tian, Tiffany Barnes, Shiyan Jiang
Artificial Intelligence
DeepEdu-v1: Efficient and Scalable Agentic LLMs for Vietnamese Education
Quang Nguyen, Hieu Nguyen, Hien Hoang et al. (+3 more)
AI tutoring could markedly improve learning outcomes for students in developing regions such as Vietnam, yet the two obvious paths both fall short. Cloud assistants such as ChatGPT route sensitive student data to foreign servers—violating data-sovereignty laws such as Vietnam's Decree 53—and, pre-trained on Western-centric corpora, are not organized around the national textbook curriculum, so their knowledge of local content is unsystematic and frequently hallucinated. Self-hosting an open model keeps data on-premise but hits a two-fold wall: post-training quantization (AWQ, GPTQ) tames the static weight footprint, yet the dynamic KV cache and prefill latency of long tutoring contexts still cause out-of-
memory failures and slow responses on consumer GPUs, while the model keeps hallucinating on region-specific material. We present DeepEdu-v1, an AI-tutoring system for Vietnamese education built on SCALE (Self-improving Context-Aware Learning Engine), a framework with two innovations. First, a long-context inference engine amortizes
token selection from per-sub-chunk to per-cluster granularity; on long-context
retrieval it issues x7.7 fewer
retrieval calls than a state-of-the-art selective-
attention baseline, cutting prefill latency (TTFT) by roughly 35% while matching or improving task accuracy. Second, a self-improving agentic layer continuously curates a verified playbook from past interactions instead of fine-tuning, a design intended to progressively reduce reliance on dominant-
language priors as trustworthy local knowledge accumulates. In its deployed configuration, DeepEdu achieves a nearly x2 TTFT speedup over standard vLLM serving and lifts agentic accuracy from 70.0% to 79.5% on complex tasks, with the strongest per-track gains across financial-
reasoning and interactive-
agent benchmarks.
Read the full abstract ▾
All authors Quang Nguyen, Hieu Nguyen, Hien Hoang, Toan Pham, Cong Tran, Nam Vu
Artificial Intelligence
Multi-agent Scaling Across Disjunctive and Compensatory Tasks
Carolina Fortuna and Blaz Bertalanic
Multi-
agent LLM systems are often expected to improve as team size increases, yet the scaling behavior may depend on task structure. Our central contribution is to introduce Steiner's taxonomy of group tasks as a framework for analyzing multi-
agent LLM scaling and focusing the analysis on disjunctive and compensatory tasks. We model independently sampled
agents as conditionally independent given the item, which yields their large-team limits: plurality voting converges to the model's modal answer, and averaging converges to the model's item-level bias. Across selected representative
benchmarks, 13 open-weight models, and teams of up to 30
agents, we find qualitatively different scaling behavior. On disjunctive tasks, the probability that at least one
agent is correct grows by 5-20 points with team size, but plurality voting over
agents that answer directly realises almost none of this potential, as the model predicts to within 0.5 points on average. Multi-round revision raises accuracy considerably, yet the gain is nearly the same with one peer as with 29. In contrast, scaling provides little benefit on
Fermi estimation, despite its natural suitability for aggregation: item-level biases shared across the samples of a model account for about 87% of the squared error, so averaging reduces error by only about 6%. Combining model families helps on
Fermi estimation but does not surpass the strongest member on disjunctive tasks. These results show that task structure, together with the mechanism combining member outputs, is a fundamental determinant of team scaling.
Read the full abstract ▾