Artificial Intelligence · Cross-listed from cs.CV
Exo2EgoHOI: Hand-Object-Interaction Aware Exocentric-to-Egocentric Video Generation
Hongjia Zhai, Xiyu Zhang, Haoran Zhang et al. (+5 more)
Egocentric
videos of human manipulation provide valuable visual experience for embodied intelligence, yet collecting such data at scale is costly. Exocentric-to-egocentric
video generation offers a scalable alternative by transforming abundant third-person manipulation
videos into first-person observations. However, existing methods often struggle to faithfully preserve demonstrated hand-object interactions (HOI) across large viewpoint changes due to insufficient fine-grained interaction guidance and weak object-centric anchoring. We present Exo2EgoHOI, an HOI-aware
video generative framework for interaction-preserving exocentric-to-egocentric translation. To preserve fine-grained HOI, we introduce a unified 4D HOI prior that combines scene geometry, articulated hand renderings, and dense hand-object relation fields, together with a dual-branch residual adapter for injecting structural and relational cues into the
video generation backbone. To preserve object consistency, we introduce Decomposed Gated Cross-
Attention, which separately encodes object and background references and adaptively integrates global semantic and local appearance features as object-centric anchors. Experiments on ARCTIC-HOI and Ego-Exo4D demonstrate substantial improvements in object consistency and HOI preservation while maintaining competitive visual fidelity. In particular, on ARCTIC-HOI, Exo2EgoHOI improves object mIoU by 32.3% and reduces MPJPE and PA-MPJPE by 34.7% and 50.0%, respectively, relative to the respective best baseline results. Project page: https://rcl-robotics.github.io/Exo2EgoHOI/.
Read the full abstract ▾
All authors Hongjia Zhai, Xiyu Zhang, Haoran Zhang, Zhichao Ye, Haomin Liu, Guofeng Zhang, Ian Reid, Xingxing Zuo
Artificial Intelligence · Cross-listed from cs.CV
After a Decade: Bringing Shadow Removal into the Real World with Agentic Training Data
Shilin Hu, Jingyi Xu, Dimitris Samaras et al. (+1 more)
Shadow removal looks nearly solved on established
benchmarks, yet remains brittle in the real world. Models have advanced; the paired training data they rely on have barely changed in nearly a decade. The reason is simple: obtaining a shadow-free target requires removing the occluder while keeping the scene, camera, and illumination otherwise unchanged, making diverse paired data difficult to capture. Meanwhile, large shadow detection
datasets already contain diverse real-world images and masks, but no shadow-free targets. To turn this abundant but incomplete data into paired supervision, we propose an offline agentic workflow combining physics-motivated generation, failure detection, feedback-driven retry, candidate selection, and deterministic correction. Using this workflow, we construct AgenticShadow, a
dataset of 17,138 image-mask-target triplets spanning general scenes, faces, and remote sensing. Our construction workflow reduces Color Distribution Difference by 50.5% over previous shadow removal work, while training existing shadow removal models on AgenticShadow reduces cross-domain LAB RMSE by 19.7-37.5%.
Read the full abstract ▾
All authors Shilin Hu, Jingyi Xu, Dimitris Samaras, Hieu Le
Artificial Intelligence · Cross-listed from cs.CL
Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent
Zheyuan Zhang, Mengyuan Chao, Ke Xiao et al. (+5 more)
Modern LLM
agents increasingly tackle complex tasks through interactive, long-horizon exchanges with users, while existing
benchmarks generally assume that users always accurately and sufficiently communicate a fixed intent. However, this oracle communication assumption rarely holds in practice: users may miscommunicate, change their goals, and run out of patience. We define this task setting as Interactive Intent
Alignment, where
agents must recover and continuously track the user's current intent despite imperfect communication and evolving goals. To study this setting, we introduce Drift-Bench++, a principled
benchmark construction pipeline for verified executable tasks with controlled misalignment and intent shifts, along with an interaction protocol featuring finite patience, diverse simulated users, and silent interaction-conditioned shifts. We further develop GRIP, a comprehensive evaluation protocol covering task grounding, user realism, inquiry effectiveness, and adaptation to evolving intent. Across diverse environments, models, and interaction conditions, stronger interaction consistently helps but remains far from oracle performance; Validation on deployed ProdAgent sessions further shows that the modeled failures are prevalent and consequential in deployment. By providing a unified, executable
benchmark for interactive intent
alignment, Drift-Bench++ offers a foundation for evaluating and advancing
agents under realistic communication and evolving intent.
Read the full abstract ▾
All authors Zheyuan Zhang, Mengyuan Chao, Ke Xiao, Ziyi Chen, Daoan Zhang, Yan Zhang, Yanfang Ye, Wei Xu
Artificial Intelligence
Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts
Abinitha Gourabathina, Haoran Zhang, Yuexing Hao et al. (+2 more)
As large
language models (LLMs) are increasingly used in clinical settings, it is critical to evaluate their reliability under realistic variation in clinical text. We study this question in clinical triage, comparing LLMs to practicing physicians under text perturbations that preserve the underlying clinical setting. We introduce a
benchmark of over 6,000 clinical scenarios, 7,000 physician annotations, and 225,000 model responses. Using this
benchmark, we make two key observations. First, LLMs are more likely than physicians to recommend unnecessary care at baseline, and this tendency increases under perturbed inputs. Further, we find that LLM recommendations are more sensitive to gender and tone perturbations than human recommendations. Together, these results demonstrate that LLMs can vary under clinically irrelevant textual changes, highlighting the need for deployment-oriented evaluations grounded in expert physician behavior.
Read the full abstract ▾
All authors Abinitha Gourabathina, Haoran Zhang, Yuexing Hao, Walter Gerych, Marzyeh Ghassemi
Artificial Intelligence · Cross-listed from cs.CL
Prompt2Skill: Unsupervised Skill Optimization From Natural Language Instructions
Bo Ni, Li Li, Ryan A. Rossi et al. (+2 more)
Skills are external artifacts that Large
Language Models (LLMs) consume at inference time to improve their performance on specialized domains by incorporating relevant procedural and domain knowledge. Expert-authored skills are expensive to produce, and the resulting artifacts are not optimized for the specific model that consumes them, whose failure modes can vary with version, scale and training. In addition, emerging tasks may fall outside the scope of existing skill libraries, creating a need to develop new skills before curated training data become available. Recent works have explored automated skill optimization through reflection, but they require a curated, in-distribution training set, which users might not always have. To address these limitations, we present
Prompt2Skill, a framework that builds skills from natural-
language task description alone. From the
prompt, the system derives a task specification, discovers or synthesizes
datasets, and refines the skill in a closed loop of reflective editing. Across four domains spanning question answering, reading comprehension, spreadsheet manipulation, and mathematical
reasoning,
Prompt2Skill consistently outperforms the direct prompting baseline, achieving an average improvement of 10.8 across open-source and frontier models.
Read the full abstract ▾
All authors Bo Ni, Li Li, Ryan A. Rossi, Franck Dernoncourt, Tyler Derr
Artificial Intelligence · Cross-listed from cs.LG
FlexRouter: Learning Complementary Model Sets for Flexible LLM Routing
Wang Wei, Harry Yang, Tiankai Yang et al. (+6 more)
Existing Large
Language Model (LLM) routing methods score LLMs independently to select top-$k$ models. However, this ignores model correlations and enforces a rigid computational budget. Consequently, routers often select redundant models that share failure modes, limiting the overall probability of success. To address this, we propose FlexRouter, a routing framework that explicitly models model complementarity. FlexRouter optimizes for answer coverage, maximizing the probability that at least one selected model yields a correct response. This objective aligns with practical inference pipelines where multiple candidate outputs are generated and a downstream verifier or user selects the final one. We formulate routing as a coverage-oriented subset selection problem and model the routing policy using Determinantal Point Processes (DPPs), which naturally capture both model competence and redundancy. To directly optimize coverage without requiring a ground-truth target subset, we introduce a training objective based on marginalizing over failure sets. During inference, we employ a greedy strategy based on marginal log-determinant gains, enabling the router to adaptively determine subset sizes without a predefined budget. Extensive experiments on the large-scale RouterEval
benchmark demonstrate that our proposed FlexRouter achieves higher coverage with lower redundancy across both in-domain and out-of-domain tasks than strong baselines while maintaining flexible inference cost.
Read the full abstract ▾
All authors Wang Wei, Harry Yang, Tiankai Yang, Samyadeep Basu, Hongjie Chen, Andy Zhao, Franck Dernoncourt, Ryan A. Rossi, Hoda Eldardiry
Artificial Intelligence · Cross-listed from cs.CV
Retargeting Motions to Diverse Skeletons via Learnable Flattening
Kia-Jüng Yang, Fabian H. Sinz and Paweł A. Pierzchlewicz
Cross-structural motion retargeting aims to transfer motion between different skeletal
topologies. Despite recent progress, existing state-of-the-art models struggle with reliability in zero-shot settings, i.e. skeletons with different
topologies which were unseen during training, and recent
Transformer-based attempts have failed to outperform specialized
geometric methods. We bridge this gap with a
Transformer Autoencoder that learns a
topology- and translation-invariant latent space. Our core contribution is a learnable flattening of skeletal graphs that captures both local dependencies and global structure. Unlike the standard
transformer architecture, which adds positional information to
token content, we integrate graph-based positional encodings multiplicatively, a design choice that follows directly from our flattening formulation. The resulting model handles diverse skeletal
topologies within a single unified architecture and trains in a fully unsupervised manner, requiring no paired retargeting data. Ablation studies show, that the graph encodings, multiplicative formulation, and
Transformer backbone is critical for the performance. In zero-shot evaluations, our method reduces global joint position error by $43-47\%$ over current
benchmarks. A user study ($n = 37$), including expert animators, further ranks our approach highest in motion
alignment and physical plausibility ($p < 0.05$). These results demonstrate that our model design is key to making
transformer architectures effective for motion retargeting, outperforming existing approaches.
Read the full abstract ▾
Artificial Intelligence
Conditional Generation of Creative Chess Puzzles with Diffusion Models
Aatu Selkee, Severi Rissanen, Xidong Feng et al. (+2 more)
While modern
language models demonstrate impressive generative capabilities, they often struggle with constrained, counter-intuitive creative tasks. To address this limitation, we explore chess puzzle generation as a rigorous testbed for computational creativity and
reasoning, a domain where altering a single piece can invalidate an entire solution. We propose a novel approach for conditional generation of creative chess puzzles using masked
diffusion models. Unlike previous methods, our non-directional
diffusion approach allows for conditioning on specific tactical themes and partial board positions. We introduce a novel auxiliary task of simultaneous best-move prediction, which improves solution uniqueness by 11.6% and theme-conditioning accuracy by 2.5%. To further optimize solution uniqueness and theme conditioning, we establish a reinforcement learning framework adapted from Denoising
Diffusion Policy Optimization (DDPO). This RL training increases the yield of unique and theme-matching positions by 89.1%. Finally, we release the first open-weights models (Appendix B) for chess puzzle generation, offering a new pathway for controllable, creative generation.
Read the full abstract ▾
All authors Aatu Selkee, Severi Rissanen, Xidong Feng, Tom Zahavy, Eric Malmi
Artificial Intelligence
Towards Model as a Library: Offline, Community-Sourced AI for Low-Resource African Languages
Fendji K. E. Jean Louis
Large
language models are frequently proposed as a route to AI-powered services for African communities, but they are least reliable exactly where the need is greatest: all African
languages remain low-resource by any standard measure, and models trained on scraped, standardised text systematically misrepresent the dialectal and regional variation of how people actually speak. We introduce Model as a Library (MaaL), a software architecture that packages small, community-enrolled
speech models as versioned on-device dependencies, enabling offline structured data collection that cannot generatively hallucinate, for populations that current
language models serve worst. Rather than relying on web-scraped corpora, MaaL's vocabulary is enrolled directly from a small number of example recordings by the speakers themselves, at the point of deployment. We describe the architecture and its central mechanism - keyword spotting that turns a closed-vocabulary text form into a voice form, filled and submitted entirely on-device - and propose transpiling the closed-vocabulary elements already present in widely-deployed digital form tools into MaaL schemas, a low-friction path to voice-first, offline data collection for the low-literacy populations these tools already reach. This is a position and system-design paper: we describe the concept, the mechanism, and an analytical feasibility case, and identify what a working implementation still requires.
Read the full abstract ▾
Artificial Intelligence
Defining and Categorising Human-AI Interactions in Clinical Trials: A Multidimensional Human-AI Classification Approach
Sandra Woolley, Tim Collins, Khalid Khattak et al. (+3 more)
This paper examines human-AI interactions (HAIIs) in clinical trials and presents a multidimensional categorisation framework that classifies interactions according to AI tasks, human-AI relationships, interaction configurations and interacting human groups. We define HAII, examine existing taxonomies and extend existing categorisation approaches through this novel multidimensional framework. We purposively sampled 15 clinical trials from a previously reported
dataset. Each trial was independently categorised by two human reviewers and six large
language model (LLM) classifiers. The proposed categorisation provides a structured method for the consistent identification, comparison and synthesis of human-AI interactions across clinical-trial records. The framework is intended to support more consistent comparison and synthesis of AI-related clinical trials and to make explicit the different forms of human involvement associated with AI interventions. The results demonstrate the potential for LLM-assisted categorisation while indicating the continuing importance of human judgement where trial records are incomplete or ambiguous. The principal contribution is a proposed multidimensional framework that brings together AI tasks, human-AI relationships, interaction configurations and interacting human groups within a single approach designed for clinical-trial records. Its significance lies in its potential to support more systematic identification, comparison and synthesis of how humans and AI interact in clinical trials.
Read the full abstract ▾
All authors Sandra Woolley, Tim Collins, Khalid Khattak, Illia Chernomorets, Ariane Arevalo, Chris Richardson
Artificial Intelligence
Demographic Pluralism: Inference-Time Modeling of Pluralistic Human Preference Distributions
Meng-Chen Wu, Qipin Chen, Ansh Jain et al. (+3 more)
Large
language models (LLMs) are increasingly used in culturally sensitive settings, where
alignment requires representing diverse preferences within populations. Yet existing methods model populations at coarse demographic or community levels and overlook within-group variation. We introduce Demographic Pluralism, an inference-time framework that estimates population-level opinion distributions without opinion-distribution training data or task-specific fine-tuning by generating multiple perspectives within demographically grounded groups. Across four backbones on GlobalOpinionQA and VITAL, it reduces Jensen-Shannon distance by 8.4%-26.4% over Modular Pluralism. Among weighted, equal-weighted, and inverse-weighted aggregation, equal weighting performs best overall; group-level error also increases with group weight, helping explain weighted aggregation's weaker performance.
Read the full abstract ▾
All authors Meng-Chen Wu, Qipin Chen, Ansh Jain, Tess Wood, Zhe Du, Si-Chi Chin
Artificial Intelligence · Cross-listed from cs.LG
Towards Universal Wasserstein Barycenters through Flow Matching
Eduardo Fernandes Montesuma
Defining a weighted mean over probability measures under probability metrics is a central tool in probabilistic machine learning. Under the Wasserstein metric, these are called Wasserstein barycenters. While most approaches compute barycenters for a fixed weight vector, approximating the whole family of barycenters over the simplex, which we call the Wasserstein simplex, remains underexplored. We refer to this problem as Universal Barycenter Approximation, and propose BaryFM, a flow matching model transporting the marginal measures into any barycenter in the Wasserstein simplex. Once trained, the network can draw samples from measures in the Wasserstein simplex through an ordinary differential equation. We validate our method on 4 downstream tasks: domain adaptation, generalization,
Bayesian posterior aggregation and algorithmic
fairness. BaryFM achieves the best average rank among 15 competing methods across 10 domain adaptation
benchmarks, matching or surpassing non-universal solvers.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
ReLaG: A Scalable Framework Generalizing Random Splits to Data with Latent Relations
Anthony Lavertu, Jacob Cote, Sophie Gobeil et al. (+3 more)
Random splitting can yield non-independent train–test subsets when a
dataset contains related samples, as is common in certain applications such as biochemical studies. This leads to overly optimistic generalization estimates. Here, we introduce ReLaG, a modality-agnostic framework that models sample relatedness through a hierarchical latent-variable process and infers groups of related samples using proximity graphs and community detection to produce independent train–test subsets. Across molecular and
protein datasets, ReLaG matches existing relation-aware methods while scaling substantially better, enabling splits at previously impractical
dataset sizes. We further introduce a label-free procedure that adapts the splitting resolution to production data, aligning evaluation with the intended deployment setting. ReLaG's inferred groups provide a cheap estimate of effective
dataset size, enabling diversity-aware
dataset scaling. ReLaG is open source and can be installed with pip install relag.
Read the full abstract ▾
All authors Anthony Lavertu, Jacob Cote, Sophie Gobeil, Jacques Corbeil, Isabeau Premont-Schwarz, Pascal Germain
Artificial Intelligence · Cross-listed from cs.LG
Does This Action Still Explain the Task? Reverse Scoring for Diffusion Language Model Agents
Jiacheng Qiu, Christopher E. Mower, Jan Peters et al. (+2 more)
Diffusion-based large
language models (dLLMs) promise to break the sequential latency bottleneck of autoregressive
agents through parallel decoding, but recent evaluations show this efficiency does not transfer to embodied agentic competence: dLLM-backed
agents repeatedly fall into retry loops, re-issuing an action long after it has failed. We give a mechanistic account of this failure and a training-free remedy. We trace the retry loop to the adaptivity of masked decoding: the sampler commits the positions it is most confident about and defers the uncertain ones, and at a failure state the context already offers a confident fill for the deferred decision, i.e. the failed action itself, so the retry is committed without the failure feedback ever being confronted. We model the resulting distortion of the action distribution as a task-blind corruption: contextually salient actions (e.g., the action just taken) receive inflated probability by a factor that depends on the state and the action but not on the task. Under this model, we analyse an invariance proposition: the task-blind factor cancels exactly from the reverse conditional, i.e. the likelihood of the task given the state and a candidate action, which coincides with the task posterior of an idealized uncorrupted model. Masked dLLMs evaluate the reverse conditional natively, unlike autoregressive models, by masking the task
tokens and denoising, at the cost of a few parallel passes per candidate. We instantiate the rule as Reflect Reverse and evaluate it on four multi-turn embodied
benchmarks, where it improves task success and progression rates over forward-scoring baselines.
Read the full abstract ▾
All authors Jiacheng Qiu, Christopher E. Mower, Jan Peters, Haitham Bou-Ammar, Matthieu Zimmer
Artificial Intelligence · Cross-listed from cs.LG
Revisiting scaling laws for reward optimization
Ali Aouad, Aymane El Gadarri and Vivek F. Farias
Scaling laws for optimization against reward models in AI
alignment have pinned down how performance depends on optimization effort—measured by a KL-divergence budget relative to a reference policy. Beyond a certain budget, over-optimization (or reward hacking) can arise: because we optimize against a proxy reward model (distinct from true rewards), performance can plateau or degrade. Naturally, the proxy reward's accuracy depends on how much preference data (often in the form of pairwise comparisons) was used to train it. However, existing research does not cleanly identify how performance jointly scales with the amount of training data and the divergence budget. Our main contribution is to provide an empirically accurate and theoretically grounded scaling law in such context. Performance roughly scales as $\Theta(\sqrt{\min\{\log(M),K\}})$, where $M$ is the number of comparisons in training data and $K$ is the policy's divergence budget. We develop an information-theoretic model to establish this upper bound and prove it is tightly achievable through a constructive procedure. Informed by this, we conduct extensive empirical evaluations using a real-world annotation setup, whereby a large 70B gold reward model generates feedback data and proxy reward models are trained from less capable models (0.6B to 4B). Our scaling law provides an excellent fit (R2 from 97% to 99%), outperforms alternative specifications, and remains robust across model sizes, noise, and optimization procedures (best-of-$N$ or policy tilting). Our evidence suggests that reward optimization is analogous to a surprisingly simple selection task: choosing from a sequence of IID Gaussian random variables using noisy preference feedback.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
MM-FinEval: A Multi-Task Multimodal Benchmark for Real-World Financial Forecasting
Dong Shu, Yanguang Liu, Huopu Zhang et al. (+4 more)
Financial forecasting from earnings conference calls requires models to reason over complex corporate disclosures,
market expectations, and subtle communication signals. However, existing financial
benchmarks are often limited to unimodal inputs or single-task settings, making it difficult to evaluate whether
multimodal large
language models (LLMs) can support real-world financial analysis. In this paper, we introduce MM-FinEval, a novel
benchmark designed to evaluate
multimodal LLMs across multiple financial tasks. MM-FinEval spans a diverse timeline from 2019 to 2022. The entire proposed
dataset contains 2,045 S&P 500 conference earning calls as inputs and 12 financial task labels as outputs. Each input contains three modalities: a word-to-word text transcript of the earning call, the corresponding presentation slides used during the call, and the entire audio recording. To establish a rigorous evaluation framework, we analyze 19 baseline models across three distinct model categories: Image-Text, Audio-Text, and Any-to-Any configurations. We observe that small-size Any-to-Any models processing all three modalities achieve strong performance, even when compared against larger proprietary models restricted to two-modality inputs. This indicates that our tri-modal
dataset design introduces useful, non-redundant information. These results validate that text, audio, and visual data serve as important, complementary signals that mimic the decision-making process of expert human analysts.
Read the full abstract ▾
All authors Dong Shu, Yanguang Liu, Huopu Zhang, Saisai Hu, Haiyan Zhao, Hekun Huang, Mengnan Du
Artificial Intelligence · Cross-listed from cs.LG
ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights
Jonathan Mei, Sang Hyub Kim, Oliver Knitter et al. (+2 more)
We introduce ShamAN-Q, a sub-1-bit post-training quantization method that extends NanoQuant by replacing each its diagonal reconstruction geometry with a tractable dense
curvature metric, using a general paradigm popularized by the Shampoo optimizer. For each linear weight, ShamAN-Q fits a Kronecker product to the empirical Fisher information
matrix of a small calibration set by Kullback–Leibler minimization, forming a Mahalanobis reconstruction loss from the result. The continuous ADMM updates from NanoQuant become solutions to Sylvester equations, while its discrete projection and deployment format remain unchanged. Because the
curvature is local to a given set of weights, ShamAN-Q re-measures the input
curvature statistic for each layer immediately before layer factorization, periodically refreshing all statistics on the partially quantized model. ShamAN-Q also redistributes the uniform rank from NanoQuant across layers at the same total number of bits. On Qwen3-Base, ShamAN-Q lowers WikiText-2 perplexity at $\approx$1 bpw from 27.56 to 22.96 (0.6B), 19.21 to 16.72 (1.7B), and 14.29 to 13.80 (4B) while matching or improving zero-shot accuracy on the Eleuther LM Evaluation Harness. On 0.6B, ShamAN-Q at $\approx$0.8 bpw matches the published perplexity of NanoQuant at $\approx$1.0 bpw.
Read the full abstract ▾
All authors Jonathan Mei, Sang Hyub Kim, Oliver Knitter, Chi Chen, Martin Roetteler
Artificial Intelligence · Cross-listed from cs.MA
From Solo to Social Learning: Characterizing Recursive Social Improvement in LLMs
Kunal Jha, Max Kleiman-Weiner and Natasha Jaques
Large
language models (LLMs) can now improve themselves by revising the instructions they follow, and LLM
agents are increasingly orchestrated to work together on complex problems. However, self-improvement methods typically optimize one system at a time, and multi-
agent frameworks often have every model work toward a shared goal. We ask a different question. When each
agent pursues its own reward, can self-improving LLMs learn from one another well enough to improve the whole population? We call this capability recursive social improvement. We study populations that revise skill files and choose whether, when, and whom to copy from. Independent search, learning from peers, and acting all share one
token budget. In controlled environments, established social-learning
algorithms benefit from peers, but three LLMs do not. They earn less reward per
token than solo learners, and explore too narrowly or run out of
tokens before acting. We then let the models write and revise their own skills. Observing peers changes how they improve, helping one model find useful skills sooner and another spend less on private search. Neither, however, outperforms independent learners at the same cost. Skills are copied, revised, and passed on, so one discovery can seed further search. Yet these exchanges concentrate the population around fewer independent discoveries. Together, these results show that LLMs can make learning more efficient by copying from peers, but not yet more effective.
Read the full abstract ▾
Artificial Intelligence
VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation
Joshua Meyer, Sahar Shayegan, Ritiz Tambi et al. (+5 more)
Voice
agents in production must handle several requests, background
speech, and customers who lose patience. We introduce VAmoS Energy, a
benchmark that combines these challenges in 100 calls about utility billing and payment assistance. Each caller makes two to four requests. The
agent has sixteen tools backed by a stateful Stripe billing twin and the Apache Fineract loan engine, with account access blocked until caller verification succeeds. The tasks use public household electricity data and a policy based on Pennsylvania's residential billing rules. An LLM-as-a-verifier checks the
agent's actions and spoken figures against explicit requirements. On a calibration run, it agrees with a code verifier on 99.1% of checks. Across fourteen voice stacks and three repeats per task, completion ranges from 17.3% to 44.7%. Grok Voice leads, and Gemini 3.8 Live and GPT-Live follow at about the same cost per call. Background television reduces pooled completion from 38.7% to 8.6%. The simulated caller often accepts an incorrect result because it hears the
agent's words but cannot inspect its actions. These findings show why voice
agents need evaluation across the whole call, including what they say, what they change, and how they handle competing
speech.
Read the full abstract ▾
All authors Joshua Meyer, Sahar Shayegan, Ritiz Tambi, Ali Khan, Sun Kim, Victor Shih, Mehdi Jamei, Andi Partovi
Artificial Intelligence · Cross-listed from cs.SE
An Empirical Study of Architectural Shift from Traditional to AI-Enabled Simulink Controllers
Hadiza Umar Yusuf and Khouloud Gaaloul
Effective AI adoption in cyber-physical systems (CPS) depends on
embedding design knowledge into engineering practice. Yet as AI-enabled components increasingly replace analytically derived control laws, this occurs without a systematic understanding of how controller architectures differ or remain similar across paradigms. We address this gap with an empirical study of traditional and AI-enabled Simulink controllers, guided by a literature-derived taxonomy of ten structural categories and nine functional roles. The study analyzes 62 real-world models spanning 8 controller types and 10 application domains, and surveys 13 practitioners, identifying three architectural tensions. First, subsystem organization dominates all controller structures regardless of paradigm, occupying 68-72% of controller footprint, while core control logic occupies minimal space. Second, AI-enabled controllers rely heavily on discrete dynamics and user-defined abstraction, categories largely absent from AI literature, exposing a gap between described and implemented architectures. Third, constraint enforcement blocks largely disappear from AI-enabled models despite practitioner expectations. This reveals a misalignment where safety mechanisms shift from explicit structure to implicit training-time artifacts, breaking traceability.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CL
Personalized State-Transition-Aware Memory for Clinical Agents
Maryam Haghifam, Zahra Rajabi, Yizhou Sun et al. (+1 more)
Large
language model (LLM)
agents that reason over clinical records must track changes in a patient's state while preserving the history needed to understand them. Simply accumulating
memories leaves it unclear which information still applies, whereas overwriting earlier
memories can erase evidence needed to reconstruct treatment history and clinical trajectories. We introduce STAM, a state-transition-aware
memory framework that records state changes as new clinical entries arrive. STAM combines semantic
retrieval with typed clinical relations to identify affected
memories, maintaining current information in Active and superseded or resolved information in History. At read time, a query-dependent gate selectively serves historical
memory. Across four longitudinal clinical
benchmarks, we evaluate STAM with downstream question answering, direct state-maintenance diagnostics, and comparisons at approximately matched context lengths.
Read the full abstract ▾
All authors Maryam Haghifam, Zahra Rajabi, Yizhou Sun, Carlos Morato
Artificial Intelligence · Cross-listed from cs.MA
PANDA: A Decentralized Architecture with Flexible Orchestration for Scalable, Fault-Tolerant Multi-Agent Systems
Matthew D. Laws and Cristina Nita-Rotaru
Existing architectures for LLM-based multi-
agent systems (MAS) cannot reliably and efficiently solve multi-step tasks at scale: they struggle to support large numbers of
agents and concurrent tasks, tolerate failures, govern
agent interactions, and accommodate the diverse planning and execution patterns different tasks require. We present PANDA, a decentralized architecture that connects a large collective of heterogeneous, independently administered
agents, letting them discover each other's capabilities and self-organize into small specialized teams per task. PANDA scales by decoupling collective communication from team communication, allowing
agents to participate in multiple teams simultaneously, load-balancing tasks across the collective, and scheduling concurrent work within each
agent. PANDA further separates the underlying architecture from the orchestration strategy, supporting three planning and execution patterns (
star, chain, and mesh) that can be selected according to the structure and requirements of each task. PANDA detects infrastructure and orchestration failures and recovers affected tasks by dynamically replanning around failed components. Finally, to provide governance without a centralized service that would limit scalability, PANDA uses a web-of-trust model to constrain
agent interactions to established trust relationships. We evaluate PANDA on the HotPotQA
benchmark, demonstrating that it scales to thousands of
agents, assembles teams in milliseconds, matches state-of-the-art accuracy at up to 8x the efficiency, and sustains 100% task completion under faults where existing systems fail.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CL
KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy
Xueting Fang, Zehui Li, Yang Yang et al. (+5 more)
Most clinical
benchmarks evaluate
language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients present information in different ways, and clinicians must obtain relevant history and determine which examinations are needed before reaching a diagnosis. Diagnostic accuracy alone therefore cannot establish whether an
agent gathered essential information or conducted an appropriate clinical assessment. Furthermore, existing
benchmarks lack professional clinicians' verification. To address this gap, we introduce KlinikeBench, a
benchmark of 333 clinician-authored tasks, each providing an isolated sandbox environment with a virtual patient, clinical tools, and task-specific success criteria. More than 35 clinicians contributed to case authoring and
benchmark evaluation. In an empirical study, clinicians gave simulated dialogues higher mean quality ratings than reference conversations, which is adapted from real conversation. In each task, an LM has a fixed budget of turns to communicate with the patient, ask about relevant history, request examinations, follow action constraints, and record a final diagnosis. We score these steps separately as well as together. Across 31 models and seven model families, the best-performing models (e.g., GPT-6-astra and Claude Opus 5) succeed on less than 30% of tasks, even though their diagnosis accuracy reaches 90.7%. Some models benefit from talking with the patient; others diagnose well from a complete chart but perform much worse in conversation. Overall, KlinikeBench provides a testbed for evaluating the full clinical encounter and reveals a substantial gap between diagnostic accuracy and performance in interactive clinical assessment. All the code and data is available on https://zehui127.github.io/klinikebench/
Read the full abstract ▾
All authors Xueting Fang, Zehui Li, Yang Yang, Camilla Giovino, Shubh K. Patel, Shailly Prajapati, Vallijah Subasri, Caihua Shan
Artificial Intelligence · Cross-listed from cs.CV
Caption-Mediated Perceived-Safety Estimation for Pedestrian Routing
Simon Parkinson, Paloma Liu, Wei Zheng et al. (+1 more)
This paper presents an explainable approach to pedestrian routing, in which perceived safety is estimated from street-level imagery through an explicit natural-
language intermediate representation. A
vision–
language model caption is generated and stored before any scoring is undertaken, and the perceived-risk class is derived entirely from structured features of that stored text, so that every segment score remains inspectable by the user. Nine captioning conditions across five model families are benchmarked against a direct Contrastive
Language–Image Pre-training (CLIP) image-
embedding baseline under an identical downstream pipeline, and the caption-mediated representation is found to reach parity with the image
embedding rather than to trail it. The approach was deployed over 654,115 images covering 36 electoral wards in two locations in Northern England (Manchester and Huddersfield). Independent field validation against 3,669 locally collected ratings of 494 images across 70 participant sessions established agreement that is statistically significant but modest, at $r=0.262$, against a measured noise ceiling of 0.737 imposed by disagreement between raters. A single-use confirmatory test then found that a pipeline 44% stronger on the supervised
benchmark did not produce measurable improvement in the field ($r=0.250$, $p=0.84$), so the
benchmark gains did not predict the deployment gains in this case. Routing behaviour varies systematically with journey length. There is negligible change below 1\,km, reaching a median increase of 12.78% in low-risk route length for a median detour of 2.73% on journeys of 3 to 6 km.
Read the full abstract ▾
All authors Simon Parkinson, Paloma Liu, Wei Zheng, Mohammadreza Sheikhfathollahi
Artificial Intelligence · Cross-listed from cs.CR
Security-Enhanced Seed-Based Weight Quantization for Large Language Models
Qiuyu Ren, Sudipta Paria, Aritra Dasgupta et al. (+1 more)
Large
language models (LLMs) incur substantial storage,
memory-bandwidth and energy costs, motivating compact weight representations. Existing seed-based compression methods reconstruct weights from compact pseudo-random representations but do not explicitly account for the non-uniform sensitivity of model weights. We introduce Seed-Q, a security-enhanced sensitivity-aware seed-based weight compression framework that uses lightweight Linear Feedback Shift Register (LFSR)-based weight generation with non-uniform bit allocation. Our approach assigns larger representation budgets to sensitive weights while aggressively compressing less sensitive regions. Importantly, this non-uniform allocation requires no side-information: the decoder deterministically reconstructs the bit-allocation schedule, with no rung depending on the decoded weights, eliminating the need to store per-block metadata or use calibration data while preserving the baseline coding rate. Experiments across diverse LLMs show that Seed-Q matches 4-bit perplexity of SeedLM with fewer bits, while at the same 4 bits/weight it reduces both perplexity degradation and zero-shot accuracy loss relative to SeedLM. We also show that Seed-Q simultaneously achieves high security against bit-flip attacks on model parameters, as bit corruption affects multiple reconstructed weights, greatly amplifying its impact and making it easier to detect. We further implement Seed-Q in an ASIC-based accelerator and demonstrate modest hardware overhead compared to prior seed-based approaches.
Read the full abstract ▾
All authors Qiuyu Ren, Sudipta Paria, Aritra Dasgupta, Swarup Bhunia
Artificial Intelligence · Cross-listed from cs.IR
Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering
Bhagyesh Rathi, Eshan Chawla and William B. Andreopoulos
Retrieval-Augmented Generation (RAG) is now the standard way to ground Large
Language Models (LLMs) in external knowledge, yet the design space of
retrieval pipelines is large and the trade-offs between variants are not well understood, especially on domain-specific corpora at realistic scale. In this work, we present a controlled comparison of six
retrieval strategies for scientific question answering: (i) classic top-k dense
retrieval, (ii) LLM-based query rephrasing, (iii) query rephrasing followed by LLM-based reranking, (iv) multi-query fusion via Reciprocal Rank Fusion (RRF), (v) an agentic tool-call pipeline in which the generator decides for itself whether to retrieve, and (vi) late-interaction
retrieval with ColBERTv2. All six pipelines share the same generator (Meta-Llama/Llama-3.1-8B-Instruct),
prompt, and evaluation protocol; the five single-vector pipelines additionally share SPECTER2
embeddings and a Chroma vector store; and all six retrieve from the full corpus of 463,971 arXiv papers dated 2024-2025. To support reproducible, large-scale evaluation, we also release a synthetic question
dataset of 19,484 problem-statement and methodology questions generated by Llama-3.1-8B-Instruct from a random sample of 10,000 papers across academic domains (query generation succeeded for 9,742 of them), and every strategy is evaluated on this same query set. We describe the architecture and implementation of each pipeline, release the code and the synthetic question
dataset, and evaluate each strategy with an LLM-as-a-judge protocol along multiple quality dimensions, together with direct gold-paper
retrieval metrics. The result is an open testbed for studying the cost and quality trade-offs of RAG design choices on a research-literature corpus, and a basis for future work on faithfulness,
retrieval robustness, and agentic
retrieval.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.IT
Derandomizing Dense Binary Hypervector Codebooks for Quantized Scalars
Dmitri Rachkovskij, Evgeny Osipov, Olexander Volkov et al. (+2 more)
Hyperdimensional computing and vector symbolic architectures often represent quantized scalar levels by dense binary codebooks whose level-to-level similarity is intended to follow a prescribed function of scalar separation. At finite dimensionality, randomized scalar codebook constructions deviate from this target because of sampling noise, random-start imbalance, update-count fluctuations, component dependence, and finite-capacity effects. We develop a transition-based derandomization framework for dense binary scalar codebooks across two target-similarity families, with similarity decaying exponentially or linearly with level separation. The framework separates the target similarity law, the derandomization variant, and the concrete generator construction, making explicit how initialization, selection, update, and capacity-handling mechanisms shape the induced similarity profile. We formalize derandomization variants that separately constrain initial Hamming weight, update-count variability, and update balance, thereby controlling distinct sources of finite-dimensional error. For each family and variant, we derive the induced mean similarity, identify realization-wise and mean target-matching regimes, and derive exact finite-dimensional expressions for bias, variance, and root-mean-square error. Simulations across dimensions, quantization ranges, reference scalar levels, and generator constructions validate the theory and show how each constraint removes or reduces a specific source of similarity mismatch. The results provide practical guidance for choosing scalar codebook generators that more closely match a desired similarity law under finite-dimensional and hardware-relevant constraints.
Read the full abstract ▾
All authors Dmitri Rachkovskij, Evgeny Osipov, Olexander Volkov, Denis Kleyko, Vaclav Snasel
Artificial Intelligence · Cross-listed from cs.CL
The Backdrop Exposes What the World Around an Agent Costs It
Nusrat Jahan Lia and Shubhashis Roy Dipta
Agent benchmarks test
agents in worlds that stay still. Deployed
agents work in worlds that other people also change. Someone texts the
agent to send the money elsewhere or an order confirmation asks it to reply with a door code. We present BACKDROP, which asks how much of an
agent's capability in a clean world survives in such a world. BACKDROP takes a task along with the
agents execution environment, and plants four everyday hazards in its world, one at a time and all together. The instruction and the correct end state stay the same. Each hazard asks one question. Authority: does a message from another person override the user? Injection: does text planted in a record redirect the
agent? Boundary: does a request pull it into an app it was not given? Fault: after a write fails without saying whether it landed, does the
agent check before it retries? Across 3,678 variants and 16 models, , the average pass rate falls from 69.5% to 31.3% once all four hazards are present; the strongest models fall furthest (Claude Fable 5.1 from 96.6% to 56.0%).
Agents have learned to resist injected text but often follow other unauthorized requests of other people. With all four hazards present, and counting only runs where the planted text reached the
agent,
agents followed another person's message in 46.4% of runs and injected text in 20.3%. The gap is consistent throughout all 16 models. BACKDROP formalizes these gaps and shows how an
agent's score in a task's world is a ceiling on real-world performance.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
Does Gradient Conflict Predict the Understanding–Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Models
Shuyang Jiang, Fucheng Deng, Yuchuan Luo et al. (+1 more)
Unified
multimodal models (UMMs) are increasingly designed around gradient conflict between understanding and generation objectives. The premise that reducing these metrics improves the downstream understanding-generation trade-off has never been tested directly. We audit it in a controlled testbed, GRIDUMM, which mirrors key structural ingredients of UMM training while making the ground-truth trade-off exactly computable. Across 63 configurations and 372 measured checkpoints, no directional conflict metric reaches an absolute Spearman correlation of 0.3 with a confidence interval excluding zero for conflict measured during training against the eventual trade-off. A dose-response intervention that monotonically suppresses conflict leaves the trade-off flat, separating correlation from causation. The norm ratio is a generation-failure detector and becomes null among configurations that master generation. Functional interference measures outperform directional conflict metrics, while training loss tracks the trade-off strongly. Our results do not show that conflict is useless; they show that its validity as a diagnostic target must be established, not assumed, and we release the audit protocol as a reusable standard.
Read the full abstract ▾
All authors Shuyang Jiang, Fucheng Deng, Yuchuan Luo, Zhenyu Wu
Artificial Intelligence
Mohamed Abouzahra
Language agents must revise planned actions when evidence changes, permission is revoked, or a stop instruction arrives. A useful response is selective: suspend affected actions, preserve unaffected work, and resume only after sufficient repair. We introduce NAQD-Env, a synthetic environment that evaluates these decisions against a deterministic reference policy over explicit evidence, authorization, and constraint dependencies. Eleven dependency families support evaluation on development structures, held-out families, and held-out combinations of structures. Metrics distinguish attempted violations from violations permitted by a simulated execution gate and jointly report policy agreement, task value, withdrawal, resumption, and event reporting. We evaluate three open-weight instruction-tuned models from two families under three
prompt conditions on 350 frozen scenarios, yielding 3,150 model-
prompt episodes before gate replay. Across the reported conditions, withdrawal recall is at most 0.06, no valid resumption is observed at eligible opportunities, and only one episode matches the complete reference policy. Under the NAQD
prompt, Qwen2.5-7B has fewer unsafe-attempt episodes than Qwen2.5-3B and Llama-3.1-8B, but also completes less useful work and preserves unaffected actions less accurately. Exploratory supervised fine-tuning probes increase Qwen2.5-3B decision accuracy from 0.45-0.54 to 0.83-0.92; separate diagnostics reveal inappropriate withdrawal after curriculum omissions and a loss of event reporting. These results motivate evaluating selective withdrawal as a distinct component of
agent reliability. The setting measures policy application with trusted structured inputs and does not establish real-world containment or source-verification ability.
Read the full abstract ▾