Artificial Intelligence · Cross-listed from cs.LG
Interactive Distributionally Robust Multi-Agent Learning with General Function Approximation
Debamita Ghosh, George K. Atia and Yue Wang
Model misspecification poses a fundamental challenge in multi-
agent reinforcement learning, where transition uncertainty can be amplified by strategic interactions among
agents. Distributionally robust Markov games (DRMGs) provide a principled framework for addressing such uncertainty, yet existing methods often rely on restrictive assumptions or scale poorly to large state and joint action spaces. We study online learning in general-sum DRMGs with general function approximation and $\phi$-divergence uncertainty sets. We propose RoMEX-$\phi$, a model-free framework that integrates
equilibrium-based exploration with dual fitted learning. Through a functional dual representation of the robust multi-
agent Bellman operator, RoMEX-$\phi$ enables tractable worst-case value estimation from nominal interaction data using a centered empirical robust discrepancy. We introduce the robust Multi-
Agent Decoupling Coefficient (robust MADC) to characterize the intrinsic exploration complexity arising from strategic interactions and
adversarial transition uncertainty. We establish sublinear robust regret guarantees governed by the robust MADC rather than explicitly by the state and joint action space sizes, replacing tabular dependence with intrinsic function-class complexity. Numerical experiments on a scalable general-sum DRMG under total variation uncertainty show that RoMEX-$\phi$ is substantially more resilient to transition shifts than its non-robust counterpart while remaining competitive with an exact tabular robust baseline. Our results provide a scalable framework for distributionally robust multi-
agent reinforcement learning with general function approximation.
Read the full abstract ▾
Artificial Intelligence
Receiver-Conditioned Latent Communication gives 94% CacheBack
Maximillian Rossi, Prajwal Raghunath, Haoqing Xuan et al. (+2 more)
Multi-
agent systems distribute large contexts across
agents that communicate to solve a task. Text messages are compact but require decoding and may omit evidence the receiving
agent needs. Recent latent communication instead transfers KV caches. This avoids text generation and can improve accuracy and latency. However, a full KV cache grows linearly with both the context an individual
agent processes, and the number of
agents that coordinate together. This raises
memory and context costs, often far exceeding available GPU resources and context window sizes. Our key observation is that
agents need only send what the receiving
agent requires for its local task – which we call receiver-conditioned communication. The receiver
agent passes the sender a small description of its information needs, which serves to filter and compress the sender
agent's KV cache. CacheBack is a simple, robust, training-free instance of receiver conditioning based on the sender's
attention weights. On FanOutQA, CacheBack with Qwen 3 removes 75% of the state the
agent would otherwise receive, improving accuracy by 14.7 percentage points and reducing median task-completion latency by 3.2x relative to text communication. We show comparable improvements across model families that span dense
Transformers, Mamba-
attention hybrids, and sliding-window
attention.
Read the full abstract ▾
All authors Maximillian Rossi, Prajwal Raghunath, Haoqing Xuan, Yusen Zhang, Eugene Wu
Artificial Intelligence · Cross-listed from cs.CV
Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence
Souhail Bakkali and Ayoub Merimi
Modern Document AI reads contracts, extracts fields, reasons over tables, and grounds answers to page regions, then forgets everything. Processing an amendment the next day begins from scratch: no schema retained, no contradiction detected, no experience carried forward. This is a structural choice, not a scale failure: current systems are stateless functions. We call this the statelessness bottleneck. This bottleneck lies beyond parameter scaling, context extension, and
retrieval augmentation: storage provides persistence and
retrieval provides access, but neither consolidates observations into knowledge that improves future processing. This survey formalizes persistent evidence-grounded document state as a unifying framework, specifying the operations and invariants required to convert
multimodal evidence into durable, provenance-linked state. We introduce a statefulness audit showing that ten representative
benchmarks, coded against eight statefulness criteria, leave cross-session state
evolution untested, and derive a longitudinal
benchmark harness with five counterfactual metrics: Experience Gain, Cost Efficiency,
Memory Harm, Forgetting Fidelity, Coverage Retention, to characterize the benefit, cost, risk, and governability of persistent document state. Document AI lacks mechanisms coupling persistent state to document-native structure, provenance, and temporal validity. The next era of Document AI will be defined by what systems retain across documents, sessions, and time.
Read the full abstract ▾
Artificial Intelligence
Reasoning Concentrates Errors, and Self-Consistency Never Notices
Asaad Althoubi
Self-consistency assumes that independent samples disagree when a model is unsure, so agreement is evidence of correctness. Holding weights fixed and toggling only a
reasoning mode, over five
benchmarks and 74,944 samples, we show that
reasoning concentrates a model's errors: the probability that two independently drawn wrong answers coincide rises in all ten
dataset-scale comparisons (p = 0.00098), and in nine of nine after restricting both arms to the problems each gets wrong. Where the answer space is unbounded,
reasoning cuts the distinct answers produced to 0.43-0.65 of the non-
reasoning count; where it is bounded, both arms hold an identical option set and
reasoning concentrates mass on it instead, which no positional prior can explain at fixed weights. The aggregate cost is smaller than the mechanism predicts, because
reasoning also shrinks the set of problems where answer diversity can decide anything, in ten of ten
cells and by 2.7x; normalized for available headroom, both arms convert a quarter of it in domain. Confidence weighting does not recover what is left. Across 280 method-
dataset-model combinations on eight models and five
benchmarks, not one beats plain majority voting after correction; weighted voting agrees with it on 98.5% of problem-method pairs and is right 56.3% of the time on the rest; and a signal's direction can invert within fixed weights, with answer log-probability predicting correctness when
reasoning is off and error when it is on. A learned six-signal combination gains nothing out of domain. Confidence signals should be evaluated on decisions, not on discrimination.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CV
Depth Any Seen: Which Surfaces and How Far?
Xiaohao Xu and Xiaonan Huang
When several surfaces are visible along a ray, recovering visible 3D structure from one image requires jointly estimating their presence and metric depth. Depth Any Seen represents these surfaces as image-conditioned multi-Bernoulli depth sets, whose components each contribute one depth or remain absent. Its auxiliary-free Exact Multi-Bernoulli objective (ExactMB) learns depth and presence by marginalizing one-to-one assignments to complete, distinct targets. Our analysis shows that matching expected count can leave component-surface assignment unresolved. We extend real and synthetic layered-depth
benchmarks to evaluate depth accuracy, recovered support, and overprediction. Compared to depth stacking, ExactMB reduces overprediction by a relative 88.2% on LD-Real and 80.5% on MD-3K while retaining most ordinal accuracy, with comparable conditional metric-depth error on LD-Syn. Further ablation studies show that ordered assignment improves depth-accurate recall and precision over marginalization, whereas the count-regularized configuration achieves higher deeper-rank precision than ordered assignment at lower recall. Our code will be publicly released.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CR
SilentCall: Hidden Tool-Call Backdoors in Open-Weight Agents, and How to Catch Them
Bhanu Pallakonda, Mikkel Hindsbo, Sina Ehsani et al. (+1 more)
Open-weight tool-calling
agents are adopted on evidence of merit, usually
benchmark scores and a record of reliable use. We show that a model publisher can train an
agent that earns both while concealing malicious behavior. Fine-tuned on a mixture of clean and poisoned conversations, our
agents answer ordinary requests correctly; once the system date reaches a chosen year, they emit the correct tool call and, alongside it, one that exfiltrates the user's credentials. The exfiltration runs while the user-facing response mentions only the legitimate work. We call this attack SilentCall. Under the trigger, it fires on at least $99.6$% of requests, and no response ever mentions it. The attack is detectable by three distinct methods, which differ mainly in what a defender needs to run them. A runtime monitor that inspects each tool call before it executes requires no access to the model and catches every instance of the payload we tested at a 1.73% false-positive rate. High-temperature probing requires only the published weights. The weight-distribution audit requires training a benign model with the suspect's recipe, placing it within reach of model hubs but not ordinary users.
Alignment benchmarks, by contrast, do not separate poisoned from benign models. SilentCall leaves no trace on standard
benchmarks. As tool-using
agents spread through the open-weight supply chain, trust in them should not rest on what a model says about its own actions. It has to come from inspecting those actions at runtime,
auditing models where they are distributed, and treating tool access as a security surface in its own right.
Read the full abstract ▾
All authors Bhanu Pallakonda, Mikkel Hindsbo, Sina Ehsani, Prag Mishra
Artificial Intelligence
A Benchmark for LLM's Understanding of Middle School and High School Science Topics
Noah L. Schroeder, Yessy Eka Ambarwati, Yuji Zhang et al. (+1 more)
Large
language models (LLMs) are increasingly integrated into educational settings, yet educators lack robust, standards-aligned tools to evaluate their effectiveness in K-12 science contexts. Existing
benchmarks predominantly assess general
language or advanced scientific
reasoning, leaving a critical gap in understanding LLMs' performance on content directly relevant to secondary science curricula. To address this gap, we developed a comprehensive NGSS-aligned
benchmark for both middle and high school science using a rigorous synthetic data pipeline, multi-judge validation, and item-level psychometric analysis. Nine open-weight LLMs were systematically evaluated using this
benchmark, indicating that several smaller, locally deployable models achieved high accuracy across diverse science domains and question types. Our findings indicate that model size did not consistently predict performance, emphasizing the importance of intentional model selection for educational deployment. We then incorporated a human reviewer into the loop, reviewing the items generated by the LLMs for
alignment with NGSS standards. The human review indicated that synthetically generated items were not in perfect
alignment with the NGSS standards, indicating the benefits of human-in-the-loop item development, the need to explore the intersection of content and pedagogical knowledge, and the need to extend
benchmarks to evaluate LLMs' capacity for interactive, evidence-based feedback in educational scenarios.
Read the full abstract ▾
All authors Noah L. Schroeder, Yessy Eka Ambarwati, Yuji Zhang, ChengXiang Zhai
Artificial Intelligence
Decentralized Master-Mind: Joint Action Refinement through Iterative Intent Denoising in Multi-Agent Pathfinding
Valeriy Vyaltsev, Anton Andreychuk, Taisia Zlotnikova et al. (+3 more)
Decentralized multi-
agent path finding (MAPF) with communication requires
agents to reach individual goals without collisions under partial observability. Learnable policies trained on expert data provide an effective approach to this problem. However, when several coordinated joint actions are valid in the same context, independently sampling from per-
agent distributions can recombine locally valid choices into incompatible joint actions. This failure can arise from the final sampling mechanism even when the per-
agent action distributions are learned correctly. DMM (Decentralized Master-Mind) addresses this by replacing one-shot action sampling with discrete, iterative refinement of action intents across communication rounds, inspired by denoising in
diffusion models.
Agents initialize random action intents and refine them through local communication, coupling their choices before commitment. DMM is pretrained with imitation learning on expert MAPF solutions and further optimized with MICPO, a critic-free group-relative reinforcement-learning method designed for multi-
agent, multi-round action refinement. DMM generally achieves higher success rates and lower solution costs than the evaluated learnable baselines. On 1,600 MovingAI tasks, DMM fine-tuned with MICPO solves 1,598, the highest coverage among the evaluated methods, while achieving solution costs close to those of the strongest baselines. DMM also scales to over one million simultaneously acting
agents in obstacle-rich environments. These results show that round-level intent refinement can improve joint-action coordination while preserving decentralized execution.
Read the full abstract ▾
All authors Valeriy Vyaltsev, Anton Andreychuk, Taisia Zlotnikova, Konstantin Yakovlev, Aleksandr Panov, Alexey Skrynnik
Artificial Intelligence · Cross-listed from cs.SD
VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale
Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby et al. (+7 more)
Expressive
speech synthesis has outpaced expressive
speech perception: systems now render fine-grained vocal performances that no public
benchmark can score. Most
benchmarks for this inverse problem stop at six to nine basic
emotion categories, largely on acted
speech. This paper introduces VoiceNet, a human-annotated representation-level
benchmark for voice performance understanding on permissively-licensed in-the-wild
speech. VoiceNet has two subsets: VoiceNet-Emo applies a 40-
emotion taxonomy with three expert ratings per item, and VoiceNet-Ext, a preliminary subset, scores 57 talking-style attributes including speaking rate, vocal tension, breathiness, and register. The paper also releases Emolia, an
emotion-annotated version of the Emilia corpus, with a curated rebalanced subset enriched by dense MOSS-Audio Thinking annotations. Two voice-text contrastive baselines train on this data: a 110M-parameter VoiceCLAP-Small for fast large-scale data filtering and a 7B VoiceCLAP-Large for state-of-the-art performance. Both outperform existing CLAP baselines, which sit near chance on VoiceNet-Emo. On VoiceNet-Emo, VoiceCLAP-Large aligns more closely with the aggregate expert consensus than individual experts agree with one another: a comparison against the majority label rather than evidence of surpassing human
emotion perception. All systems evaluated here are voice-text
embedding models: VoiceNet scores representation-level attribute recognition and
retrieval, not end-to-end spoken-dialogue behaviour. Clustering and filtering uncurated
speech corpora into subsets that span diverse talking styles and
emotions remains an open challenge; VoiceCLAP
embeddings offer a promising tool for this task. VoiceNet, Emolia, and VoiceCLAP are publicly available for research use.
Read the full abstract ▾
All authors Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby, Felix Friedrich, Maurice Kraus, Gijs Wijngaard, Kourosh Nadi, Huu Nguyen, Kristian Kersting, Sören Auer
Artificial Intelligence · Cross-listed from cs.CV
TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception
Quinlan Sykora, Sourav Biswas, Christopher Diehl et al. (+3 more)
We present TriO, a multi-modal unsupervised world model that predicts 4D occupancy, obstacle
segmentation, flow and
LiDAR. In contrast to prior work, TriO utilizes three distinct sensor modalities (camera,
LiDAR, and RADAR) as both inputs and sources of self-supervision, eliminating the need for additional human annotations. Thanks to its novel supervision, the model is able to segment any occupancy from the drivable surface, overcoming the limitations of existing open-set methods in handling long-tail objects. TriO achieves state-of-the-art results in multiple 3D and 4D tasks, including occupancy, flow, and
LiDAR prediction, as well as zero-shot road obstacle
segmentation across multiple
datasets such as Argoverse 2, and Spotting the Unexpected.
Read the full abstract ▾
All authors Quinlan Sykora, Sourav Biswas, Christopher Diehl, Andrew Cunningham, Thomas Gilles, Raquel Urtasun
Artificial Intelligence · Cross-listed from cs.LG
Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias
Aditya Parikh, Eike Petersen, Stella Frank et al. (+3 more)
Encoded demographic information in internal model representations is a commonly assumed risk factor for algorithmic bias, with demographic representation invariance often being touted as the ideal state. However, while demographic shortcut learning is a genuine threat, some degree of encoding is necessary when demographics correlate with target labels. Here, we show, mathematically and empirically, that enforcing demographic invariance can actually hamper bias mitigation and even create new biases. We distinguish marginal from class-conditional representation invariance, and show that they imply the standard group
fairness notions of demographic parity and equalized odds, respectively. We evaluate the effects on predictive performance and
fairness of enforcing both invariance types, both theoretically and empirically across five tabular and two chest X-ray imaging
datasets. Our findings support our mathematical argument that demographic representation invariance is neither desirable nor sufficient for
fairness.
Read the full abstract ▾
All authors Aditya Parikh, Eike Petersen, Stella Frank, Enzo Ferrante, Melanie Ganz, Aasa Feragen
Artificial Intelligence
What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation
Babak Barazandeh
The rank $r$ in LoRA is widely treated as a capacity control: a smaller rank is assumed to yield a simpler model that generalizes better. We show that, under hard per-factor norm budgets—the idealization of the weight decay and norm control used in practice—this intuition breaks down. The reason is structural: under such budgets, the updates LoRA can reach are exactly the
matrices of rank at most $r$ inside a
nuclear-norm ball, and every complexity and displacement functional we analyze is maximized over this set by a rank-one update—so the rank cap never binds. The consequences follow directly. The linear-readout model class we study is identical for every $r \ge 1$, its Rademacher complexity carries no dependence on $r$, and the distance the adaptation can move the source distribution obeys a rank-independent upper bound that we show is sharp. If rank does not control capacity, where does it act? We identify two places. Statistically, replacing the per-factor budgets with a joint budget on the product restores a data-dependent, rank-sensitive complexity bound—though the gain appears only for well-spread feature distributions, and the worst case remains rank-free. Spectrally, rank sets the price of adaptation: canceling the leading singular directions of the pretrained weight requires both sufficient rank and sufficient budget. We bound the smallest rank achieving a desired source–target
alignment, with upper and lower bounds that match under two-sided spectral decay. Together, these results recast rank as governing which updates are reachable and what cancellation costs—not how much capacity the model has.
Read the full abstract ▾
Artificial Intelligence
SenseAgent: An LLM Agent for Adaptive Cross-Domain IMU Sensing
Tianya Zhao, Chuan Liu and Xuyu Wang
Deep learning has improved inertial measurement unit (IMU) sensing for mobile and wearable applications. However, an IMU model trained in one domain often becomes unreliable when it is used with a new user, device, or body position. Existing methods usually treat this problem as a static model-design task: they pretrain a stronger representation, add data augmentation, or select one adaptation method before deployment. In practice, the target domain is only gradually observed, labels are scarce, and different domain shifts require different sensing actions. This paper presents SenseAgent, an LLM-guided sensing
agent for cross-domain IMU activity recognition. Instead of asking an LLM to classify raw IMU signals, SenseAgent uses the LLM as a runtime planner over sensing tools, source-domain experience
memory, online target
memory, and verifiers. The
agent builds a label-free diagnosis report from the target stream and uses it to decide whether to keep raw inference or invoke specialized tools, including
gravity-aware sensing, prototype transfer, and style normalization. Verifiers check source calibration, target-
memory reliability, and no-harm criteria before accepting high-risk tool decisions. SenseAgent also supports scarce feedback without retraining the backbone or replacing the label-free route. This design converts cross-domain IMU sensing from a fixed inference pipeline into a closed-loop sensing process that diagnoses target shifts, selects suitable sensing actions, and rejects unsafe adaptations. We evaluate SenseAgent across multiple IMU
datasets and deployment shifts. Results show that its verified route selection improves cross-domain sensing, especially under harder placement and compound shifts, and further benefits from limited user feedback.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
VC Dimension and Expressivity of Real-Valued Transformers
Gavin Dooley, Andy Yang, Yijia Jessica Zhu et al. (+3 more)
Whereas previous results on abilities and limitations of
transformers have restricted the definition of
transformers in various ways, here we study softmax-
attention, multi-layer
transformers operating on real values, with very few additional assumptions. Applying results from real geometry, we obtain upper bounds on the VC dimension and split VC dimension of such
transformers ($O(n^4)$ and $O(n^6)$, respectively, where $n$ is the input length). Conversely, we also construct specific
transformers witnessing lower bounds on these quantities ($\Omega(n)$ in each case). These results have some notable consequences. For example, within the class of symmetric (permutation-invariant) functions, we show that
transformers can uniformly express all functions over an alphabet of one symbol and non-uniformly express all functions over an alphabet of two symbols, but cannot (even non-uniformly) express some functions over an alphabet of six symbols. We also prove limitations on how many bits of a real number a
transformer can access.
Read the full abstract ▾
All authors Gavin Dooley, Andy Yang, Yijia Jessica Zhu, David Chiang, Peter Cholak, Anand Pillay
Artificial Intelligence
CSI-Agent: LLM-Assisted Few-Shot Adaptation for Cross-Domain Wi-Fi CSI Sensing
Tianya Zhao, Chuan Liu and Xuyu Wang
Wi-Fi channel state information (CSI) has enabled device-free sensing applications such as human activity recognition. However, CSI sensing models remain brittle in cross-domain deployment, where changes in users or environments can produce incorrect predictions. Existing solutions usually treat this problem as an offline model-design problem, by pretraining a stronger representation or applying one fixed adaptation method to the entire target domain. In practice, labeled target data are scarce and different classes may fail in different ways under the same domain shift. To address this, we propose CSI-
Agent, an evidence-seeking LLM
agent that reformulates cross-domain CSI adaptation as a deployment-time decision-making problem. Rather than processing raw CSI or making sample-level predictions, CSI-
Agent summarizes target-domain behavior into sensing-grounded class-level evidence. It establishes a strong target-adaptive default from complementary CSI views and uses an LLM planner to determine whether each class should retain the default or invoke a specialized action. Deterministic verification and bounded execution further reduce unreliable interventions. We evaluate CSI-
Agent on four public
datasets using five cross-domain splits covering device, user, environment, and compositional shifts. Under 1-shot adaptation, CSI-
Agent achieves the best target-domain performance across all splits and improves the average Macro-F1 by about 16% compared to the strongest baseline method.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CL
Communication between Frozen Large Language Models via Prompt Optimization in a Referential Game
Vivek Anand, Muthu Chandrasekaran and Shiva Chaitanya
We study communication between two frozen large
language models from different providers, with different tokenizers, accessed through their API endpoints. The two play a referential game: one sees an object and describes it in a short fixed-length message over a small alphabet; the other must pick that object out of a candidate set. Neither model's weights are updated. Each
agent's
prompt is rewritten by an isolated
prompt optimizer whose reflection model reads that
agent's scored interactions. In the positional setting, optimized
prompts carry a shared code that generalizes to held-out objects above a measured no-codebook baseline, including when the
memory window is removed. In a second setting, independent per-letter blocks no longer fit within the message, although a whole-object place value code does. The base system fails to establish reliable communication: the sender struggles to retain an injective rule, and the receiver has too few confirmed examples in view. A sender collision penalty, retention of successful interactions, and sequential optimization enable successful place value communication in some runs. Outcomes vary across runs and reflection models. In successful runs, the protocol is written into the optimized
prompts, where it can be read and audited directly.
Read the full abstract ▾
Artificial Intelligence
Goal-Persistent Coding Agents as Scientific Performance Engineers: A Fixed-Radius Nearest-Neighbor Case Study
Xiangyang Ju
Coding
agents can pursue persistent objectives across many tool-use turns, but evidence that general-purpose
agents can conduct rigorous scientific performance engineering remains limited. We present a repository-scale case study in which off-the-shelf Codex and Claude Code
agents optimize fixed-radius nearest-neighbor (FRNN) search for particle tracking. Starting from a PyTorch-dependent CUDA implementation, the
agents follow an executable goal that specifies exact-correctness tests, profiling requirements, and acceptance criteria without prescribing code transformations. In the primary sequential trajectory, they autonomously remove the PyTorch dependency and conduct hypothesis-driven optimization experiments. The resulting standalone C++/CUDA library exactly reproduces the targeted reference result. Its synchronous NumPy interface achieved 1.6-fold speedup over the original GPU-resident PyTorch interface, despite including host transfers. Similar speedups were observed across different GPU architectures and software stacks. An independent optimization rerun followed a different sequence of hypotheses and reached even better performance on the target workload. These results show that goal-persistent coding
agents can act as experimental performance engineers, and that executable scientific contracts are needed both to guide and to validate their optimization.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CL
Extraction of clinical findings from mammography and breast ultrasound reports: a comparison between specialists and Artificial Intelligence
Lorenzo Farias, Hanna Reckziegel, Daniela Duarte da Silva Bagatini et al. (+8 more)
Breast cancer is the leading cause of cancer-related death among women in Brazil, and the time between the request and the release of mammography reports directly influences adherence to screening, making the agility in processing these reports a critical factor for early diagnosis. In this context, this study compares the performance of a Large
Language Model (LLM) with manual extraction performed by a team of health researchers in identifying clinical findings from mammography and breast ultrasound reports written in Brazilian Portuguese. Named Entity Recognition (NER) was applied through
Prompt Engineering using a few-shot strategy, employing the Gemini 2.5 Flash model, selected from preliminary exploratory tests with four candidate models. The Gemini 2.5 Flash model demonstrated the best performance, achieving a Macro F1 of 0.91 and a Micro F1 of 0.98. The subjective validation, in which 29 exams of different formats were evaluated by health researchers using a Likert scale, yielded an agreement index of 93.1%. The model outperformed human extraction in overall Macro F1 (0.91 vs. 0.72), as in four reports the model correctly identified information that had been omitted or incorrectly recorded during manual extraction, demonstrating its potential as a complementary verification tool alongside specialists. The results confirm the hypothesis that LLMs, when instructed through
Prompt Engineering, can achieve performance comparable to or superior to manual extraction by health professionals.
Read the full abstract ▾
All authors Lorenzo Farias, Hanna Reckziegel, Daniela Duarte da Silva Bagatini, Daniel Schulz, Gabriela de Andrade Monteiro, Letícia Zanatta, Ana Laura Brill Thum, Priscila Schmidt Lora, Débora Oliveira da Silva, Ana Paula Wernz da Cunha Müller, Cristiane Drebes Pedron
Artificial Intelligence · Cross-listed from cs.NE
SNIP++: Fine-Grained Symbolic-Numerical Alignment for Symbolic Regression
Benjamin Léger, Shubham Gupta, Samy Mammeri et al. (+3 more)
Mathematical expressions and the numerical behavior they produce are two views of the same underlying function, and connecting them is central to scientific discovery. Symbolic Regression (SR) relies on this connection directly: it searches for an expression that reproduces a given behavior. Recent multi-modal models learn this connection by
embedding symbolic expressions and their numerical behavior in a shared representation space. We show that this
embedding space is only globally aligned: complete expressions correspond to complete behaviors, but the contribution of individual parts of an expression is not represented. This granularity gap leaves the model unable to tell how a local edit to an expression changes its behavior, the central operation in SR. We introduce a compositional
alignment method that closes this gap: a structural positional encoding exposes the substructure of an expression to the encoder, and a multi-granularity contrastive objective grounds each subexpression in the behavior it produces before propagating this grounding to the full expression. The resulting representations close much of the modality gap between symbolic and numerical
embeddings, reliably distinguish the effects of local edits that the original
alignment cannot, and transfer to external SR corpora.
Read the full abstract ▾
All authors Benjamin Léger, Shubham Gupta, Samy Mammeri, Kazem Meidani, Cem Subakan, Christian Gagné
Artificial Intelligence
Symbolic Guidance for LLM Agents in Distributed Multiagent Coordination
Ben Rachmut, Ning Zhang, Yevgeniy Vorobeychik et al. (+1 more)
Large
language models (LLMs) are increasingly deployed as autonomous
agents in multi-
agent systems, yet their ability to reliably execute distributed coordination protocols remains poorly understood. While AgentsNet, a
benchmark framework for distributed coordination among LLM
agents, enables such coordination, granting full
reasoning autonomy often leads to inconsistent or degraded performance in complex domains. We hypothesize that coordination can be improved by regulating
agent autonomy through symbolic guidance derived from established
algorithms. To investigate this, we introduce the Symbolic Guidance Taxonomy (SGT), which characterizes a spectrum of autonomy ranging from open-ended natural
language reasoning to fully prescribed algorithmic execution, with intermediate levels providing partial pseudocode guidance. Our results show that intermediate autonomy levels consistently outperform both unguided
agents and fully prescriptive specifications. These findings identify autonomy regulation as a key design principle for LLM-based distributed coordination.
Read the full abstract ▾
All authors Ben Rachmut, Ning Zhang, Yevgeniy Vorobeychik, William Yeoh
Artificial Intelligence · Cross-listed from cs.CV
CaptchaArena: A Large-Scale, Fine-Grained Dataset for Training Computer-Use Agents on Interactive CAPTCHAs
Zhenhao Zhang, Zhaoyu Fan, Haohan Ying et al. (+5 more)
Interactive CAPTCHAs remain challenging for computer-use
agents, while existing
datasets face trade-offs among type coverage, interaction fidelity, and trajectory supervision. To address these gaps, we present CaptchaArena, the first large-scale, fine-grained training
dataset for interactive CAPTCHA solving. It contains 50K puzzles across 20 CAPTCHA types and 5 interaction modes, with every solution verified through execution. CaptchaArena provides 50K screenshot-action trajectories, including 46K with step-by-step
reasoning annotations. It also includes fine-grained pixel-mask annotations for irregular targets. Using CaptchaArena, we train CaptchaAgent, a single 9B policy for all 20 CAPTCHA types, with supervised fine-tuning followed by reinforcement learning. The environment verifier directly provides the RL reward. Supervised fine-tuning reaches 70.5 Pass@1, and reinforcement learning further improves it to 71.7, while also improving performance on two external
benchmarks. These results demonstrate the value of large-scale, fine-grained computer-use supervision for training interactive CAPTCHA
agents. We release CaptchaArena and CaptchaAgent at https://github.com/X0X0X00/CaptchaArena.
Read the full abstract ▾
All authors Zhenhao Zhang, Zhaoyu Fan, Haohan Ying, Jingwen Hu, Hancen Fan, Junhao Zhou, Zitian Chen, Linchao Zhu
Artificial Intelligence
BioDyad: Synchronize Biomedical Discovery and Machine Learning Engineering
Xingbo Du, Fadli Aulawi Al Ghiffari, Leonard Song et al. (+5 more)
Agentic biomedical machine learning (ML) draws on complementary advances in biomedical evidence acquisition and executable program search. Existing systems connect aspects of these capabilities, but coordinating them throughout program search remains challenging. New evidence must guide candidate construction, execution outcomes must inform subsequent discovery and reuse, and validation demands must fit the search budget. We introduce BioDyad, which couples biomedical discovery and ML engineering through two hierarchies within Monte Carlo graph search. Its scientific hierarchy combines prior biomedical guidance with iterative discovery, then links biomedical plans to execution outcomes in
memory for reuse across candidates. Its engineering hierarchy moves candidate programs from smoke execution, through train/validation evaluation, to full-data retraining. We evaluate BioDyad on the 76-task BioXArena
benchmark under a two-hour per-task budget with three matched LLM backends. It achieves the highest penalized all-task score and task success rate among four
agent methods and a one-shot baseline under each backend. These results support coordinating biomedical discovery and ML engineering to integrate external knowledge into executable programs across heterogeneous biomedical tasks.
Read the full abstract ▾
All authors Xingbo Du, Fadli Aulawi Al Ghiffari, Leonard Song, Loka Li, Duzhen Zhang, Zixiao Wang, Xiuying Chen, Le Song
Artificial Intelligence · Cross-listed from cs.SE
VReAct: Verification as an Architectural Layer for LLM Agents: A V-Model Design, and a Pilot Study of Its Deterministic Core
Ali Afoud and Jie JW Wu
Large
language model (LLM)
agents built on the ReAct pattern concentrate four responsibilities in one model: selecting a strategy, choosing each action, formatting it, and judging whether the result is adequate. Nothing outside the generative loop can reject its output, so an
agent that cannot progress does not report failure; it runs until an external budget terminates it. We propose VReAct, which treats verification as an architectural layer for ReAct
agents by adapting the software-engineering V-model: specification levels descend from requirements to individual steps, each paired with a dedicated verifier; a deterministic controller enforces every verdict; and only verification outcomes write
memory. A rejection thus localizes the level that introduced the fault, and an
agent halts by declining rather than by exhaustion. Each verifier separates a zero-cost deterministic gate from an optional LLM judge, so the contribution and cost of each can be measured independently. A pilot study implements the acceptance- and unit-level verifier pairs, comparing five configurations that share one executor, tool set, and scorer and differ only in verification components, on the four-hop stratum of MuSiQue with an 8B-parameter backbone. Across 47 executions, the two unverified configurations answered none of ten questions, every run ending at a step cap or provider
token limit; the verified configuration without a planner answered eight of ten and abstained on the rest. Deterministic gates produced eight of nine observed corrections at zero marginal cost, and planning degraded performance once verification was present. These results characterize termination behavior, not accuracy at scale; we outline a twelve-month plan to complete and evaluate the full architecture, including the integration-level pair the pilot omits.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.HC
Vibe Analysis: Exploring LLM Adoption by Data Visualization Practitioners
Shani C Spivak, Aditi Krishna, Mahsan Nourani et al. (+1 more)
Large
language models (LLMs) are enticing in their promise to support data visualization (Vis) through faster and simpler workflows for data prep, analysis, and visualization creation. Yet LLMs are notoriously error-prone and not built for data visualization tasks. Few studies have explored LLM adoption among Vis practitioners. To fill this gap, we conducted semi-structured interviews with members of the Data Visualization Society, a global community of data visualization designers. Our findings show that Vis designers actively use LLMs for both creative and technical aspects of the visualization process. A new visualization workflow is emerging, a process we call vibe analysis, analogous to vibe coding. Some key challenges raised by participants parallel those of vibe coding, while others are Vis-specific, like gaps in Vis knowledge and chart verification. This work opens up opportunities for research combining LLM-mediated work with Vis tools that incorporate data visualization guidance, constraints, and best practices.
Read the full abstract ▾
All authors Shani C Spivak, Aditi Krishna, Mahsan Nourani, Melanie Tory
Artificial Intelligence · Cross-listed from cs.IR
Overview of the TREC 2025 Million Large Language Models track
Evangelos Kanoulas, Panagiotis Eustratiadis, Jamie Callan et al. (+5 more)
Agentic AI envisions ecosystems of intelligent
agents collaboratively solving complex tasks with minimal human intervention. In such ecosystems, each
agent possesses specialized expertise, making effective expert selection central to overall system performance. While most current approaches assume a small number of well-documented models, real-world expertise is far more diverse and cannot be adequately captured through static metadata or hand-written descriptions. We anticipate a future with millions of specialized
language models (LLMs), each excelling in different domains or problem types. Rather than relying on predefined capability statements, we propose a
retrieval-based paradigm in which an assistant
agent infers expertise dynamically by examining models' observable behavior. Upon receiving a user query, the assistant ranks candidate LLMs based on demonstrated competence, enabling efficient and adaptive expert selection. The TREC Million LLM Track operationalizes this paradigm by shifting the
retrieval target from documents to expert LLMs. Participants are given a discovery set consisting of queries, answers, and log-probabilities from more than one thousand LLMs and are challenged to infer meaningful expertise representations for each model. Given an unseen test query, systems must then rank the LLMs according to their expected performance, providing the first large-scale
benchmark for expertise
retrieval in agentic AI.
Read the full abstract ▾
All authors Evangelos Kanoulas, Panagiotis Eustratiadis, Jamie Callan, Mark Sanderson, Yongkang Li, Jingfen Qiao, Gabrielle Poerwawinata, Vaishali Pal
Artificial Intelligence · Cross-listed from cs.LG
ROTE: Benchmarking Neural Memorization on Complexity-Controlled Symbolic Sequences
Xinye Chen, Stefan Güttel and Mohammad Mozaffari
We introduce ROTE (RollOut Testing of Exact memorization), a benchmarking protocol for evaluating symbolic memorization of neural architectures. We study memorization and the extension of symbolic rules in neural sequence models by using sequences whose complexity is regulated by Lempel–Ziv–Welch (LZW) compression. Under ROTE, each architecture is trained as the same finite-context conditional predictor and is evaluated using teacher-forced one-step prediction as well as closed-loop rollout on the withheld symbols. Following a shared prediction-and-rollout evaluation routine, the
benchmark evaluates gated recurrent, minimal recurrent,
attention-based, and hybrid recurrent-
attention models with their native computational characteristics preserved. Beyond standard predictive metrics, the
benchmark reports normalized string distances, training time,
memory usage, and parameter count across an LZW-complexity sweep. The study establishes a connection between the complexity of algorithmic sequences and the memorization capacity of neural architectures, revealing the trade-offs involving memorization quality, rollout stability, and computational expense. Our software and reproducible experimental code can be obtained from https://github.com/nla-group/rote.
Read the full abstract ▾
Artificial Intelligence
Improving Medical Calculation of LLMs with Embedded Coding
Tianshi Ming, Yingying Zhang and Xian Wu
Large
Language Models (LLMs) perform well on medical examinations and question-answering
benchmarks, but remain unreliable on medical calculation tasks that require exact numerical outputs. These calculations support high-stakes decisions such as medication dosing, organ-function assessment, and prognostic scoring, for which even small errors can have serious clinical consequences. We introduce MedCode, a framework that improves medical calculation by training LLMs to generate embedded executable code. Given a clinical context, the model identifies the relevant calculator, extracts its input variables, and produces a script that delegates arithmetic operations to a deterministic interpreter. Executing the script returns the calculated value together with an explanation and the appropriate unit. We construct supervised fine-tuning (SFT) and preference
datasets from the MedCalc
benchmark and additionally curate a
dataset for calculation tasks in Intensive Care Unit (ICU) scenarios. We further propose weighted Direct Preference Optimization (wDPO), which adaptively emphasizes preference pairs that are difficult for the model to distinguish. Experiments with LLaMA3-8B, Qwen2.5-7B, and Mistral-7B show absolute accuracy gains of 20–30 percentage points, demonstrating the effectiveness of embedded code generation for medical calculation.
Read the full abstract ▾
Artificial Intelligence
EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks
Mukul Singh, Mansi Uniyal, Devin Devlin et al. (+5 more)
Enterprise email
agents must combine information
retrieval, structured state changes, temporal
reasoning, and multi-step coordination. Recent
agent benchmarks include productivity tasks, but few center on typed email workflows in a self-contained environment. We introduce EmailBench, a
benchmark of 206 email and productivity scenarios across 16 task categories. The
benchmark couples a typed email API specification with provider-neutral naming, a deterministic synthetic Enron-inspired corpus, and a scenario suite whose topic selection was informed by aggregate task-intent telemetry from an interactive prototype. Its hybrid evaluation protocol combines 258 executable static assertions with 211 LLM rubrics. We evaluate eight LM configurations on a fixed single-user corpus. The best-performing configuration passes only 33.5% of scenarios despite 99.7% of its tool calls completing without an observed API failure, with pass rates varying substantially across task categories. This gap shows that valid tool execution is not equivalent to task completion. EmailBench provides a self-contained environment for end-to-end email-
agent evaluation, with broader tool coverage, multi-persona testing, and repeated-run evaluation as future work areas.
Read the full abstract ▾
All authors Mukul Singh, Mansi Uniyal, Devin Devlin, Wen Xie, Big Thadawasin, Ritam Dutt, Vivian Lai, Hyeonsu B. Kang
Artificial Intelligence
Choir: An Open Protocol for Distributed Multi-Agent Autoformalization
Yidi Qi and Melanie Weber
AI
agents can now formalize entire textbooks and major
theorems in proof assistants such as Lean, but current efforts are typically centralized: a single team runs all
agents and bears the full computational cost. We introduce Choir, an open protocol for distributed formalization. Choir decomposes a project into tasks that can be completed by independent contributors, each running their own
agent with their own LLM subscription, while coordinating entirely through the project's GitHub repository. To support open participation, every contribution is checked by a deterministic gate before merge. Choir supports Lean 4, Isabelle, and Rocq, and is open source and modular, allowing projects to replace individual components or extend the protocol.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training
Emre Can Acikgoz, Yang Li, Zeyu Leo Liu et al. (+4 more)
Modern LLM post-training composes supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and on-policy
distillation (OPD) into multi-stage pipelines, yet these stages are typically designed and evaluated in isolation. We show that this composition is consequential: a stage that improves the current model can make the next stage less effective. Through controlled experiments with Qwen3 models on math and science
reasoning, we first characterize OPD across nine student-teacher pairs spanning 2x to 53x parameter ratios and show that OPD effectiveness depends on student-teacher compatibility rather than teacher scale alone. The surrounding stages of OPD reshape this compatibility in three ways: (1) A brief SFT warm-up improves subsequent OPD, while an RLVR-strengthened student regresses under
distillation from the same teacher. (2) Adapting the teacher with RLVR raises downstream OPD accuracy in proportion to the capability it adds. Following these two interventions, we find that combining teacher adaptation and student warm-up alone raise average OPD accuracy from 29.2% to 43.8% (50% relative improvement) after the same number of
distillation steps, with additional preparatory training. (3) At comparable accuracy, OPD leaves a stronger initialization for downstream RLVR than SFT, with a gap that widens as RL compute scales. Our results suggest that each post-training stage should be chosen not only for the capability it adds, but for the learning interface it creates for the next stage.
Read the full abstract ▾
All authors Emre Can Acikgoz, Yang Li, Zeyu Leo Liu, Srijan Bansal, Dilek Hakkani-Tür, Shafiq Joty, Semih Yavuz