Artificial Intelligence
AgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agents
Chengquan Guo, Zhaorun Chen, Zinan Lin et al. (+2 more)
LLM
agents are becoming increasingly capable of handling daily workspace tasks by interacting with application environments through tool calls. However, as
agents read and write sensitive user data across email, documents, and other applications, a central
privacy question is whether they preserve contextual integrity (CI), that is, whether the information flows they produce comply with context-specific disclosure constraints. Existing work either focuses on model-level
privacy leakage or evaluates
agents in limited application environments with manually constructed disclosure constraints. In this work, we introduce AgentCIScope, a unified platform and
benchmark for evaluating
agent-level contextual integrity through executable, multi-step tasks across diverse, realistic application environments. AgentCIScope transforms public and synthetic
privacy cases into executable
agent tasks, with disclosure constraints grounded in three criteria: governing
privacy policies, persona-based agreement, and a predefined sensitive-attribute taxonomy. Based on AgentCIScope, we evaluate 12
agents on 936 tasks and find that strong task-completion capability does not guarantee contextual-integrity preservation: single-run leakage rates range from 32.5% to 94.4%, while repeating each task five times further increases leakage by 4-19 percentage points. Leakage varies substantially across attributes: personal preferences and booking details have average leakage rates of 84.7% and 76.5%, compared with 17.1% for credit-card information and 29.4% for SSNs; environments: calendar has the highest average leakage rate as an information source, while WhatsApp ranks highest as an information destination; and governing
privacy policies. AgentCIScope provides deep insights into
agent privacy leakage and serves as a unified testbed for developing
privacy-preserving
agents.
Read the full abstract ▾
All authors Chengquan Guo, Zhaorun Chen, Zinan Lin, Xiaodong Liu, Bo Li
Artificial Intelligence
Compiling Expert Procedures into Verified Programs: When a Small Model Is Enough
Vishvesh Bhat, Asad Aali and Muhammad Ahmed Mohsin
In procedural tool-use domains, how much of the gap between base models is procedure rather than capability? We build an explicit program from a teacher's traces and run it in front of unchanged small models, including an 8B model. The construction, PLVR , takes the
reasoning traces of a teacher, Kimi-K3, clusters their segments to locate recurring operations, has a frontier model author one typed primitive per operation, each run by a small frozen model and assembles the program by A* search over a mined rule library, under a reference-action objective. At inference the program takes one of two forms: for troubleshooting and tool search it produces the chain of thought the student would otherwise generate for itself, and the student acts on it; for multi-turn tool matching the student proposes each action and the program checks, repairs, or retries it before it goes out. On the official tau2-Bench Retail and Telecom test sets and on LiveMCPBench, with three Nemotron base models and the same program for each, the answer depends on the domain. On Telecom, Granite-4.2-8B as the student goes from 43.0% to 91.2% under the program, and every Nemotron student reaches 93.3-94.4%, against at most 58.2% for fine-tuning on the same traces,
prompt optimisation of the student, or two
token-matched baselines. With Granite-4.2-8B as the student, the program is within 2 to 3 points of the Nemotron students on Telecom and within 3 on Retail, and 3 to 13 points below them on LiveMCPBench: where the domain is procedural an 8B student is enough, and on tool search the student still matters. On Retail and LiveMCPBench the program is competitive with the best baseline. The spread between base models shrinks from about 7 points to about 2, averaged over the suites, with no weights changed.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
MA-Bench: Measuring Operational Planning in Language Models
Rodney Lafuente-Mercado, Robert Sincero, Phil Sage et al. (+1 more)
How well can a
language model turn a large, incomplete and partly contradictory body of documents into a plan when no single correct answer exists? We introduce Mission Analysis (MA)-Bench, built on mission analysis, the first step of military operational planning, in which a staff reads the full situation and states the problem before anyone proposes a solution. Its 10 expert-built scenarios pair synthetic packets of ten to twelve documents with 1,257 expert-written pass/fail requirements, each containing information on their source, question type, and importance level. A
reasoning-model judge grades each brief against the requirements without seeing the documents; it grades like a fifth credentialed planner, leaving a four-planner panel's agreement unchanged (Fleiss $\kappa$ 0.42 to 0.41), and a judge from a second provider reproduces the model ranking (Spearman 0.94). Across twenty frontier and open-weights models the best gets 62% of the rubric items right. Models are strongest on facts stated in the documents (58%) and weakest on judgments that must be inferred (33%), and six failures in ten are omissions rather than errors. MA-Bench grades the determinations, not the prose. We release the packets, rubrics,
prompts, scorer and every graded brief with item-level verdicts.
Read the full abstract ▾
All authors Rodney Lafuente-Mercado, Robert Sincero, Phil Sage, Yuan Xue
Artificial Intelligence
Graph-Based Orchestration of Heterogeneous AI Models: A Control Node Approach to Composite Intelligence
G. Michael Youngblood, Filip Dvořák, Slavo Švancár et al. (+3 more)
We present a graph-based framework for Composite AI systems that integrate multiple AI techniques to address complex problems. We describe an architecture using specialized control nodes to orchestrate diverse models including planning, machine learning, constraint programming, and large
language models. The system employs directed acyclic graphs and behavior trees to manage model relationships and execution sequencing. Our methodology encompasses data connectors, computational services, and a containerized deployment framework supporting hierarchical, sequential, and concurrent model composition patterns. Through case studies, we demonstrate how this architecture enables the creation of composite systems that leverage the strengths of multiple AI techniques within a unified operational schema, overcoming the limitations of single-model approaches.
Read the full abstract ▾
All authors G. Michael Youngblood, Filip Dvořák, Slavo Švancár, Tomáš Balyo, Michal Ficek, Martin Dousek
Artificial Intelligence · Cross-listed from cs.CL
CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition
Tianchi Liu, Nicholas Moratelli, Khurram Azeem Hashmi et al. (+5 more)
Feature
distillation is usually expected to add to logit
distillation. We find that this expectation does not hold for two common feature losses in compact multilingual automatic
speech recognition (ASR), and analyze why through their gradients. The gradient of a projection-based loss is orthogonal to the task gradient, while that of Centered Kernel
Alignment (CKA), a similarity-based loss, is weakly aligned with it. CKA lowers the word error rate (WER) on non-English
languages, but at the expense of data-rich English. Three loss-level remedies (one trained, two probed) all fall short, and the CKA gradient overlaps with the subspace English uses in the shared encoder by 97.5% to 99.5% (about 81% by chance). Guided by this diagnosis, we propose CAFE (Confined-Adapter Feature
distillation), which confines the CKA gradient to adapters that English bypasses, addressing the trade-off in the parameters rather than in the loss. On the Open ASR Leaderboard, CAFE lowers the average WER from 8.55% to 8.02%, beats an adapter-only capacity control at 8.51%, and makes English more stable across checkpoints. At 254M parameters, it trails the 0.6B teacher by 2 WER points and is competitive with much larger models across 48 test sets in 24
languages. We use 564k hours of training data and state-of-the-art models to keep the findings relevant to practice.
Read the full abstract ▾
All authors Tianchi Liu, Nicholas Moratelli, Khurram Azeem Hashmi, Mohammadreza Zolfaghari, Rishabh Jain, Louis Lu, Changdae Park, Amril Nazir
Artificial Intelligence
Plant Memory: Technical Language Processing and a Verified Maintenance Ontology for Agent-Accessible Failure Knowledge from SAP PM Work Orders
Akash Raj
Maintenance work orders are the largest written record of how industrial equipment fails, yet in most plants they are close to unusable for analysis. Short texts are abbreviated and misspelt, asset identity is buried in free text, work-type labels contradict the work described, and the fields that matter most for reliability (symptom, root cause, action taken) are usually empty. As a result, diagnostic knowledge stays with senior technicians and leaves when they do. We present Plant
Memory, a system that applies technical
language processing (TLP) to SAP Plant Maintenance (PM) exports and builds an ISO 14224-aligned maintenance ontology in which every extracted statement is a reified fact carrying a confidence score, the evidence it was read from, and a verification state. Extraction is rule-first, with an optional large
language model (LLM) acting as a second reader whose disagreements raise a fact's review priority rather than overwrite it. Technicians verify facts through a review queue, answer targeted knowledge-gap questions where failures repeat without a recorded cause, and approve or reject ontology drift proposals when new terms appear. The resulting graph is served to people through a web application and to AI
agents through a Model Context Protocol (MCP) server, so both query the same verified knowledge and every answer cites the work orders behind it. On a synthetic 12-month export of 400 work orders from a parboiled-rice plant, rule-based extraction reaches 96.2% component accuracy, 88.9% failure-mode accuracy and 100% root-cause precision at 62% coverage; on 28 diagnostic questions, ranked root causes from the ontology are correct at top-1 in 82.1% of cases against 53.6% for the raw SAP root-cause column.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
Imagining Goals with Geometric Horizon Models for Policy Generalization
Daniel Lawson, Dhruv Sreenivas, Adriana Hugessen et al. (+2 more)
Training from offline data has allowed for substantial progress in domains such as robotics, leading to general-purpose policies that can be easily applied zero-shot or efficiently finetuned for downstream tasks. However, these policies can still have poor generalization, both due to the choice of modeling objective and from learning from a static
dataset. In this work, we focus on the challenging task of zero-shot goal generalization, where a policy is evaluated on unseen tasks that require combining its existing knowledge (compositional generalization). An avenue for improving a policy's generalization is by generating new experience with world models; however, such generation has proven difficult for longer horizons. Thus, to alleviate this issue, we propose TD-Aug, which samples from a
geometric horizon model and allows for directly imagining novel outcomes that can be achieved by composing existing knowledge. We demonstrate that training on these future outcomes as goals for goal-conditioned BC policies significantly improves generalization in stitching-based OGBench tasks.
Read the full abstract ▾
All authors Daniel Lawson, Dhruv Sreenivas, Adriana Hugessen, Khimya Khetarpal, Glen Berseth
Artificial Intelligence · Cross-listed from cs.LG
PairIT: Autoregressive Transformers for Low-Data Molecule Optimization
Hannah Lawrence, Bodhi P. Vani, Ji Won Park et al. (+5 more)
Generating novel molecules with desired properties is a central challenge in drug discovery. Although foundational work focused on de novo generation, practical use cases require a more complex setting: conditioning on an initial seed molecule, optimizing multiple properties, and operating in data-sparse regimes. We introduce PairIT, a flexible training framework for autoregressive
transformers over SMILES sequences that enables seeded generation by learning from pairs of structurally similar molecules. By leveraging matched pairs, the model learns to generate local, chemically meaningful edits that improve a property of the seed molecule.
Prompt tokens extend this framework by specifying the target property, desired direction, and approximate magnitude of change. Multiple
prompt tokens can be concatenated to enable multi-property optimization at inference time. For properties where pair supervision alone is insufficient, reinforcement learning post-training can further improve optimization, providing a tunable trade-off between proximity to the seed and the optimization objective. PairIT naturally supports pretraining on a large
dataset and fine-tuning on a smaller
dataset, which is critical for performance in data-sparse regimes. Surprisingly, we observe diminishing returns to the scale of the pretraining
dataset. We demonstrate empirically on MoleculeNet and Therapeutics Data Commons
datasets that, with appropriate paired-data curation, a standard
transformer model of modest size (68M parameters) can achieve robust seeded molecular optimization in low-data regimes.
Read the full abstract ▾
All authors Hannah Lawrence, Bodhi P. Vani, Ji Won Park, Natasa Tagasovska, Samuel Don Stanton, Andrew Martin Watkins, Michael Maser, Ewa M. Nowara
Artificial Intelligence · Cross-listed from cs.RO
Learning in Orbit: Onboard Training and Closed-Loop Attitude Control with a Thermal Segmentation Model on a Spacecraft
Elena Sorina Lupu, Hannah Grauer, Phaedrus Leeds et al. (+9 more)
Deep space missions increasingly require spacecraft capable of autonomously operating in environments with communication delays and bandwidth limitations. A key enabler of such autonomy is the ability to run learning-based
algorithms directly onboard. In this work, we present a flight deployment on a 6U CubeSat that demonstrates (1) onboard training and inference of a thermal segmentation model on an NVIDIA Jetson TX2 NX GPU in space and (2) closed-loop attitude control using the segmentation model on a real-time, thermal image feed to track the Moon. In addition to this demonstration, we implement ROS 2 Foxy as a data transport middleware and integrate an open-source file system that provides strong data integrity guarantees and self-healing features. Over the course of approximately one year, we performed 12 experiments on the spacecraft, advancing the state of the art in autonomous spacecraft systems by showing how modern learning-based pipelines, commercially available hardware, and open-source software can operate reliably under the compute and environmental constraints of Low Earth Orbit (LEO).
Read the full abstract ▾
All authors Elena Sorina Lupu, Hannah Grauer, Phaedrus Leeds, Nikhil Ranganathan, Connor Lee, Sulekha Kishore, Cailyn Smith, Shon Cortes, Brian Hardy, Benjamen Bycroft, Darren W. Rowen, Soon-Jo Chung
Artificial Intelligence · Cross-listed from cs.LG
Woosang Lim, Haneul Yoo, Vincent-Daniel Yun et al. (+1 more)
Structured
pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause
reasoning performance to collapse on long-form
reasoning tasks such as AIME. We propose LayerTrim, a training-free structured
pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving
attention and decoder depth. LayerTrim is the first training-free structured
pruning method designed to preserve long-form
reasoning without gradients or recovery training. Across three
reasoning language models and five
benchmarks, LayerTrim outperforms existing structured
pruning methods in most evaluated settings, improving average accuracy by up to 21.4 percentage points over RESP. On Qwen3-14B, LayerTrim achieves 1.24$\times$ faster prefill and 19.7% lower peak GPU
memory than the dense model, while better preserving
reasoning ability. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy. These results demonstrate that LayerTrim improves
reasoning efficiency through both structural compression and reduced
token generation.
Read the full abstract ▾
All authors Woosang Lim, Haneul Yoo, Vincent-Daniel Yun, Sungjoo Yoo
Artificial Intelligence · Cross-listed from cs.LG
Genesis Steering: Better Activation Steering Through Training Data
Tora Bodin, Eric Aubinais, Ilaria Torre et al. (+1 more)
Activation steering controls large
language models at inference time, but its effectiveness depends on the choice of reference data. We introduce genesis steering, which derives a genesis vector from contrastive examples in the model's own fine-tuning data. When these data are unavailable, we draw on the membership inference literature to identify likely training examples and construct a proxy
dataset. We establish two theoretical results on optimal displacement to the decision boundary and reduced proxy-estimation bias. We evaluate genesis steering and membership-based proxy construction on Tülu-3-8B, whose fine-tuning data are publicly available, and Llama-3.1-8B-Instruct as a control. We find that safety data from Tülu-3-8B's training mixture yield stronger steering than external
datasets on both models. On Tülu-3-8B, membership scores largely agree in identifying training examples, and vectors built from top-ranked examples achieve competitive performance against random subsets of the same size. In contrast, the results on Llama-3.1-8B-Instruct hint that a MIA method will only have a positive effect if the mean direction shift with MIA-filtered in-distribution data is much less severe than the direction shift with OoD data.
Read the full abstract ▾
All authors Tora Bodin, Eric Aubinais, Ilaria Torre, Chih-Hong Cheng
Artificial Intelligence · Cross-listed from cs.CL
From Copying to Composition: How Language Models Develop In-Context Learning
Minglai Yang, Xinyu Guo, Renxiong Wang et al. (+3 more)
In-context learning (ICL) allows
language models to adapt their behavior from instructions and demonstrations in a
prompt, but accuracy alone does not reveal how contextual information controls their predictions. We study the acquisition and trainability of ICL with a controlled diagnostic that separates copying a nearby demonstration from integrating information across the
prompt. Across 17 models, about a third prefer the copied answer, a few are undecided, and the rest prefer the rule-consistent answer, which goes with higher accuracy and with control by the requested value rather than the closest conflicting example. Checkpoint analyses show that models begin prioritizing task-relevant
prompt tokens over conflicting examples long before their accuracy improves beyond chance level. In matched training experiments, synthetic documents that define endings and then query them shift models toward compositional responses on the controlled diagnostic, whereas fine-tuning on definition exercises built from rewritten real documents improves accuracy when
prompts reassign label meanings. The results suggest that ICL is not acquired as a single, uniform capability. Instead,
prompt control, answer preference, construction without candidates, and transfer can develop along distinct trajectories.
Read the full abstract ▾
All authors Minglai Yang, Xinyu Guo, Renxiong Wang, Sunjie Hou, Daniel Yue Zhang, Yunzhong He
Artificial Intelligence · Cross-listed from cs.CV
Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models
Mahdieh Mallahnezhad, Masoud Jafaripour, Mohamadali Tofigh et al. (+2 more)
Multimodal Large
Language Models (MLLMs) have been widely applied in embodied
agents and robotics for
vision tasks. However, MLLMs suffer from limitations in spatial
reasoning and perception. Post-training techniques have shown promise for improving these capabilities; yet, there are limited data-driven surveys of such approaches. In this survey, we consider approaches to fine-tuning pretrained MLLMs for spatial
reasoning tasks, ranging from supervised to reinforcement learning, with an emphasis on the
datasets used. We investigate themes of
dataset construction, data modality and annotation, and evaluation of the resulting models. Based on these themes, we categorize research into three groups:
dataset design,
dataset construction and evaluation. We find four main methods for constructing fixed spatial annotations: rules and templates, LLM prompting, procedural generation, and human-in-the-loop. We also observe substantial variation in evaluation approaches, including different scoring methods, tolerance metrics, and
benchmark types. Finally, we identify opportunities for future research in unified dynamic annotation, efficient spatial
reasoning, standardization of evaluation, expanded spatial range, and human-centered spatial interactions.
Read the full abstract ▾
All authors Mahdieh Mallahnezhad, Masoud Jafaripour, Mohamadali Tofigh, Sheila Schoepp, Osmar R. Zaiane
Artificial Intelligence
GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning
Qingsong Xie, Jiahao Tian, Jiacheng Tang et al. (+5 more)
Constructing robust GUI
agents relies on
multimodal visual grounding: the ability to precisely map natural
language intents to actionable on-screen components. Despite abundant desktop, web, and mobile
benchmarks, high-fidelity resources tailored for diverse environments are surprisingly scarce. To bridge this gap, we introduce the Screen Referring Grounding
Dataset (ScreenRef), a comprehensive GUI grounding
dataset consisting of GUI screenshots paired with natural
language referring expressions and box or point annotations. Spanning desktop, web, and mobile domains, ScreenRef comprises 586K screenshots and over 633K referring expression instances, covering tasks such as text recognition, visual matching, spatial understanding, and refusal scenarios. To boost the model's grounding performance, we propose the
Distillation-to-Reinforcement Learning (D2RL) method to address the limited generalization of supervised fine-tuning and the sparse reward problem in reinforcement learning. Utilizing ScreenRef, we train Qwen3-VL-8B-Instruct via sequential SFT and D2RL training, yielding GroundGUI-8B. GroundGUI-8B achieves 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-
Vision. Among 8B-scale models, it produces substantial gains over models trained on open-source
datasets, with performance comparable or even superior to models trained on closed-source data. Code, models, and
datasets are available at https://github.com/PrentisAI/GroundGUI.
Read the full abstract ▾
All authors Qingsong Xie, Jiahao Tian, Jiacheng Tang, Zhifeng Li, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, Ritankar Das
Artificial Intelligence · Cross-listed from cs.RO
DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation
Zhexi Luo, Zhixuan Xu, Rui Yang et al. (+3 more)
Achieving human-like dexterous manipulation remains a central challenge in robotics. Learning from human hand-object interaction (HOI)
datasets offers a promising pathway to diverse dexterous skills, yet existing physics-based approaches train specialist policies for individual trajectories, thereby limiting skill reuse at scale. We introduce DexMimic, a generalist dexterous hand-object tracking policy jointly trained across diverse HOI trajectories. Efficient joint training requires allocating each rollout to both an undertrained phase and a starting state that supports continued tracking. DexMimic realizes this insight through Value-Guided Reset Allocation, which adapts the initialization distribution online to prioritize difficult phases and select reset states with higher expected tracking returns. We further introduce a
benchmark spanning training, fine-tuning, and generalization. Experiments show improved tracking success and training efficiency, faster fine-tuning, and the potential for zero-shot generalization to unseen trajectories and objects; larger training sets further improve efficiency and generalization. Finally, we validate DexMimic on a real-world dexterous manipulation platform.
Read the full abstract ▾
All authors Zhexi Luo, Zhixuan Xu, Rui Yang, Zeyu Xiao, Yiwen Hou, Lin Shao
Artificial Intelligence · Cross-listed from cs.LG
Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization
Chinmoy Mitra and Tarek Mahmud
Automatic
prompt optimization (APO) uses feedback from model responses to improve
prompts automatically. However, its effectiveness is usually measured through overall scores, which do not reveal which questions become correct, which previously correct answers become wrong, or whether improvements extend to unseen questions. We introduce Reflective Feedback-Driven
Prompt Optimization (RFDPO), a tool that tracks recoveries (incorrect to correct), regressions (correct to incorrect), and churn (all changed outcomes). RFDPO uses this feedback to guide subsequent revisions and records
prompt changes throughout optimization. We evaluate RFDPO across five
benchmarks covering law, knowledge, instruction following, mathematics, and
privacy, using GPT-5 as the optimizer and GPT-4o-mini and GPT-4.1 Mini as solvers. Our results show that overall accuracy can hide substantial changes: in five of nine configurations, churn was at least five times the absolute net change, and a single-item net change concealed up to 17 changed outcomes. Average validation gains were 7.9 percentage points, compared with 5.2 points on held-out tests. In a controlled ablation on LegalBench-Hearsay, item-level feedback reduced churn by 40% ($p=0.024$) while maintaining similar net gains. These findings motivate reporting recovery, regression, churn, and performance on unseen data alongside overall accuracy.
Read the full abstract ▾
Artificial Intelligence
Plan or Retrieve First? A Scale Study of Tool Discovery Architectures
Mingfei Zhang, Amit Surana, Ziauddin Majid et al. (+2 more)
Tool-augmented LLM
agents discover relevant APIs from large registries before planning execution, yet finite context windows prohibit injecting all tool definitions into a single
prompt. The dominant approach, which we term Retrieve-then-Plan (RtP), embeds the user query and retrieves similar tools via approximate nearest-neighbor search. We study an alternative Plan-then-Retrieve (PtR) paradigm in which the LLM first generates Abstract Tool Stubs (hypothesized function signatures describing needed capabilities) which then serve as
retrieval queries against the registry. Evaluating on ToolBench (46,984 unique API endpoints, 50 categories) across single-tool (G1), intra-category multi-tool (G2), and cross-category multi-tool (G3) query subsets, we find a clear complexity-dependent crossover: RtP outperforms PtR on single-tool queries while PtR outperforms RtP on multi-tool queries. This crossover holds at every
retrieval budget (k=1 to 20), is robust across registry sizes from 1k to 47k APIs, and is consistent across model families (Gemini 2.5 Flash and gpt-oss-120b). We characterize a stub-
hallucination failure mode (the LLM invents plausible but non-existent API names) that acts as a double-edged sword: on simple queries PtR's stubs abstract away proper-noun signals that RtP matches directly, yet on complex queries the same mechanism decomposes multi-faceted intent into functional sub-needs. We further analyze
token overhead and show that PtR's extra discovery call roughly doubles inference cost, which pays off on multi-tool queries but hurts on single-tool ones, motivating complexity-adaptive routing as a practical direction, one that can be driven by a zero-cost signal already present in the pipeline (RtP's top-1
retrieval score).
Read the full abstract ▾
All authors Mingfei Zhang, Amit Surana, Ziauddin Majid, Chenxi Wang, Michael Xu
Artificial Intelligence · Cross-listed from cs.IR
Revisiting Generative vs. Dense Retrieval: A Static Value Prior Reverses the Gap
Renbo Tu, Yang Liu and Shuying Sun
Generative
retrieval (GR) has emerged as a strong candidate generator for sequential recommendation as
language models and semantic-item identifiers have improved. However, GR is not a uniformly stronger replacement of dense
retrieval (DR) in online retail candidate generation. We examine their differences on two Amazon Toys
datasets under different metrics. GR leads on best relevant item value among users both methods cover, while DR leads on exact-target recall and relevant value across the ranked list. With deep analysis, we find that popularity contributes the most to GR's measured value lead. Adding the same static item value to DR in one reranking step reverses this lead and keeps recall above GR. Finally, on an industrial-scale internal e-commerce
dataset with LLM-judged relevance, we show that a 14B SID-LLM still underperforms a $47\times$ smaller dense encoder.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
How Inner Optimizers Shape Replica Scaling
Eugene Belilovsky, Stefan Horoi, Edouard Oyallon et al. (+1 more)
Training large
language models (LLMs) across geographically distributed and poorly interconnected compute pools or data centers has become a topic of significant interest. This has led to increased study of local optimization methods such as DiLoco for LLMs, especially those that use local (or inner) adaptive or preconditioned optimizers, which are critical for high-performance LLM training. However, a central challenge remains: it is well known that increasing the number of model replicas (replica scaling) while fixing the total training budget leads to monotonically decreasing performance. Existing work has empirically demonstrated that the choice of inner optimizer (
Muon v.s. AdamW) can significantly improve this trend, but offers only limited explanation of the performance difference. In this work, we consider a range of inner-optimizers and identify two important factors that impact replica scaling: (1) the covariance structure of the pseudo-gradients and (2) the impact of batch size on the pseudogradient's noise scale. Based on these insights, we theoretically and empirically demonstrate that whitening-based pre-conditioning is a powerful inner-optimizer desideratum. Our empirical evaluation extends the results to reveal that whitening-based inner-optimizers achieve strong replica scaling (for 8-96 replicas), with performance relative to DP improving as parameters and data are scaled.
Read the full abstract ▾
All authors Eugene Belilovsky, Stefan Horoi, Edouard Oyallon, Benjamin Thérien
Artificial Intelligence
RLPE: Teaching LLMs to Prompt with Reinforcement Learning
Rahul Singhal, Karime Maamari and Pradyumna Tambwekar
Prompting is a widely used method for adapting general-purpose LLMs to domain-specific tasks. Effective
prompts can be tedious to create manually, so
prompt writing is increasingly handed to LLMs, with recent work training LLMs to write
prompts for other models. These techniques either write
prompts without observing how the task model behaves, blind to the failure modes the
prompt should address, or must run alongside the task LLM at inference time, adding overhead to every query. Neither reflects how
prompt engineering is often practiced. To close this gap, we introduce Reinforcement Learned
Prompt Engineering (RLPE), which trains a model to analyze traces of a separate task model and write a
prompt targeting the failures it observes. RLPE rewards the trained model strictly on
prompt performance, so it learns the prompting strategies that improve the task model. The
prompt is generated once and reused across the task, so inference does not require the trained model. Since RLPE needs only query access to the model it
prompts, it is also compatible with closed-source models. We apply RLPE to Enigmata, BIG-Bench Extra Hard, and PlanBench. We train a 9B-parameter model that writes better
prompts than state-of-the-art frontier models (gpt-5.6-sol and claude-opus-5) and
prompt optimizers (GEPA and TextGrad) on 2/3
benchmarks. Although the model is never explicitly trained to iterate on
prompts, this capability emerges at a level that matches or beats frontier models. The learned prompting capability also generalizes, with the trained model improving task models 6x larger than the one it trained with, including ones from entirely different model families, and matching claude-opus-5 on a fourth
benchmark held out from training.
Read the full abstract ▾
Artificial Intelligence
Smart Alone, Blind Together: Why Accuracy Fails to CertifyTheory of Mind in Multi-Agent Teams
Will Beaumaster and Paul Schrater
Coordinators that act on what they infer about teammates are usually certified by the accuracy of those inferences. We argue that accuracy is the wrong certificate, for two reasons. First, the accuracy needed to act depends on what a wrong action costs, so the bar belongs to the deployment, not the module. Second, a module's errors are rarely independent, and dependence has a sign: positively associated errors help where their damage overlaps and hurt where it compounds, so an accuracy threshold fails in a predictable direction, and once wrong beliefs interact and can do harm, no rule based on accuracy is right for every module. We test these predictions in task assignment, Level-Based Foraging and Hanabi, with injected errors, trained and
Bayesian modules and five
language models, and in human judgments. Accuracy fails in both directions, approving harmful
language-model deployments in Hanabi, where shared errors compound, and rejecting beneficial ones in task assignment, where costly errors are rare. In our tasks it also buys far less redundancy than independence predicts: from GPT-5.4-mini to Claude Opus 5.5, error on teammates the models have evidence about falls from 32% to 11%, but the chance that all three observers misread one falls only from 16% to 9% (independence: 3.4% to 0.15%). We propose certifying the deployment, with a Blind-Spot Gate that prices error costs, pairwise interactions and measured dependence without deploying the module. On held-out data from the
cells it was developed on, it makes no significant false approval in 20 splits (accuracy: 3.2 per split), and in preregistered tests on modules it was not developed on, it recovers what accuracy rejects (seven classifiers: regret 0.004 against 0.479) and catches what accuracy approves (a
Bayesian module: significant false approvals in 17 of 20 splits for accuracy, 1 for the gate).
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
Graph Coarsening Enables Escaping Local Optima in Compute-Constrained Nonconvex MAX-CUT
M. Harper Langston, Pierre-David Letourneau, Dalton Jones et al. (+7 more)
Because its binary optimization landscape contains exponentially many local optima, MAX-CUT is highly sensitive to initialization. We study Particle
Polynomial Relaxation (PPR), a moment-based nonconvex continuous reformulation possessing an advantageous optimization landscape. We warm-start PPR with Combinatorial Multigrid (CMG), a spectral graph-coarsening method that produces structure-aware initial solutions in near-linear time without training data. On t2g20 ($n=400$),
benchmark instance with known exact optimum, the combined CMG+PPR pipeline reaches $97.5\%$ of optimum on average and $98.4\%$ on the best run. An extended variant with discrete local-search refinement reaches the published optimum on all sixty Biq Mac instances evaluated by a recent neural branch-and-bound method. Using a sparse CSR representation, the same approach scales to G-set instances with up to $20{,}000$ vertices reducing
memory requirements by up to $3333\times$ while maintaining a consistent improvement over the CMG warm start. Gains are largest on sparse weighted graphs; on dense graphs we complement the pipeline with spectral sparsification.
Read the full abstract ▾
All authors M. Harper Langston, Pierre-David Letourneau, Dalton Jones, Harris Teague, Peter Scott, Mingu Lee, Richard Lethin, Chris Lott, Roland Memisevic, Fatih Porikli
Artificial Intelligence · Cross-listed from cs.LG
Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models
Shashank Sule, Xinyue Yu, Boonam Shin et al. (+3 more)
Understanding the mechanism behind why
diffusion models memorize their training data remains an open question in generative modeling. In this paper, we examine this question from two perspectives: (a) the model-level perspective concerning the onset of memorization in a model during training, and (b) the sample-level perspective concerning the proclivity of an individual sample to be memorized by a given model. We show that the so-called input loss
curvature, or the Frobenius norm of the Hessian of the
diffusion model's training loss, is an effective metric for quantifying both perspectives on memorization. In particular, we empirically demonstrate that input
curvatures averaged across training and validation samples can identify when a model is likely to start regurgitating its training data when tracked over training time. However, this metric is not sufficient for measuring an individual sample's likelihood of being memorized. To address this perspective, we introduce a counterfactual definition of memorization in
diffusion models, extending the analogous notion from supervised learning proposed by [Feldman, 2019]. We prove that this counterfactual memorization is controlled by the input
curvature of the leave-one-out model, thus establishing a direct theoretical link between a sample's local loss landscape and its generation likelihood. We validate our derived bounds through several empirical experiments and show that input-dependent
curvature provides a proxy for memorization.
Read the full abstract ▾
All authors Shashank Sule, Xinyue Yu, Boonam Shin, Yuxuan Liu, Abolfazl Hashemi, Hayden Schaeffer
Artificial Intelligence · Cross-listed from cs.LG
Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment
Ruomeng Ding, Tianwei Gao, Lianrui Geng et al. (+2 more)
Pluralistic
alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduces annotation costs, but variation in simulation accuracy across individuals can leave some people's preferences poorly represented in training. We study how to allocate human feedback across people to construct training data for pluralistic
alignment, and how this allocation affects the trained policy. To make this allocation explicit and controllable, we define a social welfare objective over individual supervision utilities that measure label quality. Each query's marginal welfare gain depends on its expected supervision improvement, the recipient's current utility, and application-designated priority. Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities. We prove exact optimality of the acquisition
algorithm for the estimated welfare objective under a fixed query budget and model predictions. Experiments on six Roleplay collections and four OpinionQA regions demonstrate improved supervision and controllable allocation tradeoffs. Policy training on four Roleplay collections improves preference prediction for both seen and unseen individuals. When prioritizing a designated group and poorly represented individuals, acquiring human labels for 20% of comparisons recovers 75-86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.
Read the full abstract ▾
All authors Ruomeng Ding, Tianwei Gao, Lianrui Geng, Mohit Bansal, Zhun Deng
Artificial Intelligence · Cross-listed from cs.LG
Scaling Cheap Synthetic Pretraining Improves Indoor Radio-Map Prediction
Khoren Petrosyan, Artashes Mkrtchyan, Rafayel Mkrtchyan et al. (+2 more)
Real measurements are too limited to train an accurate indoor pathloss predictor that works across a wide range of buildings. Large-scale synthetic pretraining offers a way forward, but real building layouts are also scarce, and accurate ray-tracing simulators are slow, expensive, and often proprietary. We address both problems by introducing a generator that automatically creates diverse indoor layouts and a fast, open-source approximate ray tracer that computes
radio maps for them. Using this pipeline, we generate 128M synthetic samples, pretrain ResNet-101 and Swin-T encoder-decoders, and fine-tune them on high-quality
benchmark data. Despite simulator mismatch, our ResNet-101 model reduces error by 15% on average relative to the best known methods across five indoor
radio-map prediction tasks and improves transfer on a small real-measurement
dataset. Both architectures show predictable data-scaling trends: power laws fitted on 2M-16M samples predict the mean normalized downstream RMSE at 32M-128M with 3.33% mean absolute percentage error for ResNet-101 and 7.10% for Swin-T. These results show that pretraining on large amounts of cheap simulated data can improve indoor
radio-map prediction, even when real measurements and high-quality simulations are limited.
Read the full abstract ▾
All authors Khoren Petrosyan, Artashes Mkrtchyan, Rafayel Mkrtchyan, Hrant Khachatrian, Theofanis P. Raptis
Artificial Intelligence · Cross-listed from cs.CR
CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
Bangya Liu, Jianhong Tu, Yueyang Pan et al. (+18 more)
AI
agents are demonstrating increasingly strong offensive and defensive cybersecurity capabilities in the real world. Current
benchmarks mostly evaluate
agents in isolation against static targets. Real cyber operations, however, are dynamic and coupled: security postures constantly change as attackers and defenders adapt to each other in real time. Static
benchmarks quickly saturate and miss an
agent's ability to attack and defend simultaneously under live pressure. To address this gap, we present CyberArena, a dynamic, symmetric
adversarial benchmark built around multi-round attack-and-defense combat. In isolated sandboxes, two
agents compete under identical rules, acting concurrently as both attackers and defenders over continuously changing services. This converts static targets into evolving, semi-open challenges, directly testing an
agent's speed and precision in discovering, exploiting, and patching vulnerabilities. To sustain the
benchmark's longevity, an automated agentic pipeline continuously generates, reviews, and validates new environments. Furthermore, because pairwise contests are notoriously hard to quantify, we introduce a dynamical model that maps match dynamics into unified, multidimensional scores across offensive, defensive, and domain-specific capabilities. Evaluating 12 frontier LLM
agents across 40 challenges in 466 matches, CyberArena separates model capabilities where single-
agent baselines hit a ceiling. The trajectories read like a fight rather than an exam: defenders patch one hole and miss the next and certify repairs on stale evidence, while capable
agents think a move ahead, planting backdoors for later rounds and, in one case, writing their own packet sniffer to watch the opponent. Beyond a
benchmark, CyberArena serves as an adaptable, scalable experimental platform for future research on cybersecurity
agents.
Read the full abstract ▾
All authors Bangya Liu, Jianhong Tu, Yueyang Pan, Hanhong Zhao, Zhe Ye, Hongwei Li, Saastha Vasan, Katherine Lambert, Junmin Zhu, Hao Wang, Jienan Chen, Yunze Xiao, Minglai Yang, Zihan Wang, Hairong Yin, Hao Wang, Zhiwen Fan, Tianneng Shi, Zhun Wang, Chenguang Wang, Dawn Song
Artificial Intelligence · Cross-listed from cs.CL
Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali
Khan Mahi Al Atahar, Nazifa Tasneem and Ajwad Abrar
Abstractive summarization remains challenging across
languages, where linguistic quality and factual consistency may diverge, particularly in low-resource settings. We present a standardized zero-shot
benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets. Our evaluation combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment, providing a comprehensive analysis of both surface-level quality and factual reliability in multilingual summarization. Results reveal substantial cross-lingual variation: GPT-5.6 Luna leads several English lexical metrics but does not maintain the same advantage in Bengali, where LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores. Gemini 2.5 Flash obtains the highest observed Bengali factuality score. Furthermore, evaluation methods produce different model rankings, highlighting the limitations of relying on a single metric for multilingual summarization assessment. These findings demonstrate that strong English summarization performance does not necessarily transfer to Bengali and emphasize the need for comprehensive evaluation frameworks for low-resource
languages.
Read the full abstract ▾
Artificial Intelligence
Discovering Undisclosed Concepts in LoRA Adapters for Diffusion-Based AIGC Models
Yupeng Qiu, Han Fang, Dhananjai Sharma et al. (+1 more)
Low-Rank Adaptation (LoRA) has made
diffusion model personalization affordable and widely shareable, but it also creates moderation risks when harmful or problematic content is not faithfully disclosed. Hence, it is desired to develop a mechanism that, given a LoRA adapter, discovers the concept it was trained to generate, so as to support downstream investigation. We characterize the target concept through the conditioning input that most clearly exposes what the LoRA has learned. Such an input should induce coherent LoRA outputs that are distinctive from those of the base model and strongly align with the learned LoRA subspace. We therefore formulate concept discovery as searching for the conditioning input that jointly maximizes output coherence, LoRA-induced distinctiveness, and subspace
alignment. Even with a clear objective, however, optimizing it over an open-ended input space is prohibitively expensive, since reliable evaluation of coherence and distinctiveness requires repeated
diffusion rendering. Crucially, we observe that an input need not be rendered to determine whether it is promising. Its
alignment with the learned LoRA subspace provides a lightweight signal for localizing promising candidates before visual verification. Based on this insight, we develop a two-stage framework for text-to-image LoRAs. Subspace-Guided Evolutionary Search (SGES) uses render-free subspace
alignment to explore candidate
prompts and retain a compact shortlist, while Paired Visual Identification (PVI) renders only these candidates and compares the LoRA and base-model outputs to identify those exhibiting coherent and distinctive generation behavior. This design enables efficient and reliable discovery of previously unknown concepts encoded in LoRA adapters.
Read the full abstract ▾
All authors Yupeng Qiu, Han Fang, Dhananjai Sharma, Ee-Chien Chang
Artificial Intelligence · Cross-listed from cond-mat.mes-hall
Non-Abelian quantum Hall states at strong Landau level mixing: finding anyons with first-principles AI
Pierre-Antoine Graham, Ahmed Abouelkomsan and Liang Fu
Non-Abelian anyons, strange particles forming the basis of fault-tolerant topological
quantum computing, are predicted to emerge in fractional
quantum Hall states at even-denominator filling factors. Identifying which topological order is realized has proven to be a challenging problem that requires an accurate, non-perturbative treatment of realistic Landau level mixing, which has so far been missing from theoretical studies. Here we present the first complete solution of non-Abelian
quantum Hall problem that includes all electrons and all Landau levels, for
spin-polarized electrons at filling factor $\nu = 3/2$. We develop a momentum-resolved neural-network variational wave function, optimized solely on the microscopic
Hamiltonian, which gives direct access to the topologically degenerate ground states. Without any physics input about the expected outcome, it discovers a degenerate ground-state
manifold whose
entanglement reveals anyons with
quantum dimension $\sqrt{2}$, the hallmark of Ising-type non-Abelian order. By directly computing the overlap with trial wave functions, we further show that Landau level mixing favors the anti-Pfaffian over other candidate states. Our results demonstrate the power of neural-network wave functions in finding and characterizing non-Abelian anyons from first principles, in regimes of strong Landau level mixing, relevant to many recent experiments.
Read the full abstract ▾
Artificial Intelligence
Beyond Equation Search: Conjecture-Guided Discovery with SCOPE and ClosureBench
Yuxiao Hu, Qian Li, Ding Wang et al. (+4 more)
Most symbolic regression methods optimize candidate equations, but improving an equation's fit does not establish whether its underlying scientific
conjecture should be retained, revised, or rejected. A
conjecture may admit multiple mathematical realizations, and similarly accurate equations may imply different explanations. This motivates a shift from refining individual expressions to developing and testing the
conjectures that guide them. We introduce SCOPE, a framework that sustains multiple
conjecture-driven research trajectories. Multiple scientist
agents independently pursue
conjectures through equation modeling, question formulation, and active experimentation, then exchange evidence to revise their
conjectures and research directions. Their findings are consolidated into a scientific report linking models, conclusions, and evidence. Existing equation-discovery
benchmarks primarily assess recovered expressions, leaving the evidence and scientific conclusions established through the investigation underexamined. We therefore introduce ClosureBench, an interactive
benchmark of 60 closure-discovery tasks across five scientific domains. ClosureBench focuses on closure-discovery tasks to give equation discovery a concrete scientific purpose. Each task embeds an unknown algebraic relation in a known PDE system, where
agents control experimental conditions in OpenFOAM to acquire new data for equation discovery. Beyond equation accuracy, ClosureBench proposes three metrics to evaluate the discovery process: evidence faithfulness, mechanism insight, and experimental effectiveness. Compared with diverse baselines, SCOPE achieves higher equation accuracy on both ClosureBench and a real-world scientific discovery case, while producing more accurate, reliable, and effective scientific discoveries.
Read the full abstract ▾
All authors Yuxiao Hu, Qian Li, Ding Wang, Siheng Chen, Linfeng Zhang, Jinyue Yan, Yuntian Chen