Artificial Intelligence
Recognition Is Not Grounding: Benchmarking Multilingual Financial Evidence Extraction
Lingfei Qian, Yan Wang, Xueqing Peng et al. (+5 more)
Comparing financial disclosures across
languages requires identifying shared accounting concepts and recovering all numerical facts that express them. Existing
benchmarks evaluate aspects of these capabilities, but provide limited insight into whether both succeed on the same multilingual evidence. In particular, it remains unclear whether
language models can recognize these concepts across
languages and exhaustively recover their corresponding numerical evidence from the same contexts. We introduce MULTIFINGROUND, a
benchmark designed to investigate this recognition-grounding gap in multilingual financial reports. Constructed from FY2025 disclosures of 90 issuers in English, French, and Greek, MULTIFINGROUND aligns independently authored reports through a shared accounting taxonomy and provides traceable fact-level annotations. The
benchmark evaluates financial fact identification, concept recognition, exhaustive fact selection, and concept discrimination without candidate labels. Across 15 evaluated models, eight API and open-weight reference models achieve 97.3-99.4% concept recognition accuracy, yet miss at least one target fact in 26.6-41.9% of correctly recognized groups. Further analyses reveal substantial variation in concept discrimination and limited gains from the evaluated finance-specialized models. Our findings demonstrate that strong concept recognition can coexist with incomplete evidence recovery, highlighting the importance of evaluating semantic recognition and evidence completeness together.
Read the full abstract ▾
All authors Lingfei Qian, Yan Wang, Xueqing Peng, Polydoros Giannouris, Nadhem Zmandar, Yi Han, Yanguang Liu, Víctor Gutiérrez-Basulto
Artificial Intelligence · Cross-listed from cs.CL
How Do LLMs Understand Legal Philosophy? Diagnosing Legal-Philosophical Reasoning in LLMs
Shuyuan Zheng, Yun-seo Lee, Xing Gao et al. (+7 more)
Legal
reasoning is a fundamental capability for large
language models (LLMs) to complete legal AI tasks such as legal judgment prediction. In legal practice, this
reasoning is shaped by legal philosophy: a judge's view of the nature and sources of law guides how legal materials are interpreted and applied, so judges working from the same statutes and precedents can reach different judgments. Existing research has largely overlooked whether LLMs can understand legal philosophy and apply it in legal
reasoning, an ability we call legal-philosophical
reasoning. To fill this gap, we introduce LePhilBench, a
benchmark for evaluating the legal-philosophical
reasoning of LLMs that comprises two benchmarking tasks and a
dataset. The two tasks evaluate how well LLMs understand legal philosophy and how well they apply it in legal
reasoning, respectively. The
dataset is manually and meticulously constructed. Its test set contains 220 U.S. Supreme Court opinions. A law professor designed fourteen annotation questions on legal philosophy, and three law-trained annotators with formal training in jurisprudence answered all fourteen questions for every opinion, yielding 9,240 legal-philosophical labels. LePhilBench takes the first step toward exploring legal-philosophical
reasoning in LLMs.
Read the full abstract ▾
All authors Shuyuan Zheng, Yun-seo Lee, Xing Gao, Jialu Wu, Jiabao Hu, Katrina Qiyao Wang, Xinbo Lin, Qianying Liu, Chuan Xiao, Tao Huang
Artificial Intelligence
Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach
Jiarui Qin, Junjie Huang, Yunjia Xi et al. (+5 more)
Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into how base models behave in interactive environments. We introduce ASAP (
Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks. Rather than converting base models into chat-style
agents, ASAP preserves the completion interface and adapts existing
agent scaffolds through structured trajectory continuation, short reference prefixes, and minimal recovery mechanisms. Across software-engineering and deep-research
benchmarks, ASAP yields discriminative end-to-end performance across base models and pretraining checkpoints. Beyond aggregate scores, ASAP exposes model-generated execution trajectories, enabling analysis of exploration, editing, testing, recovery, and task completion across training stages. These trajectories provide a behavioral view of how
agent-relevant capabilities change during pretraining and how execution patterns differ after post-training. Our results suggest that end-to-end
agent evaluation can complement conventional pretraining
benchmarks by providing a practical diagnostic of interactive model behavior.
Read the full abstract ▾
All authors Jiarui Qin, Junjie Huang, Yunjia Xi, Di Yin, Yong Yu, Weinan Zhang, Xing Sun, Weiwen Liu
Artificial Intelligence · Cross-listed from cs.CV
VideoHarness-SRSI: Structured Recursive Self-Improvement for Video Understanding Harness
Zihao Lin, Haoyu Dong, Haobo Xu et al. (+10 more)
LLM system capabilities depend not only on model parameters but also on the executable harnesses that organize information acquisition, tool use,
reasoning, and
memory. Recursive harness
evolution can automate their design, but how it can reliably discover generalizable systems from limited, noisy, and repeatedly reused feedback remains unclear. We study this question in agentic
video understanding, where rich
multimodal evidence and coupled mechanisms make harness optimization challenging. We first introduce VideoHarness-BRSI, a recursive self-improvement framework that evolves
video harnesses around a frozen
vision-
language model. Analyzing over 4,000 evolved candidates, we find it competitive with hand-engineered systems but identify three interconnected failure modes: validation overfitting, search-budget dilution, and the early-experience trap. We trace all three to a common root: the same imperfect validation evidence drives both harness revisions and the proposer's search strategy. We therefore propose VideoHarness-SRSI, an evidence-governed framework in which every search decision rests on evidence that is transferable, sufficient, and gathered before the search commits. It restricts the proposer to content-free feedback, informs exploration with instrumented warm-up experiments, and develops mechanisms in isolated probe-race branches before factorial composition and joint refinement. Across four
video understanding
benchmarks, VideoHarness-SRSI improves the equally weighted mean held-out score by 6.2% relative to VideoHarness-BRSI and reduces the mean validation-test gap by 37.9%. Reliable harness
evolution thus depends on how search is organized and informed, not only on how well it optimizes validation performance.
Read the full abstract ▾
All authors Zihao Lin, Haoyu Dong, Haobo Xu, Chi-Chih Chang, Xuying Ning, Yinglong Xia, Shuang Yang, Zhuokai Zhao, Xinying Song, Dongfang Liu, Lifu Huang, Yang Song, Qifan Wang
Artificial Intelligence
Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training
Anshu Raina, Peyman Razaghi, Yuankai Chen et al. (+7 more)
Configuring distributed LLM training across hundreds of GPUs requires jointly optimizing tensor, pipeline, expert, context, and data parallelism alongside micro-batch size, recomputation, sharding, and overlap, a combinatorial space where
memory legality can hinge on a single byte of activation accounting. Today, teams navigate it through expensive multi-node trial and error, burning GPU-hours on crashes and suboptimal recipes. We present Primus Projection, which closes this loop with two coupled components. First, a hybrid projection engine that measures what it can and simulates what it cannot: GPU microbenchmarks on as few as a single device capture compute kernels and any in-scope communication; a sub-node downscale-measure-upscale workflow analytically restores out-of-scope collectives, FSDP overlap, and the full pipeline schedule at the multi-node target. A pure simulation mode runs entirely on CPU, enabling pre-silicon planning without any GPU. Second, the Primus Tuning
Agent treats the projection engine as a fast scoring oracle (seconds per candidate instead of tens of minutes) and uses it to drive an intelligent search over the configuration space. A deterministic seed planner sweeps single axes; an LLM investigation loop proposes cross-axis combinations the planner cannot reach, with an analytic
memory pre-filter that rejects infeasible candidates before any GPU time is spent. Validated on Llama 3.1 (dense) and Mixtral 8x22B (MoE) across two GPU generations, all projections stay within ~10% of measured multi-node throughput, and fidelity holds at production scale: 4.7% error on DeepSeek V3 671B at 1,024 GPUs and 1.9% on Qwen3-235B-A22B at 256 GPUs. In a Mixtral 8x22B case study, the Primus Tuning
Agent discovers a configuration delivering +27% throughput over the published 4-node BF16 reference at the same scale, in under 30 minutes of single-node exploration.
Read the full abstract ▾
All authors Anshu Raina, Peyman Razaghi, Yuankai Chen, Cheng Yao, Yao Fu, Vidushi Goyal, Devang Patel, Wen Xie, Zhenyu Gu, Emad Barsoum
Artificial Intelligence
Chenxu Wang, Xiaoyu Wen, Han Qi et al. (+4 more)
Large
language model (LLM)
agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions.
Adversarial co-
evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on
language-level safety or indirect
prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for
agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct
adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on
agent safety and general capability
benchmarks demonstrate that our method improves
adversarial robustness while maintaining general
agent capabilities.
Read the full abstract ▾
All authors Chenxu Wang, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Cheng Jin, Chaochao Lu, Qiaosheng Zhang
Artificial Intelligence · Cross-listed from cs.RO
From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents
Yue Zhang, Han Lin, Zun Wang et al. (+3 more)
Simulated environments enable scalable training data generation for embodied
agents, but more data does not necessarily provide what a particular
agent needs to learn. Existing environment-generation pipelines largely optimize generic properties such as realism, diversity, or task coverage, while different pretrained policies can fail for different environmental reasons. We therefore introduce WEDGE, a framework for weakness-guided navigation learning in which the failure modes of a pretrained
agent guide the generation of training environments at each adaptation round. Given a pretrained policy, WEDGE uses an LLM to propose executable weakness features from contrasting successful and failed rollouts, automatically validates them on the diagnostic set, and selects the strongest validated weakness based on its success-rate gap. The validated weaknesses are then mapped to the environmental factors that control them. WEDGE varies these factors to construct environments in the difficulty regimes where the policy struggles, producing targeted training experience for adaptation. Across three navigation
agents with different adaptation mechanisms, we find that policies exhibit distinct weakness profiles and benefit from different training environments. Weakness-conditioned adaptation consistently outperforms matched random and hard-example alternatives on ProcTHOR, while the resulting gains for Uni-NaVid and Gemini also extend to HM3D after ProcTHOR adaptation. Iterative re-diagnosis further shows that dominant weaknesses can persist or shift as the policy improves, providing updated guidance for subsequent environment construction. Together, these results show that policy failures are not only diagnostic signals, but actionable guidance for constructing training environments that target a policy's current weaknesses.
Read the full abstract ▾
All authors Yue Zhang, Han Lin, Zun Wang, Yidong Huang, Zaid Khan, Mohit Bansal
Artificial Intelligence · Cross-listed from cs.PL
Recursive Formation and First-Class Reuse of Compositional Capabilities
Hang Lin
We introduce FormCraft, a framework for representing acquired operations as first-class capabilities that can be recursively composed, directly invoked, and selectively retained in a persistent library through domain-specific verification. The framework separates capability well-formedness and public callability from later admission to persistent storage while preserving formation provenance and operational identity. We formalize recursive compositional closure over a uniform capability space and show that formed composites can be reused through the same public interface without re-forming their historical derivation. Experiments across continuous, discrete, propositional, and sequence domains demonstrate instance-independent reuse, independent checking, persistent acquisition, and reduced reuse-time search work under first-class exposure.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CL
Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers
Josh Barua, Kyle Mahowald and William Rudman
Vision-
language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases–e.g., a "blind faith in text"–but why preferences vary across models, domains, and input conditions remains unclear. We develop a predictive account from a
Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases. The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting. Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and
vision dominance under a single combination rule.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.DC
Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Javad Mirzaei and Jeebak Mitra
Large
Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-
token (TTFT), and decode, which determines inter-
token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CL
KoAGE: Do Large Language Models Know, Learn, and Coin Korean Neologisms?
SungHo Kim, Eojin Jeon and SangKeun Lee
Korean neologisms arise through diverse mechanisms, a demanding test of
language models beyond lexical memorization. To enable systematic evaluation, we introduce NeoTrace, an automated, rerunnable framework that collects and dates neologisms across genres. Its 4,271 entries support KoAGE, a cutoff-aware
benchmark with three acquisition and two generation tasks—the latter, to our knowledge, the first to compare model coinages with attested neologisms under their original materials and community context. Across 21 models, lexical recall diverges from contextual inference: in a five-model temporal analysis, models define only 13-30% of cutoff-year words, yet can infer meaning from one usage. However, structural understanding remains limited: even correct definitions barely improve formation analysis of pre-cutoff neologisms. Given word-building materials, single models often follow human formation mechanisms, whereas multi-
agent coinage favors abbreviation and depends on interaction framing. KoAGE thus separates what models know, learn from use, and coin.
Read the full abstract ▾
Artificial Intelligence
SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning
Dingyi Rong, Yue Shi, Chaofan Ma et al. (+5 more)
Long-horizon agentic RL is constrained by the futures visited by the current policy. When rollouts fail to reach useful behaviors, improved credit assignment cannot provide supervision for states and actions that were never encountered. On-policy
distillation (OPD) introduces teacher guidance on policy-induced states, and recent multi-turn variants improve how this guidance is scheduled, replayed, refined, or validated. However, they do not directly use the oracle-controlled future itself as the training trajectory. We introduce State-Rooted Executable Rollout Optimization (SERO), which augments on-policy agentic RL with executable oracle continuations. SERO restores a state actually reached by the current policy and lets a training-only oracle continue interacting with the real environment, making the downstream observations and oracle actions themselves available for learning. It then applies policy-reachability weighting, which weights each oracle step by the frozen behavior policy's compatibility with the preceding oracle actions. This allows deeper supervision to become effective as the policy becomes more compatible with the oracle branch. The oracle loss is optimized separately from the base RL
estimator and no oracle is used at inference time. We evaluate SERO on two challenging interactive-
agent benchmarks, ALFWorld and WebShop, using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. On ALFWorld, learning the full executable continuation is substantially more effective than supervising only the root oracle action, while policy-reachability weighting further improves out-of-distribution performance.
Read the full abstract ▾
All authors Dingyi Rong, Yue Shi, Chaofan Ma, Zongrui Wang, Jiezhang Cao, Yixin Cao, Liang He, Ning Liu
Artificial Intelligence
EpiGuard: An Autonomous Infectious-Disease Early Warning Agent Platform
Tonglong Wei, Zheng Yuan, Lingshuai Lin et al. (+3 more)
Infectious disease outbreaks often spread rapidly before signs of widespread transmission become apparent, leaving a big challenge for rapid response and effective intervention. However, current infectious-disease early-warning systems face three major challenges: fragmented epidemiological evidence across diverse surveillance sources, highly uneven data availability across diseases and geographic regions, and delayed warning procedure caused by manual data collection, analysis, and reporting. To address these challenges, we present EpiGuard, an autonomous, continually self-improving agentic platform for timely and reliable early warning of more than 50 infectious diseases across different regions. EpiGuard proactively monitors and captures newly released official surveillance records, using the latest available evidence for risk assessments. It dynamically adjusts its analytical strategy across three evidence availability regimes, enabling reliable risk assessment under diverse data conditions, ranging from the absence of historical observations to sparse and abundant evidence. Moreover, EpiGuard learns from the successful execution process by preserving effective data-acquisition and analytical procedures as reusable standard operating procedures, enabling it to progressively generalize to previously unseen diseases and regions. We use officially released infectious-disease surveillance data from the China CDC and the United States CDC to evaluate EpiGuard's effectiveness. Experimental results demonstrate that EpiGuard outperforms state-of-the-art forecasting systems, improving forecasting performance by 9.68% and 14.04% on the China CDC and U.S. CDC
datasets, respectively.
Read the full abstract ▾
All authors Tonglong Wei, Zheng Yuan, Lingshuai Lin, Yuhui Yao, Xing Xie, Na Zou
Artificial Intelligence · Cross-listed from cs.CL
ELARSpeech: A 30K-Hour Dataset Across 675 Underrepresented Languages for Long-Tail Multilingual Speech Technologies
Sangmin Lee, Yejin Jeon and David Ifeoluwa Adelani
Massively multilingual
speech modeling has made remarkable progress in recent years, yet existing training resources and
benchmarks still underrepresent much of the linguistic long tail. To address this gap, we introduce ELARSpeech, a large-scale multilingual
speech resource comprising 30,202 hours of
speech across 675 underrepresented
languages, curated from the Endangered
Languages Archive. Combining human-annotated and unlabeled recordings, the corpus spans diverse recording conditions, domains, and
language families. Beyond corpus construction, our experiments show how ELARSpeech supports extending ASR and TTS capabilities to diverse underrepresented
languages, with further benefits from pseudo-labeling unlabeled data. At the same time, our
speech representation analysis reveals remaining limitations in transferring existing multilingual models to the linguistic long tail. Together, ELARSpeech provides both a resource and a testbed for extending multilingual
speech technology further into the linguistic long tail. We publicly release the
dataset, all trained ASR and TTS models, and the complete data construction pipeline to facilitate research on long-tail multilingual
speech technologies.
Read the full abstract ▾
Artificial Intelligence
STRATA: Self-Maintaining Structured Experience Graphs for Self-Evolving Language Agents
Guanhua Zhu, Hanwen Li, Zhi Wei et al. (+2 more)
Large
language model (LLM)
agents can self-evolve at test time by accumulating experience: retrieving relevant
memories before acting and writing new lessons back afterwards. Yet most are single-level: trajectories, distilled workflows and
reasoning items all land in one undifferentiated pool indexed by
embedding similarity, with no typed abstraction hierarchy. They never forget or deduplicate, and are prone to negative transfer, where a superficially similar but directionally opposite experience is recalled and its concrete steps are copied verbatim, actively harming the
agent. We introduce STRATA, a structured, outcome-grounded, self-maintaining
memory that organizes experience into a three-layer graph of atomic skills and code snippets, task workflows and generalized meta-frameworks, connected by typed edges. STRATA maintains itself through five mechanisms sharing one similarity geometry: a multi-dimensional similarity that scores
retrieval by operation and target; a typed reflector crystallizing procedures from successes and pitfalls from failures; community detection that induces meta-frameworks; pressure-triggered garbage collection with articulation-point protection; and contrast edges that flag confusable, direction-opposite experiences. Across four interactive
benchmarks (AppWorld, WebArena, ALFWorld, ScienceWorld), two frozen backbones and seven baselines, STRATA attains the best average score while simultaneously using the lowest latency and fewest environment steps of any
memory-augmented method, and 44% fewer lines of injected
memory than the most context-hungry baseline. We isolate two failure modes of prior
memories: single-level
memory degrades sharply as the task set grows, and single-level concrete
memories suffer catastrophic negative transfer, which contrast edges repair, lifting ScienceWorld from 78.7 to 96.7 within the same system.
Read the full abstract ▾
All authors Guanhua Zhu, Hanwen Li, Zhi Wei, Mengnan Du, Chenxi Yuan
Artificial Intelligence
ixi-sLM: On-Device sLM for AI Call Agent Service
Han-Sang Lee, Kyuho Lee, Juneyoung Park et al. (+10 more)
Running an AI call
agent on-device makes it private, network-independent, and free of per-query cloud cost, but shipping one as a real-world service imposes constraints a cloud deployment never faces: ~1GiB
memory budget, in-call latency and thermal limits, static-integer NPU execution, Korean STT input far from sLM pretraining, and cloud-level quality expectations. We bring ixi-sLM, a 1.2B Korean sLM, to mobile NPUs through ON-DEVICE MODELING: domain-adaptive modeling trims the vocabulary by call frequency to shrink the
embedding and LM head that take nearly half the model under INT4
transformer weights and refits the tokenizer to call
speech (18% fewer
tokens on call traffic), while task-aware quantization calibrates on real calls and assigns per-module mixed precision by activation sensitivity. A single domain-adapted model runs ixi-O's two production services, AI Call Summary and AI Call Answering, at 94% of a state-of-the-art cloud LLM's task-weighted quality; our on-device modeling carries that quality onto the NPU with only a marginal drop in overall score, in an 803MiB model (28% of FP16) at 23% of the CPU's per-call energy. The same recipe deploys across Qualcomm and Apple NPUs and serves several million ixi-O subscribers in production.
Read the full abstract ▾
All authors Han-Sang Lee, Kyuho Lee, Juneyoung Park, Seongbae Lee, Jin Kyu Park, Minjae Kim, Yuri Hong, Jinwoo Lee, Youngwook Kwon, Seongwan Kim, Seungheon Hyeon, Jaeho Lee, Kiduk Kwon
Artificial Intelligence
EvoWeave: Composing Reusable Agent Experience through Operation Level Graphs
Guanhua Zhu, Bo Shen, Lei Zhang et al. (+4 more)
Different tasks often require interactive
agents to perform similar operations at intermediate steps. Retrieving an entire past workflow can introduce irrelevant steps, whereas retrieving isolated operations can discard the dependencies needed to reuse them. We introduce EvoWeave, a framework for experience
memory that represents abstract states as nodes and reusable operations as directed edges with input and output contracts, provenance, and capability references. A planner decomposes the current task into a dependency graph; operation
retrieval and bounded directed search then assemble relevant experience fragments. An execution
agent can inspect these fragments and their referenced capabilities, while reflection separates concrete execution evidence from reusable abstractions. We study the framework across four
benchmarks, five evaluation settings, and three
language model backbones. Across backbones, AppWorld Normal and Challenge task goal completion average 51.91 and 27.99, respectively, exceeding the baseline without
memory by 6.99 and 4.99 percentage points. An AppWorld ablation associates directed connections with an additional improvement of 2.08 points over independent operation cards. The results motivate composition of reusable operations while exposing tradeoffs in
retrieval overhead and conditions for valid reuse.
Read the full abstract ▾
All authors Guanhua Zhu, Bo Shen, Lei Zhang, Fadi Deek, Zhi Wei, Mengnan Du, Chenxi Yuan
Artificial Intelligence
DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agents
Tianle Chen, Pushkar Singh, Ketan Todi et al. (+8 more)
Language-model
agents that run long tasks often redo work they have already done: they re-issue a tool call they made earlier and get back exactly the same observation, a step that adds no new information. We identify this failure as a loop. Loops appear natively in Qwen, Llama, and Gemini
agents and become far more common once the
agent's history is compressed to fit its context window. To reduce these loops, we propose DeLoop, a self-supervised training method. DeLoop builds on a simple observation: the
agent's own rollout can provide the supervision needed to learn from its loops. We use a looping action as a negative example and the first later non-looping action in the same rollout as a positive example. DeLoop learns from these pairs through direct preference optimization and does not require task rewards, gold answers, or LLM judges. On CodeGym under compression, DeLoop reduces the share of Qwen3-8B tasks containing loops from 57.3% to 6.1%. It also uses about one third as many steps per task, with similar task success. On Qwen3-32B, DeLoop reduces loops and steps while raising task success from 37.7% to 44.8%.
Read the full abstract ▾
All authors Tianle Chen, Pushkar Singh, Ketan Todi, Biswajit Paria, Qijia He, Wenqi Wang, Yuwen Tan, Jike Zhong, Sugato Basu, Deepti Ghadiyaram, Arjun Akula
Artificial Intelligence · Cross-listed from cs.CL
Evaluating Compressed Memory: Dependence, Utility, and Content Benefit
Minsu Oh, Seunghui Jwa, Chanjun Park et al. (+1 more)
Does a
language model answer better because its compressed
memory contains useful document information? Replacing
memory vectors with zeros can lower a trained model's score without showing that it outperforms a model trained without
memory. We combine zeroing with separately trained models that cannot access stored text or receive the same constant input for every document. Across nine systems, some models rely on
memory yet perform worse than the trained alternative. For example, in experiments that predict states in generated stories, zeroing
memory worsens a model's prediction loss, yet a model trained without
memory achieves lower loss than the original model with intact
memory. On TriviaQA, giving xRAG
memory from another question's passage raises F1, which measures word overlap with accepted answers, from 19.38 without a document to 24.78. Its effect on how often outputs contain an accepted answer is statistically uncertain. The question's retrieved passage scores higher still (29.88 F1). Thus a gain over no document alone does not establish that the gain requires the question's passage. Evaluations must therefore specify the comparison model, absolute performance, and generation and scoring conditions.
Read the full abstract ▾
All authors Minsu Oh, Seunghui Jwa, Chanjun Park, Yeo-Chan Yoon
Artificial Intelligence · Cross-listed from cs.LG
Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards
Min-Young Yu
Runtime policy verifiers can prevent
agents from executing disallowed actions, but their decisions are typically used only for enforcement and discarded afterward. We ask whether these decisions can instead become verifiable policy rewards for training the
agents they guard. We introduce Policy-Verified Self-Training (PVST), a post-training paradigm in which an
agent generates its own interaction trajectories and an executable policy verifier, here a deterministic gate compiled from a written policy, evaluates whether each action complies. Compliant trajectories are reinforced, and violations become negatives for preference optimization; beyond the gate and the
benchmark's task-success signal, PVST needs no stronger teacher model, no human preference labels and no attack-specific supervision, although rollouts are collected in the
benchmark's attacked environment. On held-out AgentDojo tasks under
prompt injection, policy-verified preference training of Qwen3-8B reduces violating trajectories from 37.2% to 10.3 $\pm$ 0.9% (three seeds) and successful attacks from 12.4% to 0.5% at equal or higher pooled task utility, although utility falls on travel, compared with 22.0% when negatives are selected by an attack oracle and 31.9 $\pm$ 0.9% with negatives that contain no violation. The learned behavior holds under an attack template never used in training (37.7% to 11.0%) and on a suite excluded from training (34.8% to 16.7%), and the retained verifier's intervention rate falls from 39.2% to 8.8% (seed 0). Our analysis identifies two conditions under which the verdict is most informative: success must not already imply compliance, and guarded actions must appear in both permitted and forbidden contexts.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CL
Beyond Corrected Actions: Training LLM Agents on Textual Critiques
Thomas Zeng, Zelei Cheng, Mingyang Zhou et al. (+3 more)
Large
language models (LLMs) are increasingly used for multi-turn agentic tasks that require interactions with users and environments over extended sequences of actions. A common approach to improving LLM
agents is to use more capable expert models to provide training supervision, either by behavior cloning the expert's trajectories or by using the expert to correct the actions in trajectories generated by the weaker student model. However, expert LLMs can provide richer supervision than actions alone: they can directly generate textual critique that explain why specific actions are incorrect. We study whether training on such feedback benefits LLM
agents and propose Critique-Augmented Agentic Fine-Tuning (CAFT), a framework that elicits textual expert feedback on student-generated trajectories. For each incorrect action, the expert yields both a corrected action and a textual critique explaining the student's mistake–which we then use as the training target for fine-tuning. Across four student models from the Nemotron 3 and Gemma 4 families on the agentic domains $\tau^2$-Bench Airline+Retail and WebShop, adding critiques to action-level supervision improves performance by 8% on average over action-only
distillation. Additionally, CAFT checkpoints can also provide a better warm start for reinforcement learning than expert-trajectory
distillation. Our results demonstrate that textual critiques can strengthen supervision for multi-turn LLM
agents.
Read the full abstract ▾
All authors Thomas Zeng, Zelei Cheng, Mingyang Zhou, Sambit Sahu, William Campbell, Genta Indra Winata
Artificial Intelligence
Minghao Guo, Qingyue Jiao, Xiujin Liu et al. (+77 more)
Multimodal models and
agents have emerged to enable AI systems to learn from and interact with the real world more like humans by integrating information across multiple modalities. However, enabling
multimodal agents to continually acquire useful information from complex, heterogeneous
multimodal data remains a significant challenge as they move toward long-horizon tasks and lifelong settings. Addressing this challenge requires effective
memory mechanisms that can maintain continuity across sessions, adapt to changing users and environments, and refine
memory-management policies through accumulated experience. We introduce a two-axis analytical framework for
multimodal agent memory structured around two key questions: how
multimodal information is formed and represented (Axis I, Formation and Representation), and how its management adapts with accumulated experience (Axis II,
Memory Management). Axis I, Formation and Representation, characterizes what information is retained and how it is encoded (e.g., modality-specific
embeddings, textual and structured records, latent and parametric representations) to support subsequent
reasoning and action. Axis II,
Memory Management, distinguishes three tiers: T1 Static
Memory reuses fixed
memory; T2 Dynamic
Memory updates
memory through a stable policy; and T3 Self-Evolving
Memory updates
memory through an evolving policy. Based on this framework, we synthesize recurring design patterns and derive principles for balancing source fidelity with abstraction, maintaining temporal validity and provenance, grounding
memory use in recoverable evidence, and evolving
memory policies over time. Furthermore, we discuss the challenges and potential future directions for the
memory mechanism of
multimodal agents.
Read the full abstract ▾
All authors Minghao Guo, Qingyue Jiao, Xiujin Liu, Jianhong Pang, Ruimeng Yang, Naihao Xu, Liwei Che, Haochen Xue, Ziqi Wang, Xinying Cai, Jiaxuan Luo, Jiaxin Deng, Xuesong Ye, Jintao Qu, Wenyao Gao, Jingwen Gu, Chenyang Zhao, Shichun Liu, Guibin Zhang, Zefang Liu, Siyin Wang, Wujiang Xu, Danrui Li, Zeru Shi, Woongyeong Yeo, Shuang Liu, Wei-Chieh Huang, Weizhi Zhang, Tianyu Yang, Kangsan Kim, Yebowen Hu, Bowen Wen, Bowen Jiang, Haohao Su, Ming Yin, Sheldon Yu, Mian Zhang, Yuyan Chen, Hanrong Zhang, Kai Mei, Zihao Zhang, Mingyu Jin, Xi Zhu, Hengyi Wang, Tianxin Wei, Linyong Nan, Zhenwen Liang, Zhendong Chu, Ziqi Huang, Yongye Su, Sixun Dong, Shujian Liu, Kaiqu Liang, Jiaojiao Han, Zhaoye Fei, Kun Wang, Shihan Dou, Zhichun Guo, Xingjian Diao, Zeqi Gu, Heng Wang, Tao Feng, Cheng Qian, Hamid Palangi, Silei Xu, Xi Yun, Yi Zhu, Shilong Liu, Qingsong Wen, Sung Ju Hwang, Zuxuan Wu, Yu-Gang Jiang, Yiyu Shi, Dimitris N. Metaxas, Jiaxuan You, Julian McAuley, Mengdi Wang, Jiawei Han, Philip S. Yu, Ziyi Ye
Artificial Intelligence
DANGERLENS: Probing Dangerous Representations Hidden by Safety Alignment
Tianzhuo Yang, Shaoheng Yan, Lin Lu et al. (+5 more)
Safety
alignment teaches
language models to refuse harmful requests, yet the internal accessibility of hazardous concepts can retain traces of earlier training. Can checkpoints with equally safe behavior differ in what remains accessible behind their refusals? In this work, we investigate this question through paired training histories and a direct audit of intermediate representations. We introduce DANGERLENS, a probe that maps hidden states into vocabulary space and uses blinded semantic judgments to distinguish harmful goals and operational means from hazard recognition and safety control. Experiments on three Qwen1.5 scales show that checkpoints with prior harmful exposure generally retain greater hazardous concept accessibility after the same safety
alignment, with the strongest separation in intermediate and later layers. The same harmful update also produces different behavioral trajectories from closely matched starting points. Human relevance validation and readout ablations reinforce the measured accessibility contrast. Our findings show that training history remains visible inside behaviorally matched models and motivate internal accessibility audits alongside output safety evaluations.
Read the full abstract ▾
All authors Tianzhuo Yang, Shaoheng Yan, Lin Lu, Jiawei Chen, Kaile Wang, Fanyu Meng, Minglu Liu, Yaodong Yang
Artificial Intelligence · Cross-listed from cs.CL
Tail Competition Explains Scaling in Best-of-N Sampling for Verifiable Problems
Taro Yano, Yoichi Ishibashi and Masafumi Oyamada
Best-of-N sampling is a simple inference-time scaling strategy for large
language models, yet its behavior under incomplete reward models remains poorly understood. We study Best-of-N as a competition between the extreme reward tails of correct and incorrect responses. This view shows that success depends not only on the probability of sampling a correct response, but also on whether correct responses dominate incorrect ones in the upper tail. Using extreme value theory, we characterize three asymptotic regimes: correct-tail-dominant, incorrect-tail-dominant, and critical explaining both monotonic gains and reward-hacking-induced degradation. We further derive finite-sample scaling laws for representative reward distributions, showing that, under Gaussian rewards, incorrect responses with sufficiently large variance can dominate selection even when their mean reward is lower. Unlike oracle selection, convergence with an incomplete selector can be
polynomial rather than exponential. Finally, we propose a lightweight tail-tension criterion to estimate when additional sampling may become harmful. Experiments on synthetic data and real reward model scores are consistent with the theoretically predicted regimes, scaling laws, and diagnostic behavior, suggesting that tail competition provides a unified lens on Best-of-N sampling for verifiable problems.
Read the full abstract ▾
Artificial Intelligence
Steering Vectors for Competing Ethical Frameworks Are Interchangeable: A Competing-Target Test for Activation Steering
Weilun Xu and Zipeng Sun
Activation steering adds a vector to a model's hidden states to control a named behaviour. A gain on that behaviour, however, does not show that the vector controls it directly: it may act through a component shared with neighbouring behaviours, which unsteered and random baselines cannot separate. We introduce the competing-target test: a vector is compared with the vector built the same way for a competing target, and an equivalence test states when their difference is below a useful margin. We apply it in open-weight models to vectors for Kantian deontology, which judges an act by the rule it follows, and act utilitarianism, which judges it by the welfare it produces. On held-out situations, prompting a framework raises its score and lowers the other's, whereas vectors for either framework move both scores alike, and none reaches the margin. The two vectors share most of their direction, a request to apply some framework; the same construction separates an optimistic from a pessimistic tone, whose vectors share much less. What a vector shares with its competitors therefore sets what it controls: steering a perspective can deliver a mixture, and measuring a disposition can measure its neighbours, unless tested against the competing target.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
Katy Blumer, Da-Cheng Juan, Jon Kleinberg et al. (+3 more)
In this paper we study the behavior of
transformer networks generating text over long periods of time. We show that even a three-
token, single-layer
attention-only
transformer can fail to converge. We then study a rich class of
transformers with arbitrary finite depth, layer normalization, MLPs, RoPE encodings, and an arbitrary mix of local and global layers. Every
transformer in this class has a softmax temperature threshold above which its empirical cache distribution converges to a unique limit, independent of initial conditions. We show further that, while each
transformer has a temperature threshold, the supremum of this set is infinite, so there is no global minimal temperature that forces all
transformers to converge.
Read the full abstract ▾
All authors Katy Blumer, Da-Cheng Juan, Jon Kleinberg, Ravi Kumar, Cyrus Rashtchian, Andrew Tomkins
Artificial Intelligence
When Is Smaller Greener? Right-Sizing Language Models for Agentic AI
Sajid Siraj and Maria Barbati
AI
agents use large
language models (LLMs) to plan and act over multiple steps, yet models are usually compared per call rather than per completed task. This paper asks: which model should an
agent use to complete tasks reliably while minimising energy, cost and time? We show that per-call comparisons can be misleading for agentic AI. Because a task succeeds only if all required steps succeed, small differences in per-step reliability can produce large differences in overall task success. Failed attempts also consume resources. We therefore evaluate models using energy, cost and time per successfully completed task, treating the completed task as the functional unit. On this basis, we develop a decision model for choosing LLMs in agentic workflows. The model applies to tasks with binary or graded outcomes and to settings with or without retries. We show that each resource has a crossover task length at which a smaller, less reliable model ceases to be the greener, cheaper or faster option. These crossovers divide tasks into three regimes: for short tasks smaller models are preferred, for long tasks larger models are preferred, and only between these extremes do preferences among objectives matter. We embed these results in a two-stage framework that separates efficiency from preference. A Pareto screen first removes models that are never competitive; a weighted decision is required only when a genuine trade-off remains. A pilot study of 15 open-weight models from five developers, ranging from 0.6B to 27B parameters, on two multi-step
agent tasks with measured GPU energy supports the predicted structure. The model with the lowest energy per completed task changes with task length and task type, and observed crossover points generally match those predicted from per-step measurements. These findings provide a practical framework for right-sizing
language models in agentic AI.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.MA
Why Do Agent Teams Fail Across Boundaries? A Failure Taxonomy for Agent-to-Agent Systems with Private States
Sarath Shekkizhar, Prathyusha Jwalapuram, Zixuan Ke et al. (+12 more)
Agents are increasingly interacting with other
agents that act for different principals (people or organizations on whose behalf they work), such as a buyer's personal
agent negotiating with an enterprise's seller
agent. In such
agent-to-
agent (A2A) scenarios, each
agent has its own environment and holds information and authority that the other cannot see. Moreover, the interests of each
agent may be aligned (cooperative), partly aligned (mixed-motive), or opposed (competitive). Existing multi-
agent system failure studies assume one orchestrator, shared workspace, or single principal. We ask how A2A systems fail when these assumptions are removed. We study more than 2,500 recorded runs of A2A systems built from different
agent harnesses and models, spanning 488 scenarios across eight families (legal, CRM, security, commerce, etc.). From these traces, we derive $A^2$-Failures, a taxonomy of 20 A2A failure modes in four categories: (i) discovery and delegation, (ii) grounding and record consistency, (iii) verification, and (iv) conduct and fidelity. Three findings hold across every harness, model, and scenario studied: (1) discovery and interaction are driven by the phrasing of the task rather than the availability of peer
agents; (2) no
agent verifies a peer's factual claim on its own initiative, so an A2A system inherits the reliability of its least reliable participant unless the protocol or mechanism performs the check; and (3) the aggregate outcome score credits the
agent completing the task while staying blind to the coordination, verification, and conduct failures beneath it, so A2A systems must be evaluated per
agent, against each
agent's own principal. We release the run logs behind the taxonomy at https://huggingface.co/
datasets/a2a-salesforce/a2-failures.
Read the full abstract ▾
All authors Sarath Shekkizhar, Prathyusha Jwalapuram, Zixuan Ke, Vaidehi Patil, Jiayu Wang, Srijan Bansal, Trong Huy Nguyen, Dingfeng Jiang, Dongqi Liu, Ruochen Zhao, Wei Lu, Giuseppe Carenini, Mohit Bansal, Silvio Savarese, Shafiq Joty
Artificial Intelligence
Can We Believe AI Peer Review?
Zongrui Yang, Haoyuan Li, Zhongsheng Wang et al. (+2 more)
As large
language models become widely used in scientific review and manuscript revision, AI scores are increasingly serving as feedback for assessing paper quality and guiding edits. Yet whether a single AI score reliably reflects the reviewer's judgment, and whether revision gains based on that score are endorsed by other reviewers, remain understudied. Using 200 papers from ICLR 2025 and 2026 and approximately 14,000 AI reviews, we study score reliability, score-distribution estimation, and cross-model agreement on revision gains. Repeated reviews fluctuate even under identical conditions. Finer-grained scoring protocols reduce score variance but also systematically shift mean scores, while the gains assigned to manuscript revisions depend strongly on the reviewer.
Read the full abstract ▾
All authors Zongrui Yang, Haoyuan Li, Zhongsheng Wang, Gillian Dobbie, Jiamou Liu
Artificial Intelligence · Cross-listed from cs.CL
LLM-jp-4 Technical Report
LLM-jp: Chaoran Liu, Daisuke Kawahara, Hideyuki Tachibana et al. (+14 more)
This paper presents LLM-jp-4, a family of Japanese sovereign large
language models (LLMs) developed with an emphasis on transparency, reproducibility, and practical usability. LLM-jp-4 comprises an 8B dense model and a 32B-A3B mixture-of-experts model, both trained from scratch on approximately 12 trillion
tokens. To support independent verification and reproduction, we establish explicit criteria for selecting training data, external models, and software, and make the development process publicly available. Evaluation results demonstrate substantial improvements over previous LLM-jp models and competitive performance against open-weight models of similar scale. By documenting the development principles, training procedures, and empirical findings of LLM-jp-4, we aim to provide a reference for future sovereign AI development.
Read the full abstract ▾
All authors LLM-jp: Chaoran Liu, Daisuke Kawahara, Hideyuki Tachibana, Hirokazu Kiyomaru, Jun Suzuki, Koshi Eguchi, Kouta Nakayama, Qianying Liu, Rio Yokota, Ryo Takahashi, Sadao Kurohashi, Sosuke Hosokawa, Taishi Nakamura, Takashi Kodama, Yun-Ang Wu, Yusuke Miyao, Yusuke Oda