Software Engineering · Cross-listed from cs.PL
ASAP: Assembly-Source Aligned Pseudocode Refinement For Binary Decompilation
Yujian Zhuang, Dehong Gao, Qichao Zhang et al. (+4 more)
Large
language models (LLMs) are increasingly used in binary decompilation to refine the C-like pseudocode produced by traditional rule-based decompilers. While this pseudocode is useful, it is a heuristic and lossy abstraction rather than a faithful copy of the source code. It often contains decompiler errors, especially for aggressively optimized binaries where critical low-level details are obscured. We present ASAP, an assembly-source aligned pseudocode refinement framework for binary decompilation. ASAP learns source-aligned assembly representations from paired source and binary functions using joint function-level and snippet-level contrastive
alignment. A Q-Former then compresses chunk-level assembly features into a fixed number of assembly
tokens that condition the decompilation LLM alongside the decompiler-produced pseudocode. During refinement, we use
stochastic pseudocode masking and a relative assembly-advantage loss to reduce the model's tendency to ignore assembly features and rely only on pseudocode refining. On two decompilation
benchmarks across multiple compiler optimization levels, ASAP improves the average re-execution rate from 64.7% to 71.9% and the average recompilation rate from 91.6% to 96.6% compared with the strongest baseline, offering both a new perspective and a practical solution to binary decompilation.
Read the full abstract ▾
All authors Yujian Zhuang, Dehong Gao, Qichao Zhang, Qijing Lai, Jiaxin Wang, Libin Yang, Xiaoyan Cai
Software Engineering
ASAD: Adaptive Software Agents for Debugging
Yacine Majdoub, Eya Ben Charrada and Haifa Touati
The integration of Large
Language Models (LLMs) into multi-
agent systems has shown great potential for automated debugging. Yet nearly all current frameworks rely on rigid, predefined architectures: the number of
agents, their roles, and their interaction patterns are fixed before any analysis of the bug occurs. This one-size-fits-all approach is fundamentally mismatched to the heterogeneous nature of software defects. Simple bugs waste resources on unnecessary coordination, while complex ones suffer from insufficient or poorly aligned expertise. This paper introduces ASAD, an adaptive agentic system for debugging that configures its team according to the nature and complexity of each bug. ASAD initiates the debugging process by analyzing the faulty code and dynamically determines the number of
agents to deploy, the specialized roles they should have, and the collaboration strategy they should follow. A central coordinator orchestrates this process through iterative planning, reflection, and execution; applying fast single-pass repairs for simple issues while assembling purpose-built teams to tackle more complex failures. We evaluate ASAD on three established
benchmarks: Defects4J, DebugBench, and CodeFlaws, using multiple LLMs, such as DeepSeek-V3, Qwen-3 and GPT-5. ASAD consistently improves bug-fix rates by 12–20% over chain-of-thought(CoT) prompting and consistently outperforms static multi-
agent systems by 4–9% in fix precision while reducing average
agent usage by 32%. Crucially, our system dynamically adjusts the number and roles of
agents: it resolves simple bugs with minimal coordination and scales
agent involvement only for more complex cases.
Read the full abstract ▾
Software Engineering
Understanding and Mitigating Library-Related Issues in LLM-Generated Code
Yacine Majdoub, Rinad Hamid, Eya Ben Charrada et al. (+2 more)
Software practitioners increasingly rely on Large
Language Models (LLMs) to generate code that integrates external libraries. However, LLMs often produce incorrect library usage, such as invalid imports, outdated API calls, and hallucinated dependencies, leading to compilation or runtime failures that reduce the reliability of AI-assisted software development. In this paper, we propose an agentic approach to mitigate libraryrelated errors in LLM-generated code. More specifically, we first conduct an exploratory study to characterize the library-related issues produced by LLMs. Our analysis of 100 LLM-generated code files reveals that 84% of generated files contain at least one library-related error, with recurring patterns including incorrect import paths, missing imports, hallucinated libraries, deprecated library usage, and unused imports. Based on these findings, we design an agentic approach that integrates task analysis, documentation grounding, code generation, and automated validation to improve library usage during code synthesis. We evaluate our approach on 300 code generation tasks derived from realworld implementations of rapidly evolving Python frameworks, including LangChain and AutoGen, across five LLMs: GPT-5, DeepSeek-V3, Qwen3, Mistral, and Llama 3. The results show that our approach consistently improves code generation quality across all evaluated models, reducing library-related errors by 38.1% - 54.6% and increasing code correctness by up to 16%.
Read the full abstract ▾
All authors Yacine Majdoub, Rinad Hamid, Eya Ben Charrada, Ahmad Abdellatif, Haifa Touati
Software Engineering
Extending LLM-based support for software engineers with ADHD
Aarsh Shah, Ronnie de Souza Santos, Italo Santos et al. (+2 more)
Software engineering workflows are often not designed to accommodate the needs of developers with
Attention Deficit Hyperactivity Disorder (ADHD), despite known challenges related to task initiation, sustained
attention, and completion. At the same time, large
language models (LLMs) are increasingly integrated into programming tools, but existing systems do not account for neurodiversity or support structured progression across development tasks. In this work, we present Tether 2.0, an LLM based assistant designed to support software engineers with ADHD through workflow oriented interaction across planning, coding, debugging, and review. The tool was named Tether 2.0 because it builds upon the open source code and foundations of the original Tether system. Our approach combines structured interaction modes, activity aware context, and persistent
memory to support task progression and continuity. We evaluate the system through expert feedback and hands on use with a software engineer with ADHD. Our results indicate that Tether 2.0 supports requirement clarification, task decomposition, incremental implementation, debugging, and completion, while enabling users to maintain progress and resume work after interruptions. These findings suggest that LLM based assistants can support direction, sustained progress, and learning in software development when designed with workflow structure and context aware interaction.
Read the full abstract ▾
All authors Aarsh Shah, Ronnie de Souza Santos, Italo Santos, Cleyton Magalhaes, Kiev Gama
Software Engineering
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
Bhanu Prakash Vangala and Tanu Malik
Code generation has emerged as a central capability of large
language models, with coding
agents now able to produce functionally correct software projects from natural
language prompts. However, functional correctness alone does not capture a critical dimension of generation quality: environment specification, defined as the accurate identification of the dependencies required to execute generated code, is equally critical. We develop an
agent protocol for environment specification and introduce a three-layer framework comprising declared, runtime-installed, and necessary-and-sufficient dependencies to systematically assess coding
agents for environment specification. Using this protocol, we evaluate the extent to which coding
agents systematically misspecify software environment dependencies and how this misspecification varies across three
agents, four
languages, and fifty programming tasks. Our results show that current coding
agents exhibit systematic generalization failures along this dimension, producing dependency specifications that are inconsistent, redundant, or incomplete in ways that functional tests do not detect. Across
agents, dependency set agreement is as low as 7% for identical tasks, and newer
agents show no meaningful improvement, suggesting the failure is not resolved by scale or recency. The largest divergence occurs between the declared and runtime dependency layers, implicating environment priors learned from the models' training distributions as the primary driver. Our findings establish environment specification as a distinct, measurable axis of code generation quality that current
benchmarks do not capture, and motivate training objectives and evaluation protocols that jointly optimize for functional correctness and environmental portability.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.PL
Automatically Building Machine-Checked Assurance Cases from C Codebases to Requirements
Haokun Li, Zhongyi Wang, Guanyan Li et al. (+4 more)
Large
language models (LLMs) have shown promise in automating interactive
theorem proving, yet verification of real-world C codebases requires more than discharging individual proof goals. The task involves jointly constructing expressive function specifications and their proofs, and ensuring that library interfaces compose along intended call sequences even without a designated client. This paper presents CCV, an LLM-assisted framework for building machine-checked assurance cases: structured, auditable artifacts supporting the claim that a C codebase meets its intended requirements. To model intended cross-interface use in open libraries, CCV constructs an interface protocol that exposes permitted call sequences and resource assumptions for review, with a conditional safety guarantee under verified contracts and caller obligations. CCV coordinates two complementary phases: (i) requirement-guided analysis and bottom-up construction of candidate specifications and protocols; and (ii) modular proof construction with feedback that revises the specifications and proofs. Implemented using VST in Rocq, CCV verifies
memory safety and leak freedom for all 299 function definitions across six C
benchmarks, including industrial cryptographic components, with less than one person-day of reported human effort per
benchmark. The guarantees depend on disclosed contracts and assumptions; human review supplies the conformance judgments connecting the formal artifacts to the intended requirements.
Read the full abstract ▾
All authors Haokun Li, Zhongyi Wang, Guanyan Li, Xiao Yi, Shengchao Qin, Jianwei Yin, Mingshuai Chen
Software Engineering
Leveraging Game-Based Platform to Teach Code Refactoring: An Experience with Refactoria
Eman Abdullah AlOmar, Christopher Engelbart, Stephen Pachucki et al. (+2 more)
Refactoring is the art of improving the internal structure of the code without altering its external behavior. Because of the topic's significance, several teaching methods and strategies have been proposed in the literature. However, skills in identifying and refac- toring code smells come from training and experience, and a lack of motivation may hinder developers' adoption of refactoring tools. In this paper, we discuss the results of an experiment in the classroom that involved performing various refactoring activities to remove antipatterns using Refactoria, an innovative game-based tool that supports the acquisition of code smell and refactoring concepts. The players play as an expert chef with their sidekick Watson the Whiskbot to refactor Watson's instructions into efficient, readable, and easily maintainable code. We present an experiment with 30 student developers. In particular, we study the perception, effectiveness, and usefulness of gamification for engaging developers in code smell identification and refactoring. Our evaluation indicates a high perceived usefulness among students, and participants reported that Refactoria facilitated their understanding of the refactoring concepts.
Read the full abstract ▾
All authors Eman Abdullah AlOmar, Christopher Engelbart, Stephen Pachucki, Nerissa Lundquist, David Van Hist
Software Engineering
Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agents
Jiangrui Zhao, Chenglong Li, Meng Zhang et al. (+1 more)
Coding
agents solve repository-level tasks through sequences of actions, where a single erroneous action can misdirect subsequent decisions and increase recovery costs. Existing approaches use execution feedback for recovery or specialized checks to block errors, but deciding before execution whether intervention will benefit eventual task completion remains challenging. To address this challenge, we propose HiSentinel, a hindsight-
distillation framework that trains lightweight 0.6B and 1.7B sentinels to select pre-execution interventions aimed at improving task completion rather than correcting every imperfect action. A privileged teacher uses recorded execution outcomes as evidence for intervention judgments, which are distilled into a causal student that receives only the pre-action context and proposed action. Beyond identifying whether and when to intervene, the sentinel must also provide actionable feedback that helps the coding
agent recover or obtain necessary human input. To support these capabilities, we introduce SWE-Intervene, an action-level
dataset constructed from software-engineering trajectories that annotates whether an action should be allowed, autonomously redirected, or paused for human assistance, together with corresponding intervention feedback. Across SWE-bench Verified Mini and Ask or Assume, HiSentinel consistently improves task completion across Sentinel scales and coding-
agent families, with gains of up to 14% and 10%, respectively, while maintaining competitive
token consumption. These results demonstrate that lightweight pre-execution intervention can effectively prevent error propagation and improve the reliability of autonomous coding
agents.
Read the full abstract ▾
All authors Jiangrui Zhao, Chenglong Li, Meng Zhang, Xiaoting Du
Software Engineering
DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation?
Frances Liu, Manny Silva, Paige Calvert et al. (+2 more)
We introduce DoGBENCH (Documentation Generation
Benchmark), to our knowledge, the first
benchmark for generating and maintaining real user-facing software documentation. It asks whether an
agent can produce documentation that experienced technical writers would accept in review. The
benchmark contains 292 items from open source projects, including Helm, PostHog, and Mautic. Each item gives the
agent a pre-change repository and a trigger, such as a code pull request or a reported documentation gap. The
agent must first decide whether the documentation needs an update. For items that need one, the
agent must produce an acceptable patch in one attempt. For items that do not need updates, the
agent must abstain. Task-specific rubrics, validated with project maintainers, score each patch on accuracy, completeness, reader guidance, placement, and repository conventions. The composite score combines patch quality with correct abstention, and a score of 100 means an
agent meets every requirement for the task. Scores should not be interpreted as a percentage of an expert's capability. We evaluated seven
agents. The highest-scoring
agent reached 47.3 out of 100 on the 117-item held-out split. In a separate audit of 1,267 patches, the most common failure modes were task-completion gaps (45.5%), technical inaccuracies (36.6%), and incomplete conceptual or reference coverage (32.5%). Analysis of the corresponding trajectories identified three key patterns associated with these failures: (1) describing interfaces without examining how readers use them (36.0%), (2) missing decisive evidence and filling the gaps with plausible assumptions (33.1%), and (3) stopping after finding the first plausible documentation surface and leaving other affected pages stale (30.1%).
Read the full abstract ▾
All authors Frances Liu, Manny Silva, Paige Calvert, Ayu Adiati, Sarah Sanders
Software Engineering
COMPASS: Predicting the Relationship of Multiple Patches for Vulnerabilities with LLMs
Yi Song, Dongchen Xie, Xiaoyuan Xie et al. (+4 more)
Modern software heavily relies on code reuse, so upstream vulnerability fixes do not automatically propagate to downstream codebases. Downstream maintainers must manually adopt patches to eliminate known risks. In practice, a single vulnerability often corresponds to multiple patches, which greatly complicates downstream patch adoption because different patch relationships imply different adoption strategies. To address this challenge, we first manually inspect large-scale multi-patch vulnerabilities (about 1K) in the real world and interview experienced developers, summarizing six typical types of patch relationships, i.e., Merge, Mirror, Better Solution, Fixing-of-Fixing, Collaboration, and Separation. Based on these observations, we propose COMPASS, an automated approach that predicts the relationships of multiple vulnerability patches with large
language models. Given a CVE as input, COMPASS follows a four-phase pipeline that (i) identifies the patch group and pre-scans explicit relationships, (ii) performs individual patch analysis, (iii) infers relationship instances via a hierarchy-guided
prompt, and (iv) validates completeness and consistency of the inferred results. As output, COMPASS reports the predicted relationships within the patch group and visualizes them as a relationship graph. We evaluate COMPASS on a
benchmark of 300 multi-patch CVEs and compare it against mainstream learning-based and LLM baselines. Results show that our method achieves strong and consistent prediction effectiveness and outperforms SOTA by 85.04% on average. We publicly release an online querying website to support community reuse of patch relationships knowledge: https://patch-relation.com.
Read the full abstract ▾
All authors Yi Song, Dongchen Xie, Xiaoyuan Xie, He Zhang, Lin Xu, Chunying Zhou, Zhi Jin
Software Engineering · Cross-listed from cs.AI
Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents
Serhii Zabolotnii
Benchmarks, audits, and
agent protocols describe performance, permissions, and repair, but not how observed evidence should change an
agent's authority during a consequential task. We call this the assurance-transition gap. We propose a Runtime Assurance Contract (RAC), a policy-level formal schema binding autonomy boundaries, component eligibility, evidence state, transition policy, human-review capacity, and non-compensatory gates. Under RAC, soft metrics may inform routing, whereas a failed or unknown mandatory gate forces retry, switch, escalation, deferral, or stop; aggregate performance cannot authorize action. We define the contract, an evidence record, a permission rule, and five invariants, and illustrate them in clinical, industrial, and judicial failure probes. We then report a deterministic failure-injection study in agentic coding: 280 constructed cases evaluated by a gate conjunction, a score-only rule, and a restricted protocol baseline. At the published example weights and threshold, the score rule admits 80 of 100 block-required injections and all 40 review-required injections. Tuned in hindsight, it matches the conjunction on this corpus. For positive weights, a positive threshold, binary risk signals, zero-signal controls, and an injected case firing each signal alone, we show that exact agreement holds if and only if the threshold does not exceed the smallest weight. A separate set of 18 hand-authored traces checks version-pinned evidence and review transitions against simpler policy variants. In a further prospective synthetic holdout of 24 episodes, two blinded LLM judges assign identical labels to all 72 action attempts; RAC and a separately implemented full stateful baseline both match these labels. These studies test mechanisms on synthetic cases; they establish neither deployed safety nor cross-domain effectiveness.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CR
Aletheia: Permission-Minimality Testing for Coding-Agent Rules
Jieke Shi, Yuchen Chen, Junda He et al. (+2 more)
Coding
agents powered by Large
Language Models (LLMs) read repository rules (e.g.,
AGENTS.md and Cursor rules) to learn project conventions and development procedures. An attacker can exploit the same files to mislead the
agent into requesting excessive permissions or performing unrelated operations, such as accessing credentials or transferring data. We present Aletheia, a framework for permission-minimality testing of coding-
agent rules. It tests whether the permissions requested in these rules are necessary to complete specified tasks. An LLM first translates the rule files into a typed permission
language. Aletheia then removes one permission declaration at a time and synthesizes the corresponding restricted sandbox configuration using a trusted task environment. It executes the unchanged rule and task under both the full and restricted configurations. If both executions produce artifacts that pass the functional tests, Aletheia identifies the removed permission as unnecessary for the tested task and flags the corresponding rule file as suspicious. We evaluate Aletheia on AIShellJack, which provides 314 attack rules containing
prompt-injection payloads and five benign rules, as well as 80 manually verified benign rules from GHAgentFiles. Aletheia detects all 314 attacks while reporting only three false positives among the 85 benign rules (3.5%). On average, the evaluation takes 44.35 seconds and costs US$0.22 per rule.
Read the full abstract ▾
All authors Jieke Shi, Yuchen Chen, Junda He, Yue Liu, David Lo
Software Engineering
Self-Spec Verifiable Code Generation
Jiaru Qian, Yihong Dong, Yongmin Li et al. (+3 more)
Large
language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable guarantees. Recently, researchers have proposed several
benchmarks to evaluate the capabilities of LLMs in generating formally verifiable code, where LLMs need to formulate formal specifications, generate the corresponding code, and verify its correctness. However, existing
benchmarks have two key limitations: (I) They primarily evaluate specification and code generation stage-wise, with code generation typically conditioned on an oracle specification. This setup overlooks whether strong stage-wise performance translates into end-to-end success. (II)They mainly focus on a single proof-oriented
language and mathematically structured tasks, offering limited coverage of tasks common in software development. In this paper, we introduce VeriCodeBench, a
benchmark for self-spec verifiable code generation, where the LLM relies solely on its own generated specification and code throughout the entire process. VeriCodeBench contains 400
language-native problems across C, Java, Rust, and Python, covering practical concerns in software development. We evaluate specification coverage, code validity, and joint problem-level success. We further introduce CodeNova to enhance the capabilities of LLMs in self-spec verifiable code generation. CodeNova makes requirements explicit through constraint-guided specification and uses verifier feedback to guide targeted implementation repairs. Experimental results reveal that self-generated specifications remain a major bottleneck, while providing more sophisticated specifications may not necessarily lead to higher verification success rates. CodeNova substantially improves performance across all evaluation metrics, enabling Claude Sonnet 5 to achieve the strongest results under the self-spec protocol.
Read the full abstract ▾
All authors Jiaru Qian, Yihong Dong, Yongmin Li, Hao Zhu, Bin Gu, Ge Li
Software Engineering
EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses
Zhixuan Tan, Pengjie Gu, Zhao Li et al. (+4 more)
While large
language models have achieved remarkable success in isolated code generation, authentic software engineering requires sustained
reasoning, complex state management, and continuous cross-domain abstraction. However, current evaluations of skill
evolution in autonomous
agents suffer from a critical identifiability problem: they structurally confound genuine capability abstraction with rote solution leakage (i.e., copying highly similar code from historical training data). To resolve this, we introduce EngramBench, a rigorous, capability-grounded
benchmark governed by the strict axiom of capability overlap without solution overlap. Comprising 30 diverse learning tasks and 13 unseen transfer tasks, EngramBench challenges
agents to navigate interactive, multi-hour development cycles driven by LLM-simulated users. Our extensive evaluation across 48 multi-hour execution trajectories – corroborated by human-expert validation – reveals a profound insight into procedural
memory. We demonstrate that static skill banks do not magically bypass the "last mile" of exact code implementation, which remains bottlenecked by the base model's inherent
reasoning limits. However, they serve as an indispensable execution compass. By navigating
agents away from catastrophic,
token-heavy trial-and-error, genuine capability abstraction slashes redundant context bloat and reduces overall coding time by over 55%. Ultimately, EngramBench shifts the evaluation paradigm from trivial pattern matching to the verifiable measurement of deep, cross-domain capability transfer.
Read the full abstract ▾
All authors Zhixuan Tan, Pengjie Gu, Zhao Li, Yihan Hu, Xu He, Dong Li, Jianye Hao
Software Engineering
Trustworthy Runtime Error Healing in Real-World Repositories: A Benchmark and Guardrail
Gou Tan, Pengfei Chen, Zhensu Sun et al. (+9 more)
Runtime error healing lets a crashed program continue by generating code that repairs its live runtime state. Recent work shows that LLMs can generate such healing code, but it is evaluated only on small competition programs, and executing LLM-generated code inside a live process raises safety concerns that remain unaddressed. In this paper, we take LLM-based runtime healing toward practical use in real-world repositories. We first build HealBench, a
benchmark of 265 runtime errors from 18 real-world repositories, each paired with a reference execution on the patched version. HealBench also provides a unified framework that lets LLM
agents heal with cross-file context and live runtime state. We then design HealGuard, which requires healing code to be written in HealCore, an analyzable subset of Python, and uses static and dynamic taint analysis to check whether state changed by healing reaches operations protected by developers. We evaluate a dedicated healing method and three general coding
agents with three backbone LLMs. The best setting resumes execution in 38.11% of instances and passes the target test in 28.68%, showing that existing
agents can already heal a meaningful share of real repository-level crashes. However, among executions that pass, HealGuard flags 17.4% whose healing-changed state may reach a protected operation. On 684 controlled cases, HealGuard detects all unsafe cases, at the cost of a 68.42% false positive rate.
Read the full abstract ▾
All authors Gou Tan, Pengfei Chen, Zhensu Sun, Jieke Shi, Junkai Chen, Ting Zhang, Weifeng Sun, Junda He, Shuai Liang, Chuanfu Zhang, Lwin Khin Shar, David Lo
Software Engineering
From Verification Failures to Reusable Guidance for Coding Agents
Yuqing Zhai, Xiaohong Chen, Lingming Zhang et al. (+2 more)
Coding
agents need to establish that a program satisfies a specification and that the specification captures the requested behavior. We study how expert diagnosis of verification failures can become reusable guidance for this work. Our approach combines executable
language definitions in the K framework with a kit of procedures for constructing specifications, repairing proofs, and auditing their adequacy. A human-guided development campaign on HumanEval, a
benchmark of 164 Python programming tasks, achieves a 164/164 success rate with the semantics and the kit, measured by final AI audit Pass verdicts after two targeted repairs. To examine whether auditing detects problems that successful proofs leave unresolved, we construct 12 author-reviewed pairs of clean and defective packages. Every package passes its K proofs, and completed audits identify all defects and accept all clean packages. We then use KleverBench to test specification and proof construction for 31 programs with changed operator meanings. Comparisons with complete acceptance rules and equally long generic advice yield mixed results across two model and budget settings, motivating further work on selecting useful guidance within resource limits. Human-reviewed Optimism proofs establish expected pause reverts for six operations within declared input bounds under London semantics with unbounded gas. We report progress, difficulties, and lessons toward
agents that deliver programs with checkable correctness arguments.
Read the full abstract ▾
All authors Yuqing Zhai, Xiaohong Chen, Lingming Zhang, Sriram Vishwanath, Grigore Rosu
Software Engineering · Cross-listed from cs.CR
Approval Laundering: Systematizing Approval–Execution Binding Failures in AI Coding-Agent Harnesses
Yang Wang
Modern AI coding-
agent harnesses (Claude Code, Codex CLI, Cursor) rest their security boundary on a largely unexamined assumption: that the action A a human approves is the same action A' the harness executes, where A is fixed by a stated policy for what a scope grant or session-scoped approval authorizes. We show this assumption fails systematically and reproducibly. We introduce Approval Laundering, a taxonomy of six failure modes by which a harness's enforcement mechanism silently substitutes A' for A after approval: Scope, Argument, Temporal, Tool, Delegation, and Semantic laundering. Unlike prior work that evaluates risk classifiers against static corpora or infers implicit authorization boundaries, we study credential-binding integrity: given an already-approved action, does the harness dispatch exactly that action? Instrumenting Claude Code's pre-execution mediation point (PreToolUse), we conduct a controlled, headless, repeated-measures study of all six classes (N=19-20 runs each), reporting a Bound-Gap Rate (BGR) with Wilson confidence intervals and inter-rater agreement (kappa=1.0). We prototype Approval
Token, a keyed capability Hk(principal,
agent_id, session_id, tool, arguments, scope, expiry) issued by a mediator that never returns the key to the
agent, evaluated via paired before/after replay of 118 runs (McNemar's exact test). The
token fully eliminates Delegation laundering and, for our seeded session-identity-mismatch construction, Temporal laundering (p<10^-5), but by design leaves Scope laundering unaffected and shows no significant reduction in Argument laundering (p=1): an honest negative result, since these two classes leave every recorded dispatch field unchanged, diverging one process level below what a field-only verifier can observe. We discuss implications for defenses that bind only at the tool-invocation boundary.
Read the full abstract ▾
Software Engineering
Doing More with Less Tokens: Hierarchical Reinforcement Learning for Efficient Coding Agents
Haobin Li, Liang Jiang, Zhenyu Huang et al. (+2 more)
Recently, coding
agents have emerged as a dominant paradigm for real-world software engineering (SWE) scenarios, which solve complex tasks through multi-turn interactions with development environments. However, frequent interactions with environments would inevitably introduce substantial
token overhead, leading to high usage costs and latency. Although recent studies have explored reducing
token usage by context manipulation and interaction limits at inference time, these approaches focus on improving
token efficiency while overlooking the risk of discarding task-relevant information, thus struggling to balance the trade-off between resolution rate and
token efficiency. In this paper, we study a more general paradigm without suffering from the limitation, i.e., training
token-efficient coding
agents with promising resolution performance, which is a highly-practical yet less-explored problem. To this end, we reveal two core observations in SWE scenarios: i) Efficiency Variation: successful resolution could be achieved with fewer
tokens; ii)
Entropy Correlation: unproductive behaviors are associated with turn-level
entropy. Motivated by observations, we propose a novel reinforcement learning framework, dubbed HERO. Specifically, HERO prioritizes task resolution over
token efficiency during policy optimization and encourages efficient
reasoning patterns at both trajectory and turn levels. Extensive experiments on SWE-bench Verified and SWE-bench Multilingual demonstrate that HERO achieves a favorable trade-off between resolution rate and
token efficiency compared with state-of-the-art coding
agents and reinforcement learning methods.
Read the full abstract ▾
All authors Haobin Li, Liang Jiang, Zhenyu Huang, Mouxing Yang, Xi Peng
Software Engineering
Toward Quantum Software Automation: A Quantum-Aware Harness for LLM-Guided Evolution
Lily Jiaxin Wan, Deming Chen, Klara Nahrstedt et al. (+1 more)
Quantum software is critical for improving the efficiency and reliability of scarce
quantum hardware. However, its design still relies heavily on ad-hoc, handcrafted heuristics that are often suboptimal and quickly become obsolete as
quantum hardware evolves. LLM-guided evolutionary search offers a promising way to automatically explore complex software designs, but existing search frameworks lack the
quantum-specific support needed for efficient
evolution: verification is expensive, feedback is sparse, and heterogeneous
quantum programs require different optimization objectives. In this paper, we present QSA, a
quantum-aware harness for LLM-guided evolutionary search toward automating
quantum software design. QSA equips the search with three forms of
quantum-specific guidance: an
evolution-hardness-guided coreset and approximate scoring to reduce verification cost, static and snapshot analyses to provide fine-grained execution context, and task-specific rewards for compiler passes and runtime policies. We evaluate QSA on the IBM
Quantum platform across three
benchmark suites. For multiprogramming, QSA improves QPU utilization by 4.2%-9.5% and Hellinger fidelity by 15.2%-19.5% over the state of the art. For error mitigation, QSA reduces mitigation time by at least 96.8% while achieving comparable or better fidelity. These gains require only $6.9 in LLM API cost over 11.3 hours.
Read the full abstract ▾
All authors Lily Jiaxin Wan, Deming Chen, Klara Nahrstedt, Bo Chen
Software Engineering · Cross-listed from cs.IR
PatchHolmes: Agentic Patch Retrieval via Listwise Selection
Guanqun Yang, Yingming Zhou, Jiangrui Zheng et al. (+2 more)
Patch
retrieval, the task of finding the commit that fixes a known vulnerability, is the foundation of vulnerability management workflows, yet 60% to 63% of CVEs in the major advisory databases lack a patch link. We present PatchHolmes, a two-phase patch
retrieval system that pairs a hybrid first-stage retriever with an agentic second-stage inspection loop. Unlike pointwise prior work that scores each candidate independently, the Phase 2
agent reads the top-100 listwise: it sees the full candidate list at once and selectively reads 3 to 10 commits through four budgeted tools before submitting a single best commit. On GitHubAD, PatchHolmes beats the pointwise binary classifier Favia by 25.34% Recall@1 and the retrieve-and-CoT baseline IRCoT by 31.40%, at one
agent conversation per CVE versus Favia's ten; with the candidate set held identical, the
agent adds 27.32% Recall@1 over taking the retriever's top candidate, and the same
agent, transferred unchanged to PatchFinder_top10, lifts Recall@1 from PatchFinder's own top-1 pick (24.28%) to 39.86%. Swapping the LLM backbone within the Qwen family changes Recall@1 by under 1%, and a second model family (gpt-oss) stays far above the no-
agent floor, so the gain comes from the listwise
agent loop; the entire system runs on a frozen open-weight model over a local Git repository, without fine-tuning or external search APIs.
Read the full abstract ▾
All authors Guanqun Yang, Yingming Zhou, Jiangrui Zheng, Shudong Hao, Xueqing Liu
Software Engineering
Adaptive-GEPA: Make Your Harness Fit Heterogeneous Requests
Tianyu Chen, Yasi Zhang, Ruiyi Wang et al. (+3 more)
Reflective optimizers such as GEPA improve
language model
prompts from execution traces and evaluator feedback; full-program extensions can also rewrite tools and control flow. In practice, a user hands the same endpoint heterogeneous requests whose effective solutions require different tools,
reasoning modes, and control flow. Optimizing one shared program leaves this division of work implicit in source-code search, while optimizing a separate program per request family fixes it beforehand. We introduce Adaptive-GEPA, which learns both how to divide requests and how to solve them. It evolves a router and a library of specialist programs under one search budget. The router's instructions, each specialist's description, and its program code are plain, human-readable text, edited from feedback. To combine branches, it aligns specialists by the requests they handle and inherits descriptions together with programs. On a fixed mixture of four task families, the reported Qwen3-8B run evolves four experts without supplying family labels to the router or reflection model; its routing matches the task partition on all 651 test requests. Its family-mean test score (x100) rises from 52.6 to 70.6, compared with 62.5 for GEPA's full-program adapter and 54.0 for GRPO at a nominal budget of 18,000 scored calls. These counts do not equate total compute. Figure 1 summarizes the learning curves, final test scores, and routing agreement.
Read the full abstract ▾
All authors Tianyu Chen, Yasi Zhang, Ruiyi Wang, Xinran Zhao, Taoran Li, Mingyuan Zhou