Software Engineering · Cross-listed from cs.PL
Tacet: A Language and Type System for Automatic Statistical Validity Accounting
Chiké Abuah
Empirical comparisons between systems are a standard form of evidence in computer science research, but few are checked for statistical validity: most are never framed as statistical tests at all. Existing multiple-comparison procedures could control the resulting error, but need inputs (what an analysis examined, and how its observations are arranged) that are not recoverable from a list of p-values. We introduce Tacet, a
language in which an analysis declares what it generated, states what it expects to find, and is refused any claim it cannot afford or cannot properly test. Its core calculus T pairs a free estimation sublanguage, carrying a reported footprint and a purity bit that records whether any outcome was consulted in building a value, with a priced claim sublanguage, carrying a wealth
transformer, connected only by a mechanism that prices a comparison. A sample selected by reading outcomes sets the purity bit and is recorded as having examined everything it read, permanently, so it can never be granted a one-sided or confirmatory price, without the system ever asking whether the analyst intended to cherry-pick. Whether a comparison is paired or clustered is computed statically from the artifact schema, from declared functional dependencies between key fields alone and before any data is read, and a mechanism that assumes that structure away is refused rather than priced. Because the wealth
transformer is antitone in the realized p-value, affordability can be checked before the analysis runs too, turning pre-registration into a typing rule. We prove the metatheory machine-checked in Lean 4 with no admitted gaps, and demonstrate the approach on a reference implementation and two case studies on published artifacts, the SWE-bench Verified leaderboard and BIG-Bench Hard.
Read the full abstract ▾
Software Engineering
SWE-Prime: Fewer Trajectories, Better Performance
Dewu Zheng, Ruizhe Ye, Yanlin Wang et al. (+7 more)
To improve large
language models' ability to resolve real-world software issues, prior work has focused on constructing large-scale
agent trajectory
datasets and performing supervised fine-tuning (SFT) on successful trajectories. However, task success does not guarantee high-quality supervision: successful trajectories may still contain ineffective, redundant, or risky steps. Directly using such trajectories for SFT can introduce noisy supervision and encourage models to imitate undesirable problem-solving behaviors. Therefore, we propose SWE-
Prime, a multi-granularity, two-stage SFT data selection method that progressively filters training data at the trajectory and segment levels. Specifically, the first stage performs trajectory-level screening based on process quality, result quality, and data representativeness, selecting a high-quality and representative subset of successful trajectories. The second stage performs segment-level selection by grouping consecutive steps into semantic segments and assessing each segment based on its contribution to the final solution, learnability, and potential risks. During SFT, all segments remain in the sequence to preserve context, while only selected segments contribute to the loss computation. Experiments on SWE-Bench Pro and SWE-Bench Verified show that training on the 10% trajectory subset selected by SWE-
Prime outperforms training on the full resolved
dataset, yielding relative performance gains of up to 12.2% and 24.2%, respectively.
Read the full abstract ▾
All authors Dewu Zheng, Ruizhe Ye, Yanlin Wang, Yang Ye, Hongyu Zhang, Ensheng Shi, Xilin Liu, Yuchi Ma, Jianxing Yu, Zibin Zheng
Software Engineering
From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench
Dewu Zheng, Yanlin Wang, Xiwen Wang et al. (+5 more)
In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large
language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios. To bridge this gap, we introduce MCR-Bench, the first defect state-aware
benchmark designed for realistic multi-round code review. MCR-Bench covers five commonly-used programming
languages and consists of 2,269 real-world multi-round code review tasks, each of which is annotated with fine-grained defect information and cross-round state labels. Each task in MCR-Bench is equipped with fine-grained defect metadata (e.g., description, type, severity) alongside dynamic state annotations, capturing the complete evolutionary trajectory of a defect throughout the multi-round process. We obtain several findings through extensive experiments on MCR-Bench with mainstream LLMs. (1) Limited overall capability: experiments reveal that mainstream LLMs exhibit limited overall performance in defect detection and defect lifecycle state tracking, with performance degrading significantly as the number of interaction rounds increases; (2) Defect-sensitive performance: LLMs' performance varies substantially across different defect types and severity levels, with semantically complex or low-salience defects being significantly more likely to be missed; (3) Underlying Failure Mechanisms: our in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range
memory.
Read the full abstract ▾
All authors Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, Zibin Zheng
Software Engineering
Persona-Execution Separation: An Architecture Pattern for Evolving LLM Agents under Execution Audit
Yisen Xi
Large
language model (LLM)
agents in governed organizations must let the persona (instructions, tone, self-presentation) evolve freely, while keeping execution (stateful, audited work) traceable. A single trust domain does not satisfy both cheaply. We present Persona-Execution Separation (PES): persona and execution reside in different trust domains, connected by a governed contract bridge. The persona is singly-homed and may drift; execution is faceless and audited. Status summaries may return; data bodies remain in the restrictive domain except a data-loss-prevention (DLP) exception; identity stays continuous. An approval
matrix, DLP, and audit enforce the crossing. PES follows from three goals: free drift, execution traceability, and decoupling. Under LLM representational indistinguishability, any single-domain mechanism meeting all three must re-introduce typed change objects, an external gate, and a stable audit anchor: PES rebuilt at higher coupling cost. A development/pilot case in a regulated platform records five decisions over one month, four with rejected alternatives. A mechanism check found no execution-side re-validation under persona perturbation (five configurations) and no persona fingerprint on hard-asserted fields of completed runs. A controlled replication in regulated coding
agents reproduced the separation under isolation across five models and four providers; bridge overhead was under 0.2% of end-to-end time in both environments. A probe of a pre-separation build found the execution path decoupled from the persona by omission, not by construction. The pattern applies when multi-user deployment, execution audit, and persona churn hold jointly.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CL
BTS-AgentBench: A Deterministic, Replayable Pipeline from Read-Only Telemetry Logs to Agent Benchmarks
Jeong-Yoon Kim and Tae-Yoon Lee
Operational facilities accumulate extensive local telemetry, but
agent-ready interactive evaluations with source-verifiable answers are costly to construct from such archives. Site-specific measurement histories also motivate a local conversion workflow that can operate where raw operational data must remain under organizational control. We introduce BTS-AgentBench, a deterministic compiler that transforms read-only telemetry and equipment metadata into tool-backed, multi-turn
agent tasks, phase-level targets, evidence and executable evaluation contracts. The BTS release includes 532 episodes across nine families; a second-corpus port produces 204 XAI4HEAT episodes. Frozen-input replay reproduces released artifacts byte-for-byte, and independent raw-source checks cover all 89 BTS and 41 XAI4HEAT test episodes. Replayable provenance links 1,170 trial identities to recorded execution and scorer conditions, while 1,125 completed traces retain model responses and tool results. An obligation-level comparison of 216 scored failures found that linked records exposed additional recorded failed obligations in 202 cases (93.5%), including tool arguments, stream identity, state carryover and missing interaction steps. The pipeline thereby supports source-grounded task generation, exact reconstruction and auditable diagnosis of
agent executions. Code, released
benchmarks, and reproduction instructions are available at https://anonymous.4open.science/r/telemetry-
benchmark-reproduction-3D68
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CR
When Context Gets Root: Privilege Escalation in LLM Harnesses
Xingbang He, Yuanwei Chen, Yi Qian et al. (+6 more)
Instruction hierarchy is a model-side defense that assigns instructions different levels of privilege according to their sources. These levels constrain which content may direct model behavior. During
agent execution, however,
agent harnesses construct context for each model invocation. This construction can elevate low-level content to a higher instruction level and grant it greater model-facing privilege. We introduce instruction privilege escalation. In this attack, an attacker induces an
agent to elevate low-level malicious content to a higher instruction level. The elevated content then causes the
agent to execute instructions it would not follow at their original level. We evaluate this threat by using multi-
agent mechanisms to achieve 13 attack objectives across six coding-
agent harnesses. These objectives span confidentiality, integrity, availability, and remote code execution. With unrestricted action execution, the attacks achieve all 13 objectives on all six harnesses. Under automatic permission review, the attacks achieve all 13 objectives on all three harnesses that provide this mode. We further reproduce the vulnerability using harness-provided persistent goals and scheduled tasks. These results demonstrate the generality of instruction privilege escalation.
Read the full abstract ▾
All authors Xingbang He, Yuanwei Chen, Yi Qian, Haiyang Wei, Ligeng Chen, Zenan Fu, Linzhang Wang, Hao Wu, Bing Mao
Software Engineering · Cross-listed from cs.CR
SPA: Securing Persistent LLM Agents Across Queries with Plan-First Information-Flow Control
Dylan Girrens and Guangjing Wang
Large
language model (LLM)
agents increasingly operate over untrusted webpages, documents, tools, and persistent states while exercising authority over security-sensitive resources. Existing defenses typically protect either planning or individual tool interactions, but persistent
agents face a broader threat: attacker-controlled data can alter control flow, enter security-sensitive tool arguments, or compromise later queries. We present SPA, a plan-first architecture that secures planning, execution, and cross-query state reuse. SPA invokes the planner once per query to generate a complete executable plan in a declarative domain-specific
language, then applies dual-
lattice information-flow control to track confidentiality and integrity across explicit data flows and control dependencies. To support persistence without re-exposing untrusted payloads to the planner, SPA stores execution results as labeled artifacts and reveals only semantic metadata during later planning. We evaluate SPA on AgentDojo and AgentDojo-MQ, which is our multi-query extension for measuring secure state reuse and delayed attacks. Under the 'tool_knowledge' attack, SPA with information-flow control reduces attack success to zero on AgentDojo and 0.2% on AgentDojo-MQ. Our results show that plan-first execution combined with label-preserving persistence can substantially strengthen persistent LLM
agents, while revealing an important security-utility tradeoff introduced by strict integrity enforcement.
Read the full abstract ▾
Software Engineering
Twelve Quick Tips for Managing IT Disasters in Small Research Software Teams
Greg Wilson
In 2025, the US government launched an unprecedented series of attacks on its own scientific research groups. A year later GitHub dropped below 90% availability for the first time, while wildfires in Canada, France, Spain, and elsewhere forced researchers from the homes and labs. These events and others have reminded us just how fragile research computing systems can be, and that planning for disasters is one of the most effective ways to prevent them. This paper is a short guide to disaster planning and recovery for a small research software team. The tips assume you are doing everything yourself on top of your regular job, and that you aren't an experienced system administrator. Some of the tips do require that kind of expertise, but most research institutions have research computing groups, data librarians, and environmental health-and-safety offices whose entire job is to help with exactly these problems. This paper tells you what "done" looks like; they can often provide it.
Read the full abstract ▾
Software Engineering
A Trans-Domain Digital Twin for Bio-Aware Control of Climate and Energy in Cattle Fattening Barns Using Single-Episode Optimizer Learning
Mansoorali Amiri
In closed cattle-fattening barns, the indoor
climate and herd growth are mutually interdependent. Temperature, relative humidity, airflow, and ventilation affect thermal comfort, feed intake, metabolic heat production, daily growth, feed efficiency, and energy consumption, while body-weight gain alters the future heat and moisture loads of the barn and, consequently, its ventilation, heating, and energy requirements. This article proposes a trans-domain digital twin framework with single-episode learning capability, customized for bio-aware
climate and energy control in a closed cattle-fattening barn. The framework integrates a mechanistic
climate simulator, a livestock growth simulator, model predictive control, lightweight reinforcement learning, and structured knowledge
memory within a multi-rate temporal-loop architecture. The fast temporal loop operates every five minutes to evaluate actuator decisions and maintain short-term thermal comfort, safety, and energy efficiency, whereas the slow temporal loop provides biological guidance based on daily climatic conditions, feed efficiency, heat production, and growth-limiting factors. The results show that
climate, growth, energy, feed, biological guidance, and
memory can be linked within a single executable control cycle. Remaining limitations include the need for field validation, improved management of feed pressure, and reduction of abrupt actuator-command variations.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.AR
LLMs in Digital EDA: A perspective on shifting roles from Generation to Orchestration
Matthew Youngman, Cristian Sestito and Themis Prodromakis
Electronic design automation (EDA) has advanced engineering productivity through successive generations of tooling that progressively automate synthesis, optimisation, and verification. Large
language models (LLMs) extend this trajectory by enabling direct translation from design intent to hardware implementations. In most of the EDA literature, LLM-based solutions are typically assisting siloed design stages or tasks, however this obscured the drivers by which capability emerges and systems scale. In this Perspective, we instead define three hierarchical roles that reveal how capability accumulates: a Generator that produces design artifacts in a single pass, an
Agent that refines outputs through iterative tool feedback, and an Orchestrator that coordinates decisions across EDA-stages. Across published systems, this reveals a syntax trap in which models are trained to produce plausible code rather than physically correct hardware, compounded by fragmented tools and loss of design context that obscure how decisions affect later stages. Comparisons across the three roles show that current approaches struggle to scale to industrial designs, motivating a shift towards a standardised, physics-aware orchestrator that connects tools and
agents across the EDA flow for more reliable and accessible hardware design.
Read the full abstract ▾
Software Engineering
AgentDV: Closed-Loop Agentic AI for Hardware Design Verification
Navya Goli, Junzhe Liu, Zhenge Jia et al. (+1 more)
Register-transfer level (RTL) verification consumes a major part of modern system-on-chip (SoC) development effort. Yet, recent LLM-based verification-code generation often fails to produce runnable, design-consistent, and coverage-producing testbenches. We present AgentDV, a closed-loop agentic AI framework for automated RTL verification environment generation. AgentDV transforms single-shot LLM testbench generation into a tool-grounded verification pipeline by combining LLM-guided analysis, testbench construction, simulation, coverage measurement, and iterative refinement. The framework introduces three key ideas: 1) runnability filtering to reject invalid generated environments, 2) CSR-grounded checking to reduce hallucinated signals and incorrect expected behavior, and 3) coverage-guided iteration to regenerate tests based on measured verification gaps. We evaluate AgentDV using three LLMs on challenge DUTs and public OpenTitan peripheral and security IP blocks. From our analysis, we observed that direct single-shot prompting fails to produce a valid coverage-producing environment on
benchmarks. AgentDV achieves 100% pass rate on four DUTs and an average of 80.9% pass rate on all DUTs using Claude Sonnet 4.6. Similarly, an average of 58.7% and 60.6% pass rate is achieved for Llama and Qwen models, respectively. In addition, an average of 74.5%, 69.1%, and 64.9% of line coverage and 88.4%, 82.3%, and 76.7% of branch coverage for the
benchmarks under consideration for Claude Sonnet 4.6, Llama, and Qwen models, respectively.
Read the full abstract ▾
All authors Navya Goli, Junzhe Liu, Zhenge Jia, Umamaheswara Rao Tida
Software Engineering · Cross-listed from cs.AI
When Tool Outputs Become Commands: Separating Action Induction from Runtime Authorization in Tool-Augmented LLM Agents
Xiaokun Guo, Zhen Xu, Dongdong Huo et al. (+5 more)
Tool-augmented LLM
agents must rely on untrusted runtime Observations to complete open-ended tasks; however, when tool outputs no longer merely provide data but begin to specify concrete actions, they effectively become “commands” that can drive real-world side effects beyond user intent. We argue that this risk arises from conflating action induction with execution authorization. To address this distinction, we propose SARA, which treats action induction and execution authorization as distinct runtime roles and separates action provenance from execution authority. On the Observation side, a context-isolated Action Probe exposes action-inducing semantics and persistently records action-origin provenance across steps as a review signal; on the execution side, actual tool calls are authorized only against the user objective and audited evidence from authorized successful executions, while satisfying goal, execution-chain, and argument-level support. To preserve this separation across multi-step execution, SARA applies No-History-Promotion to prevent historical recurrence from laundering action origins into execution authority. Across AgentDojo and AgentDyn, SARA limits ASR to no more than \(0.63\%\) across four primary evaluation settings while maintaining competitive task utility, and consistently reduces ASR across additional
Agent backbones.
Read the full abstract ▾
All authors Xiaokun Guo, Zhen Xu, Dongdong Huo, Yanqiu Zhang, Wei Wang, Qinfu Yang, Dongjin Yu, Yu Wang
Software Engineering
AROMA+: A Study of Factors Affecting Reproducible Builds in the Maven Ecosystem
Mehdi Keshani, Amirhossein Rahmati, Mohammad Hossein Aref et al. (+1 more)
Modern software engineering establishes software supply chains and relies on tools and libraries to improve productivity. However, reusing external software in a project presents a security risk when the source of the component is unknown or the consistency of a component cannot be verified. Reproducible builds present a mitigation strategy, as they can confirm the origin and consistency of reused components. A large reproducibility community has formed for Debian, but the reproducibility of the Maven ecosystem, the backbone of the Java supply chain, remains understudied in comparison. Reproducible Central is an initiative that curates a list of reproducible Maven libraries, but the list is limited and challenging to maintain due to manual efforts. Our research aims to support these efforts in the Maven ecosystem through automation. We investigate the feasibility of automatically finding the source code of a library from its Maven release and recovering information about the original release environment. Our tool, AROMA+, can obtain this critical information from the artifact and the source repository through several heuristics and we use the results for reproduction attempts of packages on Maven Central. Overall, our approach achieves an accuracy of up to 99.8% when compared field-by-field to the existing manual approach. In some instances, we even detected flaws in the manually maintained list, such as broken repository links. We reveal that automatic reproducibility is feasible for 32% of the packages on Maven Central using AROMA+, and 12% of these packages are fully reproducible. We demonstrate our ability to successfully reproduce new packages and have contributed some of them to the Reproducible Central repository. Additionally, we highlight actionable insights, outline future work in this area, and make our
dataset and tools publicly available.
Read the full abstract ▾
All authors Mehdi Keshani, Amirhossein Rahmati, Mohammad Hossein Aref, Abbas Heydarnoori
Software Engineering
Mutation Testing for Reproducibility Safeguards in Machine Learning Research Software: An Empirical Study
Ilya Shulepov
Reproducibility in machine-learning research depends on experimental choices such as random seeds, dependency versions, data partitioning, and evaluation configuration. Existing repository validation workflows may execute successfully without detecting changes to such choices. We study this problem using MLReproMutate, research software that applies controlled, reproducibility-relevant mutations to ML research repositories and evaluates them against validation workflows already present in those repositories. We conducted an outcome-blind empirical study of 39 frozen repository-operator cases using four mutation classes: random seed, dependency pin, data split, and cross-validation fold count. Repository revisions, mutation candidates, and validation workflows were fixed before mutation outcomes were observed. Primary execution yielded outcomes for 13 of 39 cases; a bounded restoration procedure increased the combined evaluable set to 24. After excluding one confirmed-equivalent mutation, 23 confirmed non-equivalent mutations remained. The selected validation workflows detected 2 of these 23 mutations, corresponding to an observed detection proportion of 8.7%. These results do not imply that the corresponding repositories are irreproducible. Rather, they show that, in this sample, existing validation workflows often did not detect the particular controlled reproducibility-relevant changes introduced by the study. The findings motivate reproducibility-oriented mutation testing as a complementary way to assess whether research-software safeguards constrain experimentally important choices.
Read the full abstract ▾
Software Engineering
An Empirical Evaluation of Using Large Language Models for Automated Model-Based Test Generation
Hafize Sanli, Onur Kilincceker and Cihat Cetinkaya
Large
language models have shown strong potential for software engineering tasks, particularly software testing. Model-based testing (MBT) is a software testing technique. To address the broad scalability challenge for industrial adoption of MBTs, our paper presents an empirical evaluation of Large
Language Models (LLMs) for automated model-based test generation, compared with a state-of-the-art model-based testing tool (GraphWalker) and its built-in
algorithms (random and quick random for edge and vertex coverage settings). Our evaluation indicates strong potential to optimize and shorten test paths and step sizes using the recent five state-of-the-art LLMs (GPT-5.1, GPT-5.2, Claude Opus 4.5, Claude Sonnet 4.5, and Gemini 2.5 Pro) against four GraphWalker models (two web applications (Parabank and Testinium) and two hardware applications (TLC and RISC-V) ) of escalating complexity.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.NE
Bug Localization from Bug Reports: A Multi-Objective Approach
Waleed Ahmad, Mehtab Kiran Suddle and Maryam Bashir
Bug localization is a labor-intensive task, particularly in large software systems. When abnormal behavior occurs, developers must perform repetitive and time-consuming steps to identify faulty files. Previous studies have mainly focused on single-objective localization methods, many of which are limited to specific programming
languages. In addition, relying solely on lexical similarity between source code and bug reports is often insufficient due to the natural
language nature of bug descriptions. In this study, we propose a class-level automated multi-objective search-based system to identify and rank potentially buggy classes from bug reports. The main objective is to maximize similarity while minimizing the number of suggested faulty files. The evolutionary optimization
algorithm SPEA-2 was applied to six open-source Java projects comprising more than 22,000 bug reports. The proposed approach was evaluated against two widely used
algorithms, NSGA-II and MOEA/D. Results indicate that SPEA-2 achieved higher precision and recall than both multi-objective and single-objective baseline methods. The proposed recommender system successfully identified buggy classes or files for 88.5% of bug reports within the top 10 recommendations and 94% within the top 20. The effectiveness of the model was further validated on an industrial Android project written in Kotlin, demonstrating its adaptability across programming
languages.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.AI
A Contract-Centered Architecture for Scalable and Manageable Agentic Runtimes
Yaxiao Liu, Pengbo Liu, Yiwen Liu et al. (+3 more)
Enterprise agentic systems must coordinate changing capabilities, execution capacity, and independently governed data. We define Skill, Harness, Scaffold, and an external data substrate as responsibility contracts. The central hypothesis, cost-aware capability-capacity separability, asks whether compatible capacity changes preserve semantic outcomes while capability changes preserve the capacity-response relationship within declared margins and enforcement budgets. We operationalize the data boundary through a source-oriented Data Wiki, an output-oriented Theme Wiki, and a versioned Intermediate Relation. Executable 5W1H+Which predicates bind source identity, validity, authorization, semantics, operations, relations, and evidence requirements. Request-bound tickets add execution-time revalidation and typed rejection. A conditional soundness argument states the required trust and atomicity assumptions; dependency invalidation makes change propagation explicit. A single-process reference model agrees with a declared specification oracle on all 1,024 combinations in a finite synthetic fault domain and passes five lifecycle checks. These are conformance results, not measurements of
retrieval quality, production safety, or scaling. We specify a held-out data study and a cluster-period crossover with distinct supported, falsified, conditional-engineering, and inconclusive verdicts. The core separability hypothesis remains empirically untested.
Read the full abstract ▾
All authors Yaxiao Liu, Pengbo Liu, Yiwen Liu, Yihua Guan, Zhenghe Hou, Jiaxing Song
Software Engineering · Cross-listed from cs.CR
A Catalog of User Authentication Patterns
Alex R. Mattukat and Horst Lichter
Security patterns are intended to support the design and development of secure software systems. However, although established catalogs of security patterns exist, their practical application remains limited. In particular, despite these catalogs, concrete patterns for common security controls such as user authentication (authentication for short) are lacking. This paper aims to make an initial contribution toward closing this gap, as exemplified by authentication. It presents a novel authentication pattern catalog, comprising 14 user authentication patterns. To support the catalog's practical application, it classifies patterns by the well-known concept of authentication factors and by the usual role each pattern fulfills in practice. By cataloging common authentication techniques through authentication patterns, we aim to make an important contribution to supporting software engineers and architects in designing and developing secure software systems.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.AI
Evaluating human and LLM screening workflows in a conceptually complex scoping review: Recall–workload trade-offs and run-to-run consistency
Nikol Figalová, Lynn Huestegge and Anne Böckler-Raettig
Background. Large
language models (LLMs) are increasingly used for screening in evidence synthesis, where false negatives can remove relevant studies before full-text assessment. We compared human and LLM title-and-abstract screening workflows in a preregistered study embedded in a conceptually complex scoping review. Methods. After a conservative title-only screen, 1,131 records were screened by one review lead, four trained assistants screening non-overlapping subsets, and seven complete LLM runs using different models and processing configurations, including a nominally identical repeat run. We compared retained workload, operational recall against 316 verified eligible records, agreement, run-to-run consistency, and procedural burden. Because eligibility was verified only for records advanced and assessed in the parent review, recall estimates were operational. Results. No workflow recovered all verified eligible records. The human workflows and two GPT-5.4 file-batch runs retained 42.2-45.0% of records while achieving 82.3-82.9% recall. Gemini 3.1 file batches achieved the highest recall (83.9%) but retained 56.7% of records. All-at-once configurations recovered fewer eligible records than corresponding file-batch configurations. Two nominally identical GPT-5.4 file-batch runs agreed on 91.7% of records but differed on 94 records, including 29 verified eligible records retained by only one run. Discussion. LLM screening performance depended on the implemented workflow, not model identity alone. Processing configuration, workload, record-level variation, and human-LLM decision integration are therefore substantive properties of deployed systems. For high-recall tasks, LLMs are better suited to validated, auditable, human-supervised workflows than autonomous exclusion.
Read the full abstract ▾
Software Engineering
Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research
Lezhi Yu, Xiaogang Xu, Yuhua Zhou et al. (+2 more)
LLM
agents used for scientific experimentation must do more than generate executable code: they must implement the reference method faithfully, design experiments that test the paper's claims, and provide evidence supporting those claims. We show that
agents often produce methodological
hallucinations: silently reducing
datasets or training budgets, replacing failed learning or generative components with lookup or oracle functions, or drawing conclusions from resource-limited settings where a method's claimed advantage disappears. To detect these failures, we introduce ABE-Ralph, a reference-anchored auditing framework that represents claims, protocols, required components, baselines, and metrics as structured experimental constraints, guides implementation through an 8-step workflow, and performs quantitative, qualitative, and code-level verification. Across 30 long-horizon reproduction runs covering 12 machine learning domains, ABE-Ralph achieves a 93% robust execution rate and identifies five scientific failure modes. In 23 NatureBench discovery tasks, ABE-Ralph matches or exceeds state-of-the-art performance on 5 tasks. These results show that reliable evaluation of AI scientists must assess whether the experimental design faithfully tests the intended claim and whether the resulting evidence supports it, rather than treating code execution or plausible metrics as evidence of scientific success.
Read the full abstract ▾
All authors Lezhi Yu, Xiaogang Xu, Yuhua Zhou, Shuibing He, Aimin Pan
Software Engineering
FaultLens: Learning Compact Behavioral Test Suites for Generated Operational Programs
Zeming Liu, Hang Lyu and Jingtao Zhang
Generated operational programs are often validated with either a few hand-written examples or exhaustive regression suites. The former can miss sparse boundary and interaction faults, while the latter can be unnecessarily expensive. We introduce FaultLens, a method for learning compact behavioral test suites while preserving an auditable connection to executed evidence. It executes a rich probe domain once, stores the fault-probe kill relation as a sparse outcome cache, and learns probe orderings only from earlier program generations. A fault-driven greedy component exploits known kill structure, while a mutation-independent diversity component covers probe families, cases, templates, and temporal bins. Their alternating hybrid remains useful when a new program contains a fault mechanism absent from ordering construction. We evaluate twenty generated operational policies across four environments, ten execution seeds, 1,200 measured run summaries, 2,160 controlled program transformations, and 4,120,200 executed program-probe pairs. Of 1,960 intended faulty transformations, 1,779 alter a contract or output somewhere in the finite audit domain; 200 additional controls preserve behavior. A 32-probe hybrid learned on generations 1-3 covers 576/582 (99.0%) dynamically killable faults in generations 4-5 using 1.2-2.0% of the exhaustive domain. With an entire fault family withheld from training, diversity raises scenario-family macro coverage from 84.6% to 94.9%. In a downstream deployment study, a conservative admission rule reduces severe tail regressions from 15/20 program-environment groups to 0/20. FaultLens provides a prioritized evidence mechanism, not a proof of correctness, and makes its budget, evidence source, generalization split, and misses explicit.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.NI
Claude Code Complete User Handbook
David Soldani
Claude Code is an agentic work environment: a
language model operating in a loop with filesystem access, shell execution, browser control, scheduled and cloud execution, external tool connections through the Model Context Protocol, and multi-
agent orchestration. Its capability envelope now exceeds what one practitioner can supervise by
attention alone, and its failure modes are systemic rather than local: an unreviewed hook, an over-scoped connector, a stale completion condition, an autonomous routine inheriting every credential on an account. This book is a task-oriented reference for operating that system safely and productively, written for practitioners accountable for the result. It advances four propositions. First, capability without a defined and observable completion condition is not productivity. Second, instruction, permission enforcement, sandboxing and operating-system isolation are four distinct layers of a control stack, only two of which are enforced, and conflating them is the most common cause of loss of control. Third, third-party skills, plugins, marketplaces, channels and MCP servers are software supply-chain dependencies and must be governed as such. Fourth, the correct unit of trust in agentic work is observed evidence, not an
agent's closing statement. Thirty-four chapters run from installation to a fully verified capstone, with a governance part on managed policy, data residency and retention, observability and accessibility. Every product claim carries a citation to a primary source; an evidence ledger records where a claim in circulation was found wrong, what a later re-verification changed, and what remains unverified. Controls are mapped to seventeen external frameworks in a crosswalk, and an organisational adoption maturity model is proposed. Claims not confirmable from primary sources are labelled UNVERIFIED rather than softened.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CR
KubeCap: A Framework for Capability Minimization in Kubernetes via Static Analysis and LLM-Assisted Rule Inference
Yuhao Liu, Yingnan Zhou, Weijie Liu et al. (+2 more)
As the most widely used container orchestration platform, Kubernetes provides flexible privilege configuration by allowing developers to manage Linux capabilities via manifest files. However, developers rely on default settings or coarse-grained security contexts in practice, violating the principle of least privilege and enlarging the attack surface of containerized workloads. Existing studies either detect vulnerable patterns in Kubernetes manifests or infer required capabilities for standalone Linux programs, but they do not directly address capability minimization in Kubernetes. To bridge this gap, we first conduct an empirical study on three open-source
datasets, revealing that 74.67% of projects lack capability configurations. Motivated by our observations, we propose KubeCap, a framework for Kubernetes capability minimization. KubeCap translates deployment specifications into deterministic manifests, locates container entrypoints, performs reachability-guided system call analysis, and leverages LLM-assisted rule specification to derive syscall–parameter–capability relations from Linux kernel code. Based on these results, KubeCap infers the minimal capability set required by each workload and automatically generates repaired manifests. Evaluation on 10 representative Go-based Kubernetes projects shows an average capability reduction rate of 54.97%, outperforming rapid type analysis and class hierarchy analysis baselines while maintaining practical analysis cost. These results demonstrate KubeCap's effectiveness in enforcing least privilege in Kubernetes.
Read the full abstract ▾
All authors Yuhao Liu, Yingnan Zhou, Weijie Liu, Yan Jia, Zheli Liu
Software Engineering · Cross-listed from cs.AI
Five Primitives for Governing Autonomous AI Agents at Runtime
Jiten Oswal and John Cadeddu
Enterprise deployments of autonomous AI
agents inherit a control model built for human users and long-lived services, and the fit fails in three specific ways:
agent principals are ephemeral, appearing and vanishing faster than provisioning; their actions are selected by a model rather than programmed, so the set of things they may attempt is not known in advance; and the population is discovered rather than provisioned, because anyone who can call an API can create one. We argue that governing such
agents is a runtime problem – not a model-
alignment problem and not a build-time problem – and we derive five primitives from the questions that must be answered before an action takes effect and after it has: discovery, identity, governance, attestation, and supply chain. For each we state what fails if it is absent and why the others cannot structurally supply it. We describe an implementation in which an
agent's action is mediated against policy before it takes effect, authorised against a per-tenant action vocabulary, and recorded in a hash-linked signed ledger a third party can verify with the vendor out of the loop. We report what the architecture costs: the enforcement point sits on the request's critical path, identity requires a sidecar per workload, and fail-closed mediation converts availability incidents into denial. We are explicit about implementation status: four primitives are built and running in private pilots, and the fifth is built as separate tooling and not yet integrated into the request path. We keep it in the set deliberately: a five-part decomposition that exactly matches what its authors happened to build is not a taxonomy but a description of a codebase.
Read the full abstract ▾
Software Engineering
Processing/p5 Defined through Practice and Learning
Kit Kuksenok and Lee Tusman
Processing/p5 libraries across different programming
languages enact consistent priorities for creative coding as a designed experience. While different programming
language ecosystems, like Java and JavaScript, are each associated with their own affordances, community norms, and patterns of use, Processing/p5 sketches across these
languages share similarities. Based on case studies of building an implementation of Processing/p5 in two host
languages, JavaScript and Lua, we propose a list of software decision-making guiding aspects that constitute Processing/p5, regardless of host
language. We discuss this framework in the context of decisions in other exploratory and creative tools that demonstrate how each of the guiding aspects can be operationalized differently than in the case studies. The proposed list highlights opportunities for learning, research, and artistic practice through creation of new Processing/p5 libraries for creative coding and algorithmic art.
Read the full abstract ▾
Software Engineering
The Thousand-Graph Hypothesis: A Testable Hypothesis of Task-Conditioned Relation Materialization in Repository-Level Code Reasoning
Fei Ding
Large software repositories are often beyond model context limits. Training repository knowledge into models is costly and quickly stale, while local
retrieval can miss scattered requirements, and explicit relation graphs add ongoing maintenance burden. We propose an entity-only external interface with task-conditioned relation materialization during inference. A two-layer index separates global routing from local entity focus and is evaluated on DeepSeek-V4-Flash and SWE-bench Verified. The base, one-layer, and two-layer conditions achieve 92.1%, 94.2%, and 95.6% success, respectively, under zero pre-built entity-relation edges.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CR
Unsaid, Unsafe? Implicit Security Obligations in LLM-Based RTL Code Generation
Guang Yang, Xing Hu, Xiang Chen et al. (+1 more)
Large
Language Models (LLMs) generate register-transfer-level (RTL) code with rapidly improving functional correctness. Security of LLM-generated code, however, has been studied mainly for software, where flaws can still be patched after deployment. Insecure RTL offers no such remedy once taped out into silicon. We construct SECRTL-GEN, a multi-
language resource-access security
benchmark grounded in real SoC IP: 392 tasks over five CWE families and four HDLs (Verilog, SystemVerilog, VHDL, and Python), each with black-box functional and security testbenches. Functional specifications intentionally omit security obligations, matching how obligations are often kept out of functional docs in practice. An empirical study of five frontier LLMs shows a sharp gap: under vanilla
prompts they pass functional tests in about 73-79% of cases but security tests in only 14-35%, and stronger functional models are not safer. Adding CWE knowledge raises security, while unaided self-thinking helps less and both security-oriented
prompts cut functional pass rates, showing that the bottleneck is missing weakness awareness in the specification, not an inability to write defensive RTL. We present RTL-Obliger, a neuro-symbolic framework that infers these implicit obligations. An LLM extracts a functional-semantic graph from the specification; a symbolic engine then matches it against a CWE pattern ontology to surface mitigation-evidence gaps and signal-level obligations; the LLM finally revises RTL under those obligations in a functionality-preserving two-stage generation. Across five models and four
languages, RTL-Obliger raises mean all-pass from 49.6-51.4% (SecV/RESCUE) to 61.6%, with higher security and functional rates than these secure-generation baselines.
Read the full abstract ▾
All authors Guang Yang, Xing Hu, Xiang Chen, Xin Xia
Software Engineering
DeepRepro: State-Aware Subplanning for Paper-to-Code Reproduction in Evolving Repositories
Hongru Song, Ruqing Zhang, Jiafeng Guo et al. (+2 more)
Recent advances in agentic large
language models (LLMs) have enabled increasingly autonomous software engineering workflows, yet automatic machine learning (ML) paper-to-code reproduction remains a challenging long-horizon problem. Unlike conventional code generation, this task requires constructing and maintaining a fully functional repository whose state continuously evolves during execution. Existing systems typically rely on static upfront planning followed by sequential file-level generation, which often leads to inconsistencies as dependencies, interfaces, and execution feedback change over time. We propose DeepRepro, a state-aware framework for paper-to-code reproduction based on execution-state-aware subplanning. DeepRepro dynamically transforms evolving repository states and runtime feedback into fine-grained implementation subplans, keeping planning aligned with execution throughout repository construction. The framework further incorporates repository-aware orchestration and a lightweight process-aware interface for transparent monitoring of long-horizon reproduction. Experiments on PaperBench Code-Dev show that DeepRepro consistently outperforms strong scientific and commercial code-
agent baselines.
Read the full abstract ▾
All authors Hongru Song, Ruqing Zhang, Jiafeng Guo, Xueqi Cheng, Maarten de Rijke
Software Engineering · Cross-listed from cs.CE
Report of the 2026 Workshop on Next-Generation Ecosystems for Scientific Computing: Harnessing Community, Software, and AI for Cross-Disciplinary Team Science
Lois Curfman McInnes, Dorian Arnold, Prasanna Balaprakash et al. (+45 more)
Scientific computing is undergoing rapid transformation as advances in artificial intelligence, heterogeneous computing, automation, and data-intensive research reshape not only computational tools but also the institutions, workforce models, and collaborative practices that support scientific discovery. This report synthesizes insights from the 2026 Workshop on Next-Generation Ecosystems for Scientific Computing, the second in a three-year series focused on strengthening scientific computing ecosystems through socio-technical co-design. Workshop discussions identified four interdependent strategic themes: software ecosystems for AI-enabled scientific discovery; trust, validation, and traceability; human-AI teaming and paradigm shifts; and workforce, pedagogy, and governance. The report translates these themes into eight priorities for community action spanning shared research infrastructure, trust and traceability, user experience, human-AI teaming, workforce development, cross-sector coordination, stewardship and sustainability, and evaluation of scientific value. Together, these priorities outline directions for building scientific computing ecosystems that remain trustworthy, sustainable, innovative, and resilient as AI assumes a growing role in scientific work.
Read the full abstract ▾
All authors Lois Curfman McInnes, Dorian Arnold, Prasanna Balaprakash, Mike Bernhardt, Franck Cappello, Beth Cerny, Deborah DiazGranados, Anshu Dubey, Nichole Etienne, Roscoe Giles, Diego Gomez-Zara, Denice Ward Hood, Mary Ann Leung, Vanessa Lopez-Marrero, Olivia B. Newton, Irene Qualters, Keita Teranishi, Stefan M. Wild, Gabrielle Allen, Richard Arthur, Alexandra Ballow, Tony Baylis, David E. Bernholdt, Daniel Bielich, Johanna Cohoon, Jeremy Crampton, Charles Ferenbaugh, Stephen M. Fiore, Thomas Herault, Tanzima Islam, Stephen Jacobsohn, Meifeng Lin, Charles Lively, Satoshi Matsuoka, Stasa Milojevic, Daniel Nichols, Chris Oehmen, Santiago Ospina Tabares, Michael E. Papka, Katherine Riley, Damian Rouson, Sudip K. Seal, Brittany Segundo, John Shalf, Andrew Siegel, Valerie Taylor, Jim Willenbring, Lou Woodley
Software Engineering · Cross-listed from cs.MA
GVS5H: Zero-Shot Self-Orchestration with Ledger-Based Control for Improved LLM Coding Performance
Victor Gao, Vida Khosrowshahi, Ali Khosrowshahi et al. (+3 more)
Frontier coding performance is typically attained with large, costly proprietary models. We introduce ledger-based zero-shot self-orchestration (GVS5H), a training-free method in which fresh instances of one model decompose problems and coordinate through a shared file system. Across eleven open and closed-weight models on the 100 latest hard LiveCodeBench problems, the method yields as much as 25.6 points improvement, boosting several cheaper models to frontier-level performance. Orchestrated Qwen3.8 Flash Next scores 93.0% against Fable 5's 90.4% at 9% of the cost, while the smaller Qwen3.8-27B reaches 92.4%. Gains are not universal: some models are unchanged or worse. Transcript analysis attributes the gain to decomposition and persistent context. Inference-time organization can reach or exceed frontier coding accuracy at a fraction of the cost on self-hostable weights.
Read the full abstract ▾
All authors Victor Gao, Vida Khosrowshahi, Ali Khosrowshahi, Xihao Sun, Juhyun Lee, Simon Lee