Software Engineering · Cross-listed from cs.CR
Correct Verdicts, Flawed Reasoning: Structured Auditing of LLM-based Vulnerability Reasoning
Boyue Caroline Hu, Kaivalya Ahir, Ronghao Ni et al. (+1 more)
Large
Language Models (LLMs) are increasingly deployed for automated software vulnerability analysis. Binary classification alone is insufficient; practitioners need explanations to triage bugs and engineer patches. Standard practice relies on Chain-of-Thought (CoT) prompting, but free-form
reasoning allows models to obscure logical leaps, hallucinated execution steps, and internal inconsistencies behind plausible prose. Our manual audit reveals that approximately 60% of correct vulnerability verdicts are accompanied by fabricated or unverifiable claims, and free-form explanations allow
reasoning errors to evade LLM-as-a-judge evaluation. We present Vulnerability Explanation
Reasoning Auditor (VERA), an automated framework for auditing LLM vulnerability
reasoning. Rather than accepting free-form text, VERA asks models to output a Structured
Reasoning Record (SRR) encoding tracked pointers,
memory operations, and state transitions in machine-readable fields. A multi-stage judge audits each SRR against eight
reasoning failure modes using deterministic checks, with LLM calls reserved for semantic interpretation. The standardized SRR schema also enables automated mutation testing to
benchmark judges at scale without human annotation. Our evaluation shows
reasoning flaws occur in correct verdicts just as frequently as incorrect ones, and VERA exposes 87% of
reasoning errors that free-form LLM-as-judge systematically miss.
Read the full abstract ▾
All authors Boyue Caroline Hu, Kaivalya Ahir, Ronghao Ni, Limin Jia
Software Engineering
LLM Non-Determinism in Deterministic Processes: A Controlled Variation Space for Constraining the Propagation of Non-Determinism
Paul Darius Mandl, Peter Mandl and Martin Häusl
Large
Language Models are increasingly used in software development processes that rely on defined states, rules, and approvals. This creates a tension between the flexibility of LLMs and the need for reliable and traceable process states. We introduce Controlled Non-Determinism, a process-oriented approach that allows variation without letting it directly determine the authoritative process state. We place LLM-based processing within a controlled variation space that is bounded by a Deterministic Envelope. Before execution, we define the process-relevant properties of the result and the exit conditions. We also record the task configuration. These elements remain fixed during execution. At the end of the variation space, an exit check determines whether the result satisfies the predefined conditions and may become the next authoritative state. Different runs may produce different admissible results while the transition to an authoritative state remains controlled. We illustrate the approach with regulated software development and derive design guidelines for LLM-based changes whose results can be checked before they become part of the authoritative software state.
Read the full abstract ▾
Software Engineering
Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents
Hai Dang Truong, Rayner Goh, Thanh Le-Cong et al. (+1 more)
Autonomous coding
agents now resolve a substantial share of real-world GitHub issues. However, passing functional tests differs fundamentally from producing a high-quality contribution acceptable for merging. Mature open-source projects publish repository-specific contribution policies, spanning style, git, testing workflows, to ensure code quality and long-term maintainability. Because existing
benchmarks evaluate patches solely on unit tests,
agent compliance with repository governance remains unknown. In this paper, we introduce SWE-CC, a
benchmark evaluating code and process compliance in autonomous software engineering. We develop a semi-automated pipeline that converts developer documentation across 12 open-source repositories into 823 machine-checkable atomic policies. SWE-CC introduces two features: 1) lightweight, deterministic checker functions that represent each policy, 2) a comprehensive auditing mechanism that inspects both
agent runtime behaviors and final deliverables. We evaluate the compliance of
agent workflows in 500 end-to-end software contribution tasks extended from SWE-bench Verified. Our evaluation of four LLMs under two
agent scaffolds shows that modern
agents suffer from coding compliance issues: although
agents produce functionally correct patches, they still violate 43.1 percent of applicable project policies, with nearly half of all violations occurring during intermediate execution steps. These results show that functional correctness does not guarantee real-world readiness, highlighting that future software engineering
agents must reliably conform to repository governance to enable safe and trustworthy deployment.
Read the full abstract ▾
All authors Hai Dang Truong, Rayner Goh, Thanh Le-Cong, Yintong Huo
Software Engineering · Cross-listed from cs.CR
Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair
Wenji Bai, Muhammad Waseem, Zeeshan Rasheed et al. (+2 more)
Localizing where an LLM-based
agent first fails to uphold security during a repair can show which stage of its workflow needs an additional safeguard. This is difficult for silent failures, which are patches that pass syntactic and functional checks but still contain a security vulnerability. Because such patches give no observable failure signal, existing failure attribution methods, which rely on observed task failures and labelled failure steps, are less suited to them. We propose Security Awareness Gap Evaluation (SAGE), a trace-based method that combines an assessment of the security
reasoning recorded at each turn with the reconstructed code history to identify the earliest turn at which a repair diverges from the task's security intent. We evaluate SAGE on 95 confirmed silent failures drawn from 3,684 repair traces produced by six
agent frameworks and six base models on SecurityEval and CVEfixes. SAGE assigned an origin in 93 cases. Most origins were an unaddressed security requirement or an inadequate defence choice, and only five coincided with the code change itself. When the
agent introduced the vulnerable code, the origin preceded the write in 14 of 19 cases. Repeated scoring and a second judge reproduced the origin type more consistently than the exact turn, and agreement was lowest for traces that kept only the final file.
Read the full abstract ▾
All authors Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, Pekka Abrahamsson
Software Engineering · Cross-listed from quant-ph
When Equivalent Quantum Circuits Lose Synthesis Choices
Boshuai Ye, Peng Liang and Arif Ali Khan
Quantum compilers synthesize high-level operations, such as the
quantum Fourier transform and multi-controlled X gates, into gate-level
circuits. Across compilation stages, a
circuit may be serialized, exchanged as OpenQASM, converted between compilers, or lowered to gates. These representation changes can preserve computation while removing the high-level operation itself, leaving the receiving compiler unable to apply a requested synthesis method. We call this loss synthesis availability. We study synthesis availability in Qiskit, TKET, and Cirq through controlled experiments, repository analysis,
circuits captured from project tests, and a Munich
Quantum Toolkit pipeline. Synthesis availability survives serialization when deserialization restores the high-level operation and survives compiler conversion only when the receiving compiler supports that operation. It does not survive the evaluated OpenQASM routes. After a direct OpenQASM 3 round trip, requested synthesis has no effect in all 360 delivered conditions even though compilation succeeds. In the MQT pipeline, OpenQASM 2 interchange increases the two-
qubit gate count by 37.2 percent for an eight-
qubit Grover
circuit. We also introduce Verified, which records high-level operations before a representation change and reconstructs them only when a check confirms that the record still matches the delivered
circuit. Verified restores requested synthesis in all 830 recovery runs, with median verification times from 2.0 to 638 ms per operation family. The check rejects every stale record but also some valid records. These results show that preserving functional equivalence alone is insufficient when later synthesis choices depend on retained high-level structure.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CR
Security Is More Than a Library Call: How Security Features Live in Code
Kevin Hermann, Sven Peldszus and Thorsten Berger
Implementing security features—functionalities that protect sensitive data or prevent malicious actions by attackers—is important for ensuring the security and integrity of software systems. Correctly implementing access control, cryptography, or other security features is challenging as they require substantial domain expertise, careful design, and custom implementation to integrate them within the software system. Previous work has thoroughly studied security awareness, perception, practices, and expertise via surveys, interviews, and experiments. While they have shown that the implementation of security features are often supported by security libraries and frameworks, they have also shown that developers rarely are security experts, and make mistakes that introduce vulnerabilities into software when using them. Despite this extensive knowledge of security practices and developer behavior, we still lack a comprehensive understanding of how security features actually manifest at the code level across full software systems. We close this gap by conducting a mining study of security features in 9 popular and large open-source Java systems. We manually inspected 2,127,761 lines of code across 19,121 files, identifying and annotating 183,395 lines implementing 561 security features corresponding to 54 security features in our taxonomy. We analyzed the characteristics of the identified security features, such as their size, scattering, tangling, common implementation patterns, and the use of internal and external functionalities. Our findings show that security features are far more than mere calls to external libraries. Security features, such as access control, can grow large in size, scatter across the whole codebase, and frequently tangle with each other. External security libraries are commonly used, but they require substantial amounts of code for integration.
Read the full abstract ▾
Software Engineering
Where Does the Budget Go? Structural Concentration in Learning-Based Mutant Selection
Taha Rostami, Jeongju Sohn and Mike Papadakis
Learning-based mutant selection is used to reduce the cost of mutation testing by ranking and selecting the mutants that seem to be the most promising. Prior work has shown great potential for this approach in fault revelation and subsuming-mutant selection. However, the risks and benefits of these approaches – especially the distribution of selected mutants across code locations and how that distribution affects behavioral diversity and fault revelation – remain unclear. We investigate this distribution and its effects, and introduce LinePool, a simple, model-agnostic stratification step that distributes selections across source lines while retaining the underlying ranking signal. Across two
datasets and budgets (2% and 5% of the mutants), we find that learning-based selection concentrates the budget on a few code locations, and may thereby systematically introduce 'blind spots', untested code areas, allowing faults to escape detection. On larger programs, the selected mutants also exhibit substantial overlap in kill behavior. LinePool substantially increases spatial coverage and reduces kill-set overlap. It also improves fault revelation at tight budgets and makes selection more robust to imperfect or shifted ranking signals. Comparisons with clustering and established diversification methods show that none consistently outperforms LinePool across
datasets, supporting its use as a simple diversification step. In general, our results suggest that behavioral diversity should be considered alongside effectiveness when designing and evaluating mutant-selection methods, and that exploiting structural elements of the program is one simple way to obtain such diversity while maintaining effectiveness.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CR
Backdooring Sparse Autoencoders
Enrico Ahlers, Daniel Passon, Tobias Kiecker et al. (+2 more)
Sparse autoencoders (SAEs) are increasingly used not only to interpret
language models but also to intervene on their internal representations. We show that this creates a supply-chain attack surface: a maliciously modified SAE can induce attacker-chosen behavior when inserted into the forward pass of an otherwise unchanged
language model. We introduce a decoder-only SAE backdoor that leaves both the underlying LLM and the SAE encoder frozen, restricting the attack to a single auxiliary component at a single insertion layer. Using code generation as a case study, we demonstrate high rates of unsolicited code insertion across three
language models and a wide range of insertion layers, as well as trigger-dependent behavior conditioned on a
prompt cue. We further evaluate the modified SAEs using HumanEval and selected SAEBench metrics. While attack effectiveness varies across models and layers, strong backdoor behavior can coexist with relatively small changes in several conventional SAE quality measures. These results establish that SAEs can carry behavioral backdoors without modifying the
language model itself and should therefore be treated as security-sensitive components.
Read the full abstract ▾
All authors Enrico Ahlers, Daniel Passon, Tobias Kiecker, Eik Reichmann, Lars Grunske
Software Engineering
Let the Agent Do It? How Software Practitioners Understand and Make Permission Decisions in Agentic AI Assistants
Larissa Salerno, Haoyu Gao, Gregory Gay et al. (+2 more)
Agentic AI assistants increasingly act on developers' behalf by modifying files, executing commands, and accessing external resources. These actions often require permission, yet little is known about how practitioners make permission decisions while still benefiting from
agent autonomy. To address this gap, we conducted a sequential mixed methods study, interviewing 18 practitioners who use AI
agents and then surveying 115 practitioners based on the interview findings. We find that practitioners often understand
agent behaviour through what they can directly observe and review, while decisions, data use, and other activity behind the scenes remain less clear. This uncertainty also shapes permission decisions, which depend on the scope and risk of an action, whether it fits the task, familiarity with the
agent, and the environment in which it operates. Practitioners respond by adjusting how closely they oversee
agents, from setting limits in advance to monitoring execution and reviewing work afterwards. How much scrutiny they apply depends on factors such as trust, task importance, time pressure, and the consequences of an action. Our findings suggest that permission systems should make consequential actions easier to review, distinguish what an
agent is allowed to do from what the user intended, make reversibility clearer, avoid treating repeated approvals as stable preferences, and distinguish rejecting a single action from rejecting an entire approach.
Read the full abstract ▾
All authors Larissa Salerno, Haoyu Gao, Gregory Gay, Alexander Serebrenik, Philipp Leitner
Software Engineering
Experimentation Practices in Indie Game Startups
Elizabeth Bjarnason, Johan Linåker and Fabian Fagerholm
Small independent (indie) game startups face fierce competition when developing games with limited time, budget, and resources. Continuously testing and experimenting with new ideas, features, and content is essential to validate and guide development toward a viable and successful game. Such continuous experimentation (CE) requires access to users and user data, which is often very limited in the early stages of development, both in general and, in particular for indie game companies. We provide insights and recommendations on how to perform experimentation and game testing in pre-release stages of indie game development based on an exploratory interview survey with 10 independent game developers from different companies. We provide a framework for CE that highlights key parts to consider when planning and implementing an experiment. Current practices for experimentation within indie game development are also described for the early development stages of ideation, preproduction, feature production, and content production. Our empirical findings offer insights and recommendations into how to conduct experiments during pre-release stages of game development, specifically for Indie game companies. Other software startups, particularly those with a value proposition centred around interactive and satisfying user experiences, can also benefit from our research.
Read the full abstract ▾
Software Engineering
AgentSpy: Making AI Agent Behavior Observable
Christoph Bühler, Matteo Biagiola, Luca Di Grazia et al. (+1 more)
AI
agents built on large
language models (LLMs) run shell commands, read and write files, and reach the network, typically with their user's privileges. However, what an
agent does during an execution is difficult to understand: tests assert on the result, and the
agent's trajectory records only what the
agent reports about itself, which may omit behavior executed by its subprocesses. We present AgentSpy, an approach that observes an
agent from outside the
agent. AgentSpy runs the
agent in an isolated environment, configured by a declarative specification, and records the system calls and network traffic of the
agent and of every process it executes. Based on this monitoring, AgentSpy supports two families of analyses: conformance analyses, which measure obligations, i.e., what an
agent execution should do, and safety analyses, which check prohibitions, i.e., what an
agent execution must never do. We instantiate one analysis of each family. The reliability analysis uses rules to summarize each run by the environment resources the
agent uses: the commands it executed, the files it accessed, and the hosts it contacted. The security analysis applies deterministic rules to the system calls of an execution. For reliability, we evaluated AgentSpy on 77 tasks with the codex harness and three recent LLMs, executing each task three times. Sets of repeated runs of the same task are more similar than sets that include runs of another task in 92.2% of the comparisons. Among tasks for which all three runs pass outcome-based tests, the
agent performs task-unrelated activities in 18% of the cases, reads the grading files in 7%, and does not use the developers' guidance in 17%. For security, the generic rules of AgentSpy detect four of five attack categories we considered, with no false positives across 50 runs.
Read the full abstract ▾
All authors Christoph Bühler, Matteo Biagiola, Luca Di Grazia, Guido Salvaneschi
Software Engineering · Cross-listed from cs.OS
No More Translation at Runtime: LLM-Empowered Static Binary Translation
Zhibo Liu, Huaijin Wang, Wai Kin Wong et al. (+2 more)
While AArch64 CPUs are becoming strong
market contenders, their software ecosystem lags behind the mature x86-64 environment, hindering the adoption of the new architectures and impacting user experience. Binary translation bridges this divide by converting binary code from one architecture (e.g., x86-64) to run on another (e.g., AArch64), allowing legacy software to benefit from modern hardware's performance and energy efficiency advantages. Current translation methods are typically either dynamic, which adds significant runtime overhead, or static, which struggles with reliability due to the inherent complexities of binary analysis. This paper introduces a new static, assembly-to-assembly translation paradigm that transforms binary code ahead of execution, generating portable, efficient native-like binaries that run on AArch64 devices without runtime frameworks. Benefiting from recent breakthroughs in large
language models (LLMs), we provide a practical and automated translation engine that produces high-quality code with minimal human intervention. To ensure correctness, we introduce a crucial verification step, where we split the assembly code into simplified snippets, enabling efficient and scalable semantic verification. Our evaluation shows that this approach significantly outperforms existing open-source solutions with a large margin, producing binaries with near-native performance. Furthermore, it shows substantial improvements over the leading industrial translator, ExaGear, illuminating a promising new direction for cross-architecture binary translation research.
Read the full abstract ▾
All authors Zhibo Liu, Huaijin Wang, Wai Kin Wong, Daoyuan Wu, Shuai Wang
Software Engineering
From Trust-by-Design to Continuous Trust in the Era of Artificial Intelligence
Peter Fettke and Wolfgang Reisig
Trust is crucial in the digital world, particularly in the age of systems based on artificial intelligence. Traditionally, system correctness is demonstrated by proving that the system satisfies the required properties. However, digital systems involve humans, who can make mistakes, break rules, or even intentionally cheat the system. Technical devices may also malfunction or cause errors. Therefore, external stakeholders need to trust a system throughout its operation. To address this challenge, we propose the concept of continuous trust. In this paper, we present a detailed explanation of this concept, based on a running example of a business's refund department. The paper concludes with a discussion of related work, such as compliance checking and certification, as well as an outlook on further questions regarding the formalization and the necessary tool support.
Read the full abstract ▾
Software Engineering
Mechanizing the User's Eye: Pre-Registered Deployment of a Sabotage-Validated Fail-Plausible Observer in a Production LLM Agent Runtime
Wei Wu
A prior longitudinal study of silent failures in a production LLM
agent runtime (arXiv:2606.14589) found that about 70% were discovered by a human looking at the product as a user while thousands of tests and governance checks stayed green, and posed mechanizing part of what the human eye does as an open problem. This paper reports our attempt. We built an automated user-viewpoint observer targeting the most dangerous class, fail-plausible failure, in which an internal error becomes fluent, plausible output to the user. It is a two-layer pipeline: five deterministic signals distilled from incident postmortems escalate to an LLM judge whose verdicts must cite verbatim evidence or be discarded. Ground truth is 24 labeled production postmortems with explicit honesty boundaries: 16 of 24 are structurally invisible to any content-reading observer, and we say so. Offline, the deterministic layer achieves 6/6 regression detection with 0/4 false positives, each detector proven load-bearing by sabotage; held-out recall on novel patterns is 0/4. It is, so far, a regression engine. Deployment followed pre-registration: shadow mode caught and retired one systematic false positive on its first run, the 26-day shadow window then ran clean, flip criteria were fixed before the shadow data was read, and the analysis protocol was frozen before the enforcing-mode window opened. That window (12 observed days) fired zero verdicts; per the pre-registered path we report live precision as undefined rather than narrating quiet as success. The observer also produced ten silent failures of its own, confirming that the judge inherits the taxonomy it judges. We release the corpus, detector, and scorecard as a runnable bench: mechanization today retires the human's regression scanning so the eye can specialize in novelty; prediction remains open but is now measurable under frozen rules.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.LG
Technical Report on the Turba Fertilizer Machine Learning Stack in Morocco
Abdelghani Belgaid, Zakaria Mahmoud, Fahd Chibani et al. (+3 more)
Site-specific fertilizer recommendation systems adapt nutrient advice to location, soil properties, crop type, and production targets, but scientific reuse is constrained when recommendation functions remain accessible mainly through interactive interfaces, outputs are not versioned, and trained approximations cannot be independently loaded or benchmarked. This technical report presents the Turba fertilizer machine learning stack, a three-layer open-source implementation for reproducible site-specific fertilizer recommendation in Morocco. turba-client provides programmatic access to publicly accessible site profiles, crop-specific target-yield spaces, and N, P$_2$O$_5$, and K$_2$O recommendation workflows; turba-data distributes analysis-ready snapshots; and turba-models packages crop-specific machine learning surrogates of recommendation outputs. The architecture links upstream
retrieval, versioned analytical snapshots, reproducible cross-model benchmarking, and loadable offline surrogates while preserving the distinction between recommendation-system outputs, observed agricultural data, and model-generated predictions. The first
dataset was constructed from 44,096 unique ESA WorldCereal locations. Scenario expansion across supported cereal workflows generated 132,017 crop-location recommendation requests under a medium target-yield setting. The resulting 22-variable
dataset spans 10 regions, 66 provinces, and 1,149 communes. Nine regression families were evaluated under a fixed deterministic 80/20 protocol, and the current release packages five best-performing crop-specific models. The machine learning task is recommendation-function emulation rather than prediction of observed crop response. The stack provides a reproducible basis for spatial and temporal validation, uncertainty estimation, field-trial comparison, and future integration with additional data.
Read the full abstract ▾
All authors Abdelghani Belgaid, Zakaria Mahmoud, Fahd Chibani, Oumnia Ennaji, Younes Boudoul, Dounia Rachid
Software Engineering
Adopting InnerSource in a Public Sector Organization: A Case Study
Muhammad Usman, Johan Linåker, Deepika Badampudi et al. (+1 more)
InnerSource (IS) is increasingly viewed as a way to improve large-scale software development. However, evidence on its adoption in public sector organizations remains limited. This paper reports a study of IS adoption in a Swedish Public Sector Organization. Drawing on nine semi-structured interviews and an internal workshop with 13 participants, we examine motivations, challenges, and improvement strategies for IS adoption. The results show that the main motivations are improving reuse, strengthening cross-team collaboration, and enabling faster and more cost-efficient development. Key challenges relate to a lack of an open culture and mindset, management of reusable assets (e.g., ownership, documentation, and code visibility), and ways of working (e.g., siloed practices and limited time for contributions). The study also identified improvements to address challenges, including fostering an enabling culture and mindset for IS and knowledge sharing, strengthening management support, and establishing internal platforms and governance for reusable assets.
Read the full abstract ▾
All authors Muhammad Usman, Johan Linåker, Deepika Badampudi, Hannes Salin
Software Engineering
PreMaQ: Predicting Maintainability-Related Quality of LLM-Generated Code Before Generation
Taiki Wakamatsu, Yuta Ishimoto, Masanari Kondo et al. (+1 more)
As large
language models (LLMs) become increasingly capable of code generation, adopting generated code in software development requires assessing not only its functional correctness but also its maintainability-related quality. If such quality could be estimated before generation, developers could avoid the cost of generating, reviewing, and discarding low-quality code. Although prior work has shown that the functional correctness of the LLM-generated code can be predicted in advance, it remains unclear whether maintainability-related quality is similarly predictable. We introduce Pre-Generation Maintainability-Related Quality Prediction (PreMaQ), which predicts the Code Smell Score (CSS) and Maintainability Index (MI) of generated code from the internal representations of LLMs before generation. Our evaluation covers four open-weight LLMs and four Python code generation
benchmarks, comprising 2,695 tasks in total. Our results show that predicted CSS and MI consistently correlate with their observed values across all 16 model-
benchmark combinations, achieving mean Spearman rank correlations of 0.57 and 0.65, respectively. When used for model selection, PreMaQ achieves 59.5% of the maintainability-related quality improvement attainable by an ideal maintainability-based selector over random selection on tasks for which multiple models generate functionally correct code. Combining predictions from PreMaQ and
prompt embeddings increases this proportion to 60.7%, indicating that the two signals are complementary. These findings suggest that PreMaQ can be used to predict and improve the maintainability-related quality of LLM-generated code.
Read the full abstract ▾
All authors Taiki Wakamatsu, Yuta Ishimoto, Masanari Kondo, Yasutaka Kamei
Software Engineering
Finite State Machine-Based Traffic Orchestration for Multi-Engine Analysis Platforms
Ankit Kumar and Thirumoorthi Thangamani
Modern application-security platforms route live traffic from a single proxy into several independent analysis engines – behavioral anomaly detection, authentication analysis, resource discovery, access-control inspection, among others. Each engine's appetite for traffic changes on its own schedule, yet the proxy can honor only one forwarding instruction per resource. We present a traffic-orchestration method that models the combined needs of all engines as a single composite finite state machine (FSM) whose state space is the Cartesian product of per-engine states. Every reachable composite state pre-maps to one forwarding configuration, so runtime reduces to a lookup rather than a negotiation. We add two refinements: time-based expiry treated as a first-class FSM transition enabling declarative traffic decay, and per-state multi-tier outputs that let one machine serve every service level. The design is fully declarative, engine-agnostic, and scales to N engines without re-architecture.
Read the full abstract ▾