Software Engineering
Silent Success: A Release Gate That Passed on Checks It Never Ran, and Eight More
Dong Hyeon Jeon
A blocking quality gate in a production release pipeline reported PASS on a run in which one subgate had executed neither of its two checks and another had run six of its eight. Both keys that decided the run asked whether a violation had been observed, and both computed that from a population already stripped of the cases that failed to run, so absent data answered "no" and a run that checked almost nothing scored perfectly, for two weeks of green builds. Introducing a third value, pass, violate, and unable to determine, turned those silent passes into failures and put the shortfall into the exit status; separate work on the same specifications then found a detector firing with a margin of 0.000177 percentage points, its firing floor holding at the sixth decimal place. Eight further instances of the same form followed: five more from the same engagement, two in open-source projects, a gateway where a configured cache TTL was declared but never applied on the write path, and an inference server whose count of free cache slots was credited on every step for releases that returned nothing, and one committed by the author while writing this paper, using tooling built to prevent exactly it. What the nine cases are offered for is not the novelty of the category but the convergence of the remedy, nine different missed questions, one kind of act, seven of the nine answerable by a single query, command, or comparison, and that convergence is falsifiable, which is what this paper asks to be judged on.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.DC
Distributed Service Orchestration in Edge-Cloud Continuum for Digital Healthcare
Johirul Islam, Hafiz Faheem Shahid, Ijaz Ahmad et al. (+3 more)
Today's digital healthcare services rely on various applications and functions that must be continuously accessible. Cloud computing enables global access to these services through public networks, which often also introduce increased latency, higher bandwidth consumption, and additional security risks compared to local operation. Edge computing mitigates these challenges by deploying cloud services closer to the end users and data sources, thereby improving resilience to network disruptions and reducing latency, bandwidth usage, and exposure to security threats. However, service deployment typically relies on the availability of centralized registry servers. Consequently, if the network connection or the registry server itself becomes unavailable, service deployment at the target edge node may fail. To address this, we propose a three-tier registry architecture to enhance deployment reliability and service availability, considering DockerHub as a remote public registry, an MEC-based off-premises registry as a remote private registry, and a LAN-based on-premises registry as a local private registry. The performance and efficiency are analyzed through measurements related to the estimated deployment time, inflicted network and computational load, and energy consumption, while the required nanoservices are deployed from different tiers. The experimental results indicate that alongside the improved tolerance to network disruptions, the local private and remote private registries also outperform the remote public registry in the deployment performance. The findings highlight the effectiveness of proximity-aware service distribution in improving the resilience, performance and efficiency of service deployment.
Read the full abstract ▾
All authors Johirul Islam, Hafiz Faheem Shahid, Ijaz Ahmad, Tanesh Kumar, Ayan Mondal, Erkki Harjula
Software Engineering · Cross-listed from cs.AI
RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?
Mithil Salunkhe, Haochen Ding, Samridhi Verma et al. (+1 more)
Reproducing a machine learning paper involves most research steps, from installing software and debugging to running experiments, work that AI
agents increasingly do. We introduce RECLAIM, a
benchmark of 100 NeurIPS 2025 papers that can be rebuilt yearly from new conferences. For each paper we fix in advance the result to reproduce, what counts as a successful reproduction, and a GPU-hour budget. An
agent must reproduce that result using the paper and whatever its authors released. What the authors released decides the difficulty tier. Run-tier releases include code, data, and weights; Retrain-tier releases lack weights, so the
agent trains the model; Reimplement-tier releases lack code, so the
agent writes it. A separate
language model grades runs from logs and outputs rather than
agents' reports. We run four
agents once per paper; the best
agent in each tier reproduces only 41% of Run-tier papers, 27% at Retrain, and 15% at Reimplement, where every
agent does worst. Failed attempts use on average 29% of their budget, so most stop with budget left. The most common
agent error is writing the method without checking any part against the paper's numbers, in 63 of 400 runs.
Read the full abstract ▾
All authors Mithil Salunkhe, Haochen Ding, Samridhi Verma, Volodymyr Kindratenko
Software Engineering · Cross-listed from cs.PL
Pack Iteration in Swift: Ordinary Control Flow for Variadic Generics
Serafima Nerush and Eitan Frachtenberg
Variadic generics are a powerful tool for type-safe meta-programming. Yet in most widely used
languages, they remain an "expert-only" feature due to their reliance on complex patterns such as recursive decomposition or expansion expressions that do not compose naturally with ordinary control flow. In C++, for example, accessing elements of parameter packs has traditionally relied on unintuitive recursive patterns that "peel off" elements. This paper presents Pack Iteration, a feature introduced in Swift 6.0 that allows developers to iterate over parameter packs using a familiar, imperative for-in loop. By treating pack expansion as a first-class source for iteration, Swift bridges the gap between high-level expressiveness and advanced generic programming. We detail the design and implementation of this feature within the Swift compiler, focusing on the challenges of bridging static type-checking in the constraint system with dynamic execution in the Swift Intermediate
Language. Unlike traditional models that expand packs at compile time, Swift's implementation supports on-demand evaluation, enabling efficient dynamic iteration and short-circuiting control flow. Our empirical evaluation confirms that pack iteration provides performance comparable to - and sometimes significantly better than - the complex workarounds previously required.
Read the full abstract ▾
Software Engineering
Category-Based MLM: Unifying Powertypes with Superclasses
Mira Balaban, Azzam Maraee and Arnon Sturm
MultiLevel software Modeling (MLM) suggests that conceptual modeling in broad subject domains might require abstraction of multiple classification levels. The MLM approach relies on philosophical arguments, claiming that faithful modeling of real-world domains involves repeated type classification as in ontologies of natural kinds. MLM leveled architecture is interwoven and defined by instance-of interlevel relationships between clabject classes in lower levels to classes termed category classes, in upper levels. The instance-of relation denotes membership of clabjects as type objects in their (powertypes) category classes, and is not transitive. All MLM approaches support forms of deep characterization, i.e., category classes can influence classes in lower levels. Deep characterization is an essential feature of superclasses and contradicts the non-transitive membership meaning of instance-of. In this paper, we introduce the Category-Based MLM (CatMLM) model, in which category classes have dual superclass and powertype facets, based on the distinction between category features that do not participate in deep characterization, and object features that do. This distinction clarifies the role of levels, provides a clear quantifiable criterion for leveling, and yields a decision rule between the subclass and instance-of relations. The contribution of this paper is to introduce a well-defined MLM model that (1) is based on simple, quantifiable level decisions; (2) clarifies how leveling emerges from domain needs; and (3) analyzes gains and losses of MLM vs. plain OO modeling.
Read the full abstract ▾
Software Engineering
Soundness Checking of Taint Flow Models
Samarth Kishor, Victor Nicolet and Joey Dodds
Existing state-of-the-art static taint flow analyses for imperative programming
languages can scale to large applications by using precise user-provided taint flow models of library methods. However, manually and precisely modeling a method's taint flows is tedious and potentially unsound. Furthermore, automatically modeling the method via an inter- procedural taint analysis can be inefficient. To solve this problem, we propose a guess-and-check approach: (1) an LLM
agent that generates a precise taint flow model of a method and (2) a symbolic
algorithm to check the soundness of the model. The
algorithm deduces which taint flows must not occur in the method for the LLM's taint flow model to be sound, and uses lightweight static analyses (e.g., type system and pointer analysis) to prove these must-not-flows. When these analyses are insufficient, the
algorithm deduces maximally-general callee models and recursively verifies their soundness, avoiding a full inter-procedural taint analysis in most cases. Since a more precise model requires fewer must-not-flows to be verified, the precision of the LLM's model directly determines the efficiency of our approach. We evaluate our approach on 97 LLM-generated taint flow models for methods in 6 large Go codebases and prove the models sound for 93% of the methods they cover. The proven-sound LLM-generated models are also precise, resulting in no new false-positives when proving taint flow properties.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CY
Towards a Platform for Mastering Personal Sovereignty
Judith Michael, Bernhard Rumpe, Antonio Bucchiarone et al. (+1 more)
The ongoing digital transformation of work, administration, health, mobility, and social interaction is profoundly reshaping everyday life, steadily shifting control from individuals to large platform providers. Although data is often labeled the "gold of the 21st century", its real value is realized through services that access, combine, and exploit it. Today, individuals have little sovereignty: life events (e.g., changing an address, insurance, job, or marital status) require fragmented, repetitive interactions across numerous systems, leaving users overwhelmed rather than empowered. We argue for a fundamental rethinking of this situation and propose a virtual, trustworthy platform, called "MyVirtualME" that acts on behalf of the human individual towards companies, administrations, and other actors. This platform centers services, data, permissions, and data usage around humans, not foreign actors, allowing genuine control and transparency. Our
vision goes beyond a pure data focus: supported by trusted services ranging from basic notifications to intelligent assistants, MyVirtualME aims to reduce digital bureaucracy, improve and even automate interactions, and support informed oversight of an individual's health, financial, and administrative situation. We firmly believe that a human-centered MyVirtualME is essential for restoring personal sovereignty in our digital future.
Read the full abstract ▾
All authors Judith Michael, Bernhard Rumpe, Antonio Bucchiarone, Dominik Bork
Software Engineering · Cross-listed from cs.CR
CONCURDEP: Event-Guided Analysis of Dependency Invalidation in CPython Concurrency
Baihong Chen, Hadley Westover and Wen Li
Removing CPython's Global Interpreter Lock (GIL) exposes native code to concurrency absent from ordinary C types. Mutation or re-entry can revoke a borrowed object, storage pointer, traversal state, or lease between acquisition and use while its owner remains alive, causing native
memory errors and runtime-state corruption. Race analyses track conflicting accesses. Python/C lifecycle analyses track individual object states. These reporting units leave implicit owner-subject-storage relations disconnected from later uses under parallel and re-entrant events. We present CONCURDEP, a source-level static analysis of dependency invalidation. Its key insight is to represent the runtime property a native use requires and ask which target-matched event can revoke it within the dependency's live region. CONCURDEP recovers runtime-semantic dependencies, connects them to events through an event-aware native concurrency dependency graph, and applies property-specific state and protection transfers through a shared engine and six mechanism plugins. CONCURDEP correctly classifies all 180 matched semantic-conformance cases and analyzes each of three production CPython releases with five-run medians of 22.13-27.66 seconds and 670-784 MiB peak resident
memory. Source auditing confirms 1,094 of 4,273 unique production fingerprints (25.60% confirmation yield). Removing derived relation recovery loses 25-44 represented roots per release; removing cross-entry events loses 73-94. The study identifies 144 distinct bugs across free-threaded and conventional-GIL builds, including 95 previously unreported in public sources. These results show that explicit dependency, event, and property semantics expose consequential runtime failures across API boundaries and execution modes at release scale.
Read the full abstract ▾
Software Engineering
Developing a Unified Verification and Validation Activity Standard at JPL
Maximilian Vierlboeck, Ellen Van Wyk, Natalia Sanchez et al. (+6 more)
Verification and validation practices (V&V) at NASA's Jet Propulsion Laboratory (JPL) have diverged over the past decade, creating fragmentation that increases overhead, reduces cross-project efficiencies, and inhibits institutional knowledge transfer. We present a unified V&V activity schema developed through human-centered design workshops involving 29 practitioners across multiple mission types and disciplines. The schema builds on a relationship-based architecture that allows for separating methods (Test, Analysis, Inspection, Demonstration, and Review of Design) while maintaining a common attribute set. Formalized as a platform-agnostic SysML model, the schema defines bidirectional relationships between requirements, V&V activities, venues, and evidence. Implementation in JPL's Jama platform demonstrates controlled customization through templates and modular item types, balancing rigor with agility while enabling automations, pattern reuse, and digital thread integration.
Read the full abstract ▾
All authors Maximilian Vierlboeck, Ellen Van Wyk, Natalia Sanchez, Bogdan Oaida, Torrance James Eberhart, Marie Piette Gomez, Christopher Delp, Marijke G. Jorritsma, Alex Desharnais
Software Engineering · Cross-listed from cs.CR
Agent Approval Laundering: Transitive Effects Beyond the Approved Invocation
Jinqian Zhang, Haojun Xia, Shujiang Wu et al. (+4 more)
Coding-
agent approval interfaces bind a human decision to a command or tool call, while developer tools execute the transitive workflow that invocation activates. Package installation can run lifecycle hooks and write files; an MCP call can exercise network authority. We call the resulting record-coverage failure approval laundering: the durable record names the entry invocation but omits effects exercised by its workflow. We present the first systematic security analysis of this record-to-closure relation in
agent systems. We formalize closure-bound approval over six effect classes and derive an information limit: identical policy-visible fields can require different effect-specific decisions, so no record-only policy can guarantee both. The Approval-to-Action Security
Benchmark binds approval objects and decision-time metadata to post-execution evidence. Across 111 fixed approval-object/trace pairs, residual records fall from 40 under explicit fields to 17 with command semantics and 13 with decision-time metadata. Across 11 fixed-SHA executions, the ladder reaches zero metadata residuals; two exact mappings recur across three product frontends. For prospective recovery, effect-bound records commit frozen, source-backed predictions and provenance before authorization. On 17 prespecified holdout workflows, predictions achieve 0.926 macro recall and 0.941 macro precision; binding them cuts residual effects from 10 to 3. A Claude Code PreToolUse integration carries the frozen record through the permission path without automatic approval. These results establish approval laundering as a measurable, recurrent record-coverage failure despite truthful invocation identity. They motivate binding each invocation before authorization to a source-backed prediction of its workflow's transitive effect boundary and preserving that binding with the decision.
Read the full abstract ▾
All authors Jinqian Zhang, Haojun Xia, Shujiang Wu, Jingkun Yue, Xia Zhang, Zhangpei Cheng, Bibo Tu
Software Engineering · Cross-listed from cs.LG
Time-Series Foundation Models That Understand Data Revisions
Taimoor Ahmad
Historical observations are not always fixed: statistical agencies revise previously published values as new evidence arrives. Forecasting from a contemporary download can therefore expose a model to information unavailable at the date it purportedly made a prediction. We propose VINTAGE-TS, a revision-aware adaptation of a time-series foundation model that distinguishes observation time from information-availability time. Its targets are the next period's first-published value and the value available a fixed number of days after that publication; neither is declared final truth. A joint predictive distribution preserves dependence between these targets and exposes uncertainty about their difference. We specify an ALFRED-based rolling evaluation, a matched Chronos-2 comparison, conventional and revision-aware baselines, and a separate audit of pretraining overlap. The accompanying software implements validity-interval reconstruction, delayed-label filtering, a frozen-backbone adapter interface, and reproducible diagnostics. An executed synthetic demonstration and a 25-configuration sensitivity suite verify the workflow, expose variation across seeds and revision regimes, and illustrate how hindsight contamination changes measured performance. Thirty one automated tests check temporal and integration contracts. Real ALFRED and Chronos-2 experiments have not been executed; no empirical foundation-model advantage is claimed.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CR
Where Cyber Agents Struggle: Bottleneck Analysis of Multi-Stage LLM Agents
Saeedeh Lohrasbi, Mohammad Mamun, Ahmed Yehia et al. (+2 more)
Multi-stage LLM-based cyber
agents may complete attack workflows while remaining brittle, costly, or reliant on incorrect interpretations of execution evidence. Success rates alone obscure inefficiency, adaptation through retries, and recognition of success or failure. We present an end-to-end diagnostic study of an Autonomous Adversary system with orchestrator, executor, and validator LLMs in enterprise-like lateral-movement scenarios. Six frontier models are evaluated across two scenarios and three modes: expert-defined, self-scaffolded, and fully autonomous. We assess validator consistency and evidence grounding; introduce a subtask-conditioned, cost-aware score for abnormal
token use, retries, and runtime; and use comparative LLM-as-a-Judge analysis to identify planning deficiencies, including tool misalignment, plan similarity, over-specification, inadequate probing, and weak recovery. Validators are generally relevant and evidence-grounded but often nonspecific and overly optimistic. Bottlenecks cluster in credential and lateral-movement tasks, spread with scenario complexity, and vary more under full autonomy. Reliable evaluation must assess outcomes, evidence interpretation, resource use, and adaptation after failure.
Read the full abstract ▾
All authors Saeedeh Lohrasbi, Mohammad Mamun, Ahmed Yehia, Scott Buffett, Sherif Saad
Software Engineering
Pretraining and adapting a language model on a dependency-free stack: GPT-2 124M from random weights, reproduced against llm.c, and a clinical adapter for Qwen3-0.6B
Thang Tran and Lan Dang
Almost every
language model in service was trained by one family of software. That concentration makes a question hard to settle: how much of what is known about training a
language model describes
language models, and how much describes that software? Settling it needs a second implementation able to carry a model through a whole lifecycle rather than reproduce one operator. We report such a lifecycle. Using numbat, a machine-learning stack written in Zig with no third-party runtime dependencies, we pretrain a 124.4 M-parameter GPT-2 from random initialisation over 9.91 B
tokens of web text, then adapt a separate small model to clinical question answering. A reference implementation runs on identical hardware at both stages, and a sidecar with authority to halt a run supervises each. Agreement is close. Held-out cross-
entropy finishes at 3.2588 against a published 3.29, and HellaSwag at 0.3053 against 0.299; across 8 paired evaluations it sits below a same-machine reference at every point, by 0.0608 on average. Re-running that reference on different hardware moves it 0.0035, which bounds how much of any gap is method rather than framework. Throughput does not pay for agreement: measured in one session at a production configuration, numbat reaches 43,374
tokens per second against PyTorch's 41,202, scaling 2.769x over three cards. Clinical adaptation ends at 2.1899 held-out loss against 2.1941. Neither model is a medical device, and neither is validated for clinical use.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CR
Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behavior
Chuyi Wang, Xiaohui Xie, Tongze Wang et al. (+2 more)
LLMs increasingly operate through coding-
agent harnesses that inspect repositories, invoke tools, and modify files. Substituting the model behind such an
agent can therefore change security-relevant decisions, including whether it verifies changes or recovers safely from failures. Existing LLM fingerprints largely infer identity from direct text or
token distributions. In coding
agents, these signals are mediated by system instructions, controller logic, tools, and execution feedback, limiting their transfer. We present
LIDAR (LLM Identification from Decisions and Actions at Runtime), an active black-box fingerprinting method for coding-
agent execution. Three coding probe pairs expose post-edit verification, transient-failure recovery, and specification–test conflict resolution under controlled changes.
LIDAR represents the resulting trajectories with complementary instance-level and distribution-level features and compares them with clean references using a lightweight probabilistic identifier. It requires no access to model weights, logits, or provider internals. Across 36 models from seven families and two
agent harnesses,
LIDAR achieves high Top-1 accuracy and MRR and outperforms four existing fingerprinting and API-auditing baselines. Ablations confirm that the two feature levels, all probe pairs, and their controlled variants contribute. These results show that
agent execution behavior provides model-identity evidence beyond final outputs.
Read the full abstract ▾
All authors Chuyi Wang, Xiaohui Xie, Tongze Wang, Fangchen Luo, Yong Cui
Software Engineering
Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband et al. (+3 more)
Large
language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing repository-level QA
benchmarks mainly evaluate static code understanding and often rely on LLM-based evaluation, while execution-
reasoning benchmarks are mostly limited to snippets or functions. We introduce SWE-Flux, a repository-level
benchmark for dynamic execution
reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs. The
benchmark covers singletest and multi-test questions over control flow, loops, program state, dataflow, exceptions, and program invariants. Evaluating five LLMs shows that this task remains challenging. The best model achieves only 37% accuracy. Models perform better on localized behavior such as invariants, intra-procedural control flow, exceptions, and simple loops, but struggle with dataflow, inter-procedural execution, precise state
reasoning, and suite-level aggregation. Finally, we show that the oracle-harvesting pipeline can generate fresh
benchmark variants using input perturbation. It successfully harvests valid variants for almost 90% of the selected instances, and the resulting variants are substantially more challenging for the evaluated models.
Read the full abstract ▾
All authors Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati
Software Engineering
Entangle: Uncovering Collaboration in the GitHub Quantum Software Ecosystem
Angel Luis Lara-Martín and Ricardo Pérez-Castillo
Quantum computing is moving from research laboratories towards early commercialization and broader socio-technical adoption, supported by sustained hardware progress and a rapidly expanding open-source software ecosystem. This momentum is especially visible on GitHub, where many
quantum and hybrid software projects coexist around frameworks such as Qiskit, Cirq, PennyLane and Amazon Braket. However, this ecosystem remains fragmented, making it difficult to understand who shapes
quantum software, where expertise is concentrated, how collaboration flows across organizations and disciplines, and which actors connect otherwise separated communities. This paper presents Entangle, a data-driven analysis of the open-source
quantum computing ecosystem on GitHub. Starting from 71 domain keywords, Entangle identifies more than 1,500
quantum repositories, 27,000 contributors and 400 organizations, revealing an ecosystem strongly organized around four leading industrial vendors, but also supported by 2,387 contributors who connect projects, organizations and domains. These findings provide practical evidence for responsible
quantum innovation by making visible patterns of influence, dependency, collaboration and knowledge transfer. They also offer actionable indicators for strategic decisions on investment, hiring, partnerships, ecosystem stewardship and capacity building. More broadly, Entangle shows how open-source intelligence can support a more transparent, measurable and governable
quantum software ecosystem, helping align technical development with responsible innovation, public–private coordination and long-term sustainability.
Read the full abstract ▾
Software Engineering
From Agent Output to Authorized Transition
Christopher Koch
Agentic engineering systems can edit repositories, run tools and tests, build firmware, synthesize schematics, and prepare deployable or manufacturable artifacts. The assurance problem is therefore shifting from whether an
agent can produce an output to whether an engineering lifecycle is justified in acting on claims about that output. Current products and standards provide sandboxes, approvals, hooks, traces, policy enforcement, attestations, bills of materials, and assurance representations, but these capabilities remain fragmented. This paper presents the Agile-V Assurance Spine, a cross-domain transition contract for software, firmware, and PCB engineering. Evidence is admitted only when it establishes required properties through an authoritative source profile, is bound to the exact artifact and frozen policy baseline, remains current with respect to declared dependencies, and satisfies risk-appropriate independence and authority. Gate decisions are recorded as receipts; approvals and exceptions are exact-scope and time-bounded; and authorization is rechecked at the effect boundary before merge, deployment, flashing, release, or fabrication. A bounded review of contemporary research, commercial platforms, open-source infrastructure, and standards positions the model relative to evidence-gated lifecycle control, continuous assurance, runtime admission, provenance, and AI/ML inventories. The paper contributes a precise vocabulary, compositional architecture, domain profiles, mapping to open-source implementations, and an
adversarial evaluation agenda. It does not claim regulatory conformity or demonstrated production superiority.
Read the full abstract ▾
Software Engineering
Scenario-Driven Neuroevolution: Using Models to Guide Test Generation for Games
Gijs van Cuyck, Patric Feldmeier, Jan Tretmans et al. (+1 more)
Automatically generating test inputs for games is challenging, as test generators must master the game to reach advanced program states while also ensuring robustness against the heavy program randomisation inherent to games. The test generator Neatest therefore optimises test suites consisting of neural networks that reach advanced program states and are robust to program randomisation, as they generate test inputs dynamically based on the current program state. Neatest is a white-box testing approach that aims to generate a network
agent for each yet-uncovered statement or branch of the code using neuroevolution. Due to this iterative test generation approach, the
algorithm does not scale well to larger programs that may contain thousands of branches. Furthermore, covering every statement or branch in a game often does not correspond to playing the game as intended. To alleviate these shortcomings, we propose combining Neatest with a model-based testing approach that allows game testers to define test scenarios via abstract game models. The test generator then no longer optimises networks to reach all branches or statements of a program, but instead trains networks to replicate the concrete desired testing behaviour expressed by the abstract game model. An evaluation on 13 Scratch games across varying genres demonstrates that Neatest, combined with model-based testing, is able to optimise
agents that replicate the desired gameplay behaviour defined in the game models while increasing achieved branch coverage by 7% compared to the traditional code-guided Neatest approach.
Read the full abstract ▾
All authors Gijs van Cuyck, Patric Feldmeier, Jan Tretmans, Gordon Fraser
Software Engineering · Cross-listed from cs.CL
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
Makar Ulesov, Vladislav Smirnov, Omar Ibrahim et al. (+1 more)
Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96-item paired
benchmark in which every flawed backtest has a clean control that holds strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. A deterministic scorer separates flaw recall, clean-control false positives, evidence localization, and fix relevance. Over 1440 cached audits from four text endpoints, the primary DeepSeek auditor reaches 100.0% closed and clean-aware code recall, but open
prompts over-flag 93.8% of clean code controls, and clean-aware all-three specificity is 87.5% even where recall saturates. A clean-aware warning drops DeepSeek code false positives from 20.8% (95% CI 11.7–34.3) to 0.0% (0.0–7.4) at unchanged recall, while the budget anchor still flags 38/48 clean controls under the same
prompt. Reporting recall alone would rank three of these four models identically; reporting the clean-control rate separates them by 79 points.
Read the full abstract ▾
All authors Makar Ulesov, Vladislav Smirnov, Omar Ibrahim, Arsenii Bobovnikov
Software Engineering
LLM-Assisted Workflow for Structural Difference Visualization in Evolving Software Requirements
Koi McFarland and Songhui Yue
This paper presents an LLM-assisted workflow for visualizing structural differences in evolving software require- ments. Implemented in the OntologyWeb environment, the work- flow represents baseline and current requirements as triple-based semantic graphs and supports side-by-side comparison of curated graph snapshots. The comparison view aligns matched entities and uses visual encoding to highlight structural changes.
Read the full abstract ▾
Software Engineering
Satisfaction Is Not Explanation: Auditing Vacuity and Training Influence in Temporal-Logic-Guided Reinforcement Learning
Lorenzo Bacchiani
A reinforcement learning policy that satisfies its temporal-logic specification has passed a test, not an assurance argument. The clause that matters to a reviewer may never have mattered to training: it may have been avoided entirely, forced by the environment regardless of what the policy learned, or redundant next to the ordinary task reward. Satisfaction probability and task return cannot tell any of this apart. This paper introduces an audit layer that can. It measures whether a specification clause was actually exercised, whether that role was forced or chosen, and whether the obvious way to test causation, weakening the clause and retraining, is even valid. Often it is not: we prove that comparable weaker/stronger training objectives can share perfect optima under standard acceptance-derived rewards, show related ablation hazards across a large corpus of published specifications, and then show that a properly designed intervention detects the effect it should. Across standard reinforcement learning
benchmarks and published external artifacts, the audit layer separates six regimes that a single satisfaction number collapses into one. A policy that satisfies its specification has answered whether. This paper asks why.
Read the full abstract ▾
Software Engineering · Cross-listed from astro-ph.IM
A ready-to-deploy MLOps software platform for satellite and NEO detection at meter-class ground-based observatories
Rafael Carrillo Navarro, Pablo García-Martín, René Duffard et al. (+2 more)
Ground-based astronomical observations frequently contain streaks produced by artificial satellites, space debris, and potentially Near-Earth Objects (NEOs). While machine-learning models can reliably detect these features, their practical adoption in observatory operations is often limited by the lack of integrated tools for visual inspection, validation, workflow management, and structured data storage. This paper presents the StreakMind Workbench, a framework that applies MLOps practices to bridge research-oriented machine-learning pipelines with routine observatory operations. Rather than introducing new detection
algorithms, the Workbench addresses a software-engineering challenge in astronomical computing: maintaining a single authoritative source of scientific processing code while providing astronomers with an operational environment for workflow execution and result inspection. Integrated with the reference StreakMind AI model of Carrillo et al. (2026) and implemented in Python using PyQt5, the Workbench supports the complete workflow from FITS ingestion to database storage, including inference, result inspection, database exploration, training management, and Minor
Planet Center formatted observations. Validation on 273 images from La Sagra Observatory demonstrates successful end-to-end workflows while maintaining consistency with the underlying StreakMind scientific code. The platform facilitates operational use of a research ML pipeline in meter-class observatories and moderate-scale campaigns, supporting Space Situational Awareness and planetary defence.
Read the full abstract ▾
All authors Rafael Carrillo Navarro, Pablo García-Martín, René Duffard, Juan Luis Orellana Montes de Oca, Oscar Ortega Ríos
Software Engineering
Foundations of Algebraic Architecture Theory: A Rising Sea of Geometry, Transport, Comparison, and Reconstruction
Hiroyuki Nakahata
AI-generated software changes make it increasingly important to determine what a change preserves, where local consistency fails to extend globally, and which alternatives remain. We develop the foundations of Algebraic Architecture Theory (AAT) from Atoms, typed primitive facts, and Laws, equations that objects must satisfy. A reading specifies what counts as structure and which operations and laws to preserve. The main reconstruction
theorem identifies the category of full geometries and all their structure-preserving morphisms with an independently defined category of local models, up to equivalence. Objects are recovered up to isomorphism and morphisms between fixed endpoints uniquely. The theory addresses gluing, diagnosis, transport, classification of changes, and reconstruction. From finite Atom families we construct cores closed under operations and geometries with sites and coefficients. We give conditions under which a Cech obstruction detects the existence of a global state and, through comparison with repair semantics, a global repair. We compare diagnoses and give a finite criterion for uniform invariance given computable finite data. Transport along exact changes has a universal property and commutes with base change on exact pointed pullback squares. Comparisons of routes generated from the same square, finite comparison diagram, and geometry factor into an invertible comparison and an idempotent normalization. We characterize when observations determine comparison preservation and classify compatible lifts. Encodings of lens and protocol semantics preserve and reflect laws and recover semantics-preserving morphisms. Applications classify and count operation-preserving changes and extend morphisms uniquely from finite tables. Corresponding Lean declarations are listed in the appendix.
Read the full abstract ▾
Software Engineering
AST-Based Automated Elimination of break and continue Statements in Java Code
Andrés Juárez, José Chicano and Rubén Saborido
This work presents the development of an automatic refactoring tool for Java code built on top of the Eclipse JDT API. The proposed approach transforms control structures containing break and continue statements within different types of loops into semantically equivalent constructs that avoid their explicit use. To achieve this, auxiliary boolean variables are introduced to restructure the control flow while preserving the original program behavior. The main objective of this transformation is to improve code structure and enable the application of subsequent automated refactorings, particularly those based on the Extract Method operation, which are typically restricted by the presence of jump statements. The implementation relies on the analysis and rewriting of the Abstract Syntax Tree (AST), ensuring semantic equivalence in all addressed scenarios. The tool was validated through 54 manually designed test cases and 151 units tests, all of which produced satisfactory results. In addition, it was applied to 139 methods from seven open-source projects, generating code without compilation errors and preserving the original behavior as verified by the projects' test suites. The results demonstrate that the proposed approach safely automates the restructuring of code containing break and continue statements, facilitating further
evolution and structural analysis.
Read the full abstract ▾
Software Engineering
Unity Insight: A Production Code–Asset Index for LLM Coding Agents in Unity Projects
Shenhua Gu, Hongqiang Zhu, Fan Zhang et al. (+2 more)
LLM coding
agents increasingly operate inside game-engine repositories, where application logic is inseparable from serialized assets: a single gameplay change may span C\# scripts, prefabs, scenes, and ScriptableObjects wired together by Unity GUIDs. The
retrieval tools
agents carry today—shell utilities and code-only indexes—cannot answer basic cross-file questions, because these relationships live in .meta files and YAML assets rather than in code. We present Unity Insight, to our knowledge the first persistent, LLM-facing,
agent-integrated cross-file code–asset index for Unity projects, shipping in production with Tuanjie Codely, the
agent CLI of Tuanjie Engine, since its public launch on 2026-07-28. In a paired experiment—28 project-specific questions on two Unity games, same model and harness, one run per arm per question—the index-backed
agent spent 53% fewer
tokens and 52% less wall-clock time than a general-purpose exploration
agent (exact paired sign tests, $p{<}0.004$), using only its typed index-query tools.
Read the full abstract ▾
All authors Shenhua Gu, Hongqiang Zhu, Fan Zhang, Jinming Zhang, Hao Chen
Software Engineering
FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration
Weihang Ding, Junfei Zhan, Yueting Li et al. (+1 more)
Deployment requires an
agent to turn application code into a running system whose services connect, become ready, and remain observable. FDE-Bench evaluates this capability with 136 deployment-configuration tasks spanning Docker images, multi-service Compose stacks, and Kubernetes, in greenfield and diagnose-and-repair modes.
Agents submit declarative artifacts that are collected, rebuilt, and redeployed in a pristine environment. Four gated binary check layers measure build, readiness, behavior, and conformance to the deployment specification, using programmatic checks without an LLM judge. A four-arm release gate requires a resolving reference solution and rejects tasks solved by do-nothing, specification-transcription, or generic-stub submissions. The released check annotations expose the link between 2,145 checks and their specifications, including seven documented gaps. Three additional
adversarial strategies test shortcuts in the grading signals; none resolves any of the 135 tasks they cover, while a vacuous health probe passes readiness and exposes the need for downstream checks. On the 136-task evaluation grid, seven
language models from four providers use the same four-tool scaffold and resolve 52.9-75.0 percent of tasks. The three zero-intelligence floors resolve none and reach a mean Deployment Score of at most 0.44. Readiness is the largest failure stage, accounting for 110 of 313 unresolved episodes. Mean resolution rate is 30.7 percentage points higher on the repair task group than on the disjoint greenfield group, with a positive gap for every model; ten tasks resist all seven. In a 25-task case study, one practicing engineer directing Claude-Sonnet-5 resolves 92 percent against 72 percent for the autonomous baseline. FDE-Bench links deployment success and failure to artifacts that can be inspected and replayed.
Read the full abstract ▾
All authors Weihang Ding, Junfei Zhan, Yueting Li, Qirong Guo
Software Engineering · Cross-listed from cs.CR
Issuer-Sovereign Agentic Payments
Dishant Sharma, Rajneesh Kaushal and Ashu Kanaujia
AI
agents are beginning to make real payments. Current approaches let an
agent pay by relying on a credential provider that, in the approaches deployed today, typically sits outside the cardholder's bank. The spending rules are then enforced by the card network or that provider, and not by the bank itself. This leaves the issuing bank, which carries the financial risk, with little direct control at the moment a payment happens. This paper describes Issuer-Sovereign Agentic Payments, a method that keeps that control with the issuer. The cardholder approves a spending rule once, and the bank's own authentication component records it. Later, when the
agent pays a specific merchant, the bank checks the merchant against the approved rule and generates the card authentication value only if the merchant is allowed. The payment then travels the normal card rails and is validated by the issuer, with no extra dependency introduced at execution.
Read the full abstract ▾
Software Engineering
Constraint-Driven Context Engineering: Designing Domain Interfaces for AI Systems
Xiwei Xu, Chen Wang, Mengmeng Yang et al. (+6 more)
Generative AI systems are increasingly deployed to address domain problems. These systems operate under technical, regulatory, institutional, and normative constraints that define acceptable AI behaviour and outcomes within their domains. We observe a recurring pattern in our industry engagement: partners often arrive with a functioning but relatively generic AI solution. The challenge is no longer to build an AI system from scratch, but to improve the quality and domain appropriateness of an AI-generated solution. In these settings, the limiting factor is often the quality, scope, and structure of the context available to the system. Yet, existing context engineering approaches primarily focus on supplying domain knowledge through
retrieval,
memory, and tools, with limited support for systematically identifying and operationalising the constraints that govern AI systems in their operational environments. This paper proposes Constraint-Driven Context Engineering (CDCE), a design approach for engineering domain interfaces for AI systems. Drawing on software architecture design and Domain-Driven Design (DDD), CDCE treats domain constraints as first-class design drivers. It identifies and characterises constraints, determines the required context assets, and designs representations through which these assets are made available to AI systems. We conducted a comparative multiple-case study with industry and public-sector partners across educational assessment, healthcare decision support, and financial-distress prediction. Depending on their characteristics, constraints can guide AI behaviour, enforce permissible boundaries, or support verification of AI-generated outcomes. The cases demonstrate CDCE's applicability across contrasting domains and show how constraint characteristics shape the resulting domain interfaces.
Read the full abstract ▾
All authors Xiwei Xu, Chen Wang, Mengmeng Yang, Yipeng Zhang, Jacky Jiang, Suyu Ma, Youyang Qu, Ming Ding, Liming Zhu
Software Engineering
FairTest: Search-Based Fairness Testing for Multi-Agent Reinforcement Learning Systems
Xiaotong Wang and Xuan Xie
Multi-
agent Reinforcement Learning (MARL) trains a team of
agents that share one environment and learn their policies together. Training maximizes the team return, and a high return does not imply that the rewards are shared fairly among the
agents in every episode. Testing is an established way to discover the failures of deep reinforcement learning, yet few methods address the
fairness of MARL. In this work, we propose FairTest, a search-based testing approach that seeks the unfair executions of a MARL policy. The design combines search guidance with test prioritization. The guidance scores each candidate with three fitness functions. One measures the
fairness of the runs already performed, another predicts the
fairness from abstract states and
fairness features, and the third reads the decision uncertainty from the policy. Crossover and mutation derive further candidates from the observed executions. The prioritization ranks the candidates by the predicted
fairness and the decision uncertainty, so that the runs reach the candidates where failures are expected. FairTest is evaluated on three environments and two MARL
algorithms, and four baselines are given the same budget. It detects the most
fairness failures compared to three baselines with statistical significance and large effect sizes. The failure count exceeds that of the strongest baseline by 221% on average and coverage improves by an average of 23%.
Read the full abstract ▾
Software Engineering
Teach-to-Crash: A Closed-Loop Student-Teacher LLM Framework for Collision-Inducing Test Scenario Generation
Zaid Ghazal, Khouloud Gaaloul and Bruce Maxim
Validating Autonomous Driving Systems (ADS) in simulation requires testing architectures that can discover rare, safety-critical failures while generating scenarios that are executable, diverse, and useful for downstream failure analysis. We introduce Teach-to-Crash, a closed-loop testing framework that combines a constrained ego-centric scenario representation, stagnation-aware search control, and a dual-LLM architecture for adaptive failure discovery. A high-
reasoning Teacher LLM acts as an adaptive search controller, while a low-
reasoning Student LLM emits simulator-executable scenarios in a strict JSON schema. The Teacher intervenes only when rolling collision rate and time-to-collision metrics stagnate, providing strategic guidance to redirect the search. In a CARLA case study with two experimental setups that vary the ego vehicle's speed policy, Teach-to-Crash achieves the highest Collision Hit Rate (90.79%), the shortest mean Time-to-Collision (18.31 s), and a competitive Collision Discovery Rate (136.21). PAFOT attains a higher mean CDR (179.44), but with substantially larger variance. Teach-to-Crash also yields the highest diversity (0.547) and, averaged across both setups on the CARLA Traffic Manager controller, the highest avoidability-based usefulness proxy (60.04%) among the compared methods. These results, within the evaluated CARLA scope, provide evidence that closed-loop dual-LLM
reasoning can steer
adversarial simulation-based testing over a constrained executable program space, generating failures that are frequent, structurally diverse, and assessed as more frequently avoidable.
Read the full abstract ▾