Software Engineering
Localizing Post-Wire Semantic Changes in MCP Agent Frameworks
Aditi Patodiya
Valid Model Context Protocol (MCP) messages do not guarantee that an
agent framework preserves the distinctions downstream software needs. We present a differential testing method that follows a fixed tool result through each framework's public interfaces and checks explicit consumer requirements. Applied to 18 designed fixtures in four pinned Python integrations, the method identifies 13 unique fixture-task divergences involving structured values, declared errors, and rich content. Google ADK satisfies every primary contract in its observed path; the other integrations show interface-specific changes or an execution failure. Whether a change matters depends on the consumer: treating absent optional fields as equivalent to null explains most of OpenAI's strict rich-content failures. In an exploratory replay, parsing JSON text recovers more structured values but also returns incorrect values and values for fields absent at the source. Documented settings help in conflicting-output stress cases without restoring a separate structured slot. Fault challenges expose an oracle weakness and test its repair on fresh cases. The study provides reproducible, interface-specific evidence from controlled cases, not production failure rates or model-behavior measurements. Its practical implication is that post-wire regression tests need to specify both the information required and how the consumer reads it.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CR
(Don't) Trust, but (Don't) Verify: Developers' Attention to Security in AI-Generated Code
Hamza Khalid, Ronald E. Thompson III, Alejandra Sabater et al. (+3 more)
AI coding assistants are rapidly transforming software development, but are known to produce insecure code. Prior work has measured whether AI-assisted developers produce secure code, but less is known about how they evaluate AI-generated code: whether they can identify vulnerabilities, what cues they use, and how trust shapes their decisions. This evaluation step is foundational to secure development with AI, whether using auto-complete, chat tools, or AI
agents. As a first step, we conducted a remote observational study with 100 participants isolating this evaluation stage. Participants were tasked with producing secure and functional code for four C linked-list tasks. For each, participants were able to cycle through five AI-generated suggestions varying in security and functionality, select one, and edit their choice into a final submission. Participants also completed a post-study survey about their decision-making and perception of AI-generated code's security and 23 completed a more in-depth interview.
Read the full abstract ▾
All authors Hamza Khalid, Ronald E. Thompson III, Alejandra Sabater, Perucy Mussiba, Kelsey R. Fulton, Daniel Votipka
Software Engineering
Helpful but Fallible: Developer Experiences of AI Tools Under a Coordinated Industrial Roll-out
Andreas Bexell, Rushali Gupta, Lo Gullstrand Heander et al. (+5 more)
AI-enabled software development tools (AI-devtools) are being industrially adopted under strong expectations of productivity gains, yet developers' experiences of such roll-outs are underexplored. Organizations commit budgets, evaluate staff, and revise practice on a partial picture, since the evidence base is mainly tool evaluations, productivity metrics, and surveys, with few qualitative in-situ accounts of ongoing, coordinated roll-outs. We report a case study of a coordinated roll-out of AI-devtools at a large Swedish telecommunications company, investigating how developers experience the roll-out and how they anticipate their profession will change. We conducted semi-structured interviews with 12 software professionals across three sites, analyzed with process coding and thematic analysis, and interpreted through the extended Technology Acceptance Model (TAM2) as a post-hoc analytical lens. Our findings on use cases, productivity, frustrations, and tool limitations corroborate prior survey work. Beyond corroboration, the interviews surface a management-developer expectation gap that maps onto the TAM2 constructs of subjective norm and voluntariness, and show that participants weigh perceived risk heavily, a factor that TAM2 and similar acceptance models do not represent. AI-devtools emerge as helpful but fallible assistants whose value is shaped by organizational expectations, system scale, and developers' skills.
Read the full abstract ▾
All authors Andreas Bexell, Rushali Gupta, Lo Gullstrand Heander, Emma Söderberg, Per Runeson, Sigrid Eldh, Wei Shi, Konstantin Malysh
Software Engineering
Quantifying Overclaiming Propensity in Frontier LLM Agents
Nolan Smyth, Yorguin-Jose Mantilla-Ramos, Pascal Jr Tikeng Notsawo et al. (+6 more)
Frontier coding
agents are increasingly trusted to work autonomously for long periods of time, yet what they actually did is often hard to tell from their final response. We quantify the propensity of such
agents to overclaim task completion, which may mislead the user. We operationalize overclaiming as a final response that reports work that the
agent's own transcript shows it did not do, for example, claiming to have read a file it never opened. This criterion requires no inference about intent and does not depend on whether the delivered work is correct; it asks only whether the reported work was done. We introduce OverclaimBench, an evaluation suite of five file-review scenarios with transcript-based coverage measurements and registered planted defects. We evaluate eight proprietary frontier models in their own production command-line interfaces and four open-weight models under a single fixed harness, and find that 1)
agents fail to read every file they were asked to review in 67.9% of runs; 2) among these incomplete runs,
agents are misleading 80.4% of the time (59-96% per model), either falsely claiming a complete review or leaving the gap undisclosed; 3) requiring delegation to subagents increases coverage, but a large majority of reviews that remain incomplete are still misleading; and 4)
agents that falsely claim a complete review miss planted defects at about 1.8 times the rate of
agents that read every file, showing that claims of completion can conceal substantive failures. Together, these results show that
agents' final responses are not reliable accounts of their actions.
Read the full abstract ▾
All authors Nolan Smyth, Yorguin-Jose Mantilla-Ramos, Pascal Jr Tikeng Notsawo, Saskia Helbling, Alberto Tosato, Mohamed Amine Merzouk, Nouha Dziri, Gauthier Gidel, Tommaso Tosato
Software Engineering · Cross-listed from cs.AI
An Empirical Study of Harness Design for Coding Agents
Run-Ze Fan, Zihao Zhang, Simin Ma et al. (+6 more)
Coding harnesses shape how autonomous coding
agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering
agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.
Read the full abstract ▾
All authors Run-Ze Fan, Zihao Zhang, Simin Ma, Yebowen Hu, Shouju Wang, Kaiqiang Song, Fei Liu, Hamed Zamani, Xiaoyang Wang
Software Engineering · Cross-listed from eess.SY
Large Language Models as Falsifiers for Cyber-Physical Systems
Ali ArjomandBigdeli, Jiawei Zhou and Stanley Bak
Falsification searches for counterexamples to formal specifications in cyber-physical systems (CPS). With specifications written in Signal Temporal Logic (STL), falsification can be formulated as a robustness optimization problem, traditionally tackled with black-box search
algorithms. In parallel, large
language models (LLMs) have recently emerged as surprisingly effective optimizers when coupled with iterative prompting. In this work, we connect these ideas and introduce LLM-Falsifier, an LLM-based approach that falsifies specifications by minimizing the STL robustness degree. Beyond generic
prompt-based optimization, our key idea is to expose the LLM to semantic information that is natural for
language models but absent from standard numerical optimizers, including natural-
language input and output names, output trajectories, and critical-time witnesses for the minimum robustness value. These additions enable smarter and more sample-efficient robustness search. On the ARCH-COMP falsification
benchmarks, LLM-Falsifier is shown to outperform existing falsification tools based on a range of optimization paradigms, from surrogate-based and
Bayesian optimization to search-based testing, on 14 of 21 specifications when measured by the average number of simulations required to find a counterexample.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.AI
Q&A on Any Spreadsheet Requires Interpreting Its Grid Structure
Zofia Smoleń
Semantic
cell annotation improves chunking interpretability for spreadsheets in LLM-driven RAG systems, aiding answer generation through enriched context rather than improved
retrieval accuracy. We propose a novel framework of splitting any spreadsheet into interpretable chunks using
cell role annotation. Our framework beats the state of the art, yet it faces a hard ceiling. Spreadsheets are fundamentally two-dimensional unstructured data with continuous relationships and infinite potential
cell roles. Because classification models are restricted to finite, pre-defined classes, they cannot perfectly capture this structural nuance, even with human-level annotation. We show that addressing the spreadsheet-to-LLM bottleneck requires moving beyond discrete
cell classification. Instead, the field must develop dimensionality-reduction techniques to directly flatten 2D unstructured spreadsheets into 1D unstructured text. Text chunks would be easier for downstream RAG to interpret and generate from.
Read the full abstract ▾
Software Engineering
Spotlights: Discovering Improvement Opportunities in Software Repositories
Udi Barzelay, Ophir Azulai, Idan Friedman et al. (+9 more)
Coding
agents and evolutionary code-search systems can improve implementations once a target and evaluation criterion have been specified. Applying these methods to an existing software repository raises an earlier question: which implementation choices are worth investigating for a high-level engineering objective? We introduce optimization-opportunity discovery, the repository-level task of identifying candidate source regions, explaining how they relate to the objective, and proposing possible changes. The task takes as input a repository, an engineering objective, and optional runtime evidence such as offline telemetry observations or profiles. It does not require the user to specify a defect, bottleneck, or code location. We present Spotlights, a system that performs this task through logical repository mapping, successive
agent reviews, and optional research linking candidates to relevant techniques. We evaluate Spotlights across model serving, document
retrieval, blockchain ordering, and document processing. Across three cases, it recovers seven of nine expert-selected targets. In the reliability study, 70% of the top ten candidates meet the stated correctness and severity thresholds. Across five repeated
retrieval runs, 73.6% of candidate occurrences have a matching source region in all five runs. Spotlights also rediscovers the target of a withheld
retrieval optimization and connects it to a relevant tiling technique. In an implementation study, a discovered change reduces end-to-end page-processing runtime by 10.6% while preserving measured output quality. These results establish optimization-opportunity discovery as a distinct and empirically evaluable step between a broad engineering objective and subsequent implementation and validation.
Read the full abstract ▾
All authors Udi Barzelay, Ophir Azulai, Idan Friedman, Inbar Shapira, Foad Abo Dahood, Yevgeny Burshtein, Orit Prince, Michael Soloveitchik, Oshri Naparstek, Roi Pony, Tal Drory, Michael Factor
Software Engineering
Assessing the Construct Validity of Object-Oriented, Class-Level Code Quality Metrics
Hera Arif, Miikka Kuutila and Paul Ralph
Background: Code quality metrics are intended to measure latent properties of software source code. Although numerous code metrics have been proposed and used, their construct validity is rarely evaluated. Thus, the extent to which code metrics actually measure what they claim to measure is often unclear. Aim: Drawing from modern measurement theory, we investigate the construct validity of common class-level, object-oriented code quality metrics by identifying their factor structure using Exploratory Factor Analysis (EFA). The metrics were extracted from the Apache Maven project by three software tools: Designite, JHawk, and Understand. The factor structure was later verified using Confirmatory Factor Analysis (CFA) on 22 randomly selected open source projects meeting a predetermined eligibility criteria. Results: 24 code quality metrics that correspond to six constructs: Cohesion, In-Coupling, Out-Coupling, Size, Sub-Inheritance (related to subclasses), and Sup-Inheritance (related to superclasses) were revealed in the underlying factor structure. Ten metrics did not correspond to any known dimension of software quality and were removed in the EFA. Ten additional metrics exhibited low loadings in the CFA, suggesting their removal from the final measurement model. Size, Cohesion, Inheritance, and Coupling were the constructs retained, with subcategories identified for Inheritance and Coupling. Conclusions: Our results strongly support the construct validity of 24 code quality metrics. Coupling and Inheritance are revealed as multidimensional constructs, since they require measuring two different concepts, revealed as sub-categories in our analysis, and Complexity may be better explored in a multilevel model. Overall, our study demonstrates the value of applying modern measurement theory and latent variable modeling in validating software code quality metrics.
Read the full abstract ▾
Software Engineering
AdaRepair-Mem: Adaptive Experience Orchestration for Repository-Level Program Repair
Z. C. Luo, J. C. Guo, W. J. He et al. (+10 more)
Recent
memory-augmented repository-level program repair methods reuse historical repair experiences to improve LLM-based issue resolution. However, our analysis reveals three limitations in existing repository-level
memory retrieval. First, episodic
memory is highly imbalanced across repositories, leaving low-resource repositories with little effective support. Second, more
memory does not monotonically lead to higher repair success, suggesting that relevance, quality, and redundancy matter more than raw
memory volume. Third,
memory accumulation is phase-misaligned: repositories may contain many reproduction experiences but few patch or refinement experiences. To address these problems, we propose an adaptive experience
retrieval framework for repository-level program repair. Our framework introduces coverage-aware
retrieval, which falls back to cross-repository or repair-type-based
memories when same-repository
memory is insufficient; quality-aware selection, which ranks
memories by relevance, historical utility, specificity, and redundancy; and stage-aware routing, which separates and retrieves
memories for reproduction, localization, patch generation, patch refinement, and validation. Evaluated on SWE-Bench-Lite and SWE-Bench-Verified, the proposed framework improves repair performance on under-covered repositories, reduces noisy
memory retrieval, and better supports failed-to-fixed patch refinement. Our results show that the key to
memory-augmented repair is not simply accumulating more experiences, but retrieving the right experiences for the right repair context.
Read the full abstract ▾
All authors Z. C. Luo, J. C. Guo, W. J. He, S. Y. Wang, J. C. Yu, F. M. Zhao, Y. Chen, T. Cao, L. Q. Liu, N. Zheng, W. Xu, J. Jiang, Z. M. Zhao
Software Engineering
HapCiD: Detecting API-related Compatibility Issues in OpenHarmony Apps
Daihang Chen, Yonghui Liu, Mingyi Zhou et al. (+1 more)
OpenHarmony, an emerging open-source mobile platform, is rapidly gaining
attention in the mobile development community. Its fast-evolving Software Development Kit (SDK) introduces numerous Application Programming Interfaces (APIs) to boost developer productivity but inevitably introduces compatibility challenges, an issue well known from platforms like Android. However, existing compatibility analysis tools are ineffective for OpenHarmony due to its newly introduced ArkTS programming
language and the lack of a mature behavioral model to capture app execution semantics. To bridge this gap, we present HapCiD, an open-source tool for automatically detecting API-related compatibility issues in OpenHarmony apps. Applied to 4,478 apps, HapCiD successfully detects 2,040 compatibility issues across 163 apps with 100 percent accuracy. Based on these findings, we conduct a comprehensive study of compatibility issues, categorizing their types, examining existing mitigation strategies, and proposing future improvements.
Read the full abstract ▾
All authors Daihang Chen, Yonghui Liu, Mingyi Zhou, Li Li
Software Engineering · Cross-listed from cs.CR
Mind the Gap: How SBOM Specification Ambiguities Lead to Divergent Software Bills of Materials. An Empirical Tool Study
Alan Prado, Olivier Zendra, Philippe Boinot et al. (+1 more)
Software Bill of Materials (SBOMs) will become mandatory starting in December 2027 under the European Cyber Resilience Act (CRA) [8]. Although previous studies have highlighted significant differences among SBOM generators, the reasons for these discrepancies remain unknown, as does whether they stem from implementation errors or deliberate design choices. In this paper, we evaluate three widely used SBOM generators across more than 3,000 JavaScript and Rust projects, using a groundtruth baseline derived from dependency lockfiles. Our results show that these tools diverge in terms of both dependency coverage and SBOM completeness. Importantly, most of these discrepancies are systematic rather than accidental: they arise from differing assumptions regarding dependency scope, naming, provenance, and representation, while others reflect inconsistent support for fields defined in SBOM specifications. These findings demonstrate that many of the observed discrepancies cannot simply be ”fixed”: they require clearer standardization. As SBOM generation becomes a legal compliance requirement, the choice of tool itself can influence the resulting SBOM, potentially becoming a source of undetected non-compliance. We argue that future SBOM standards should define canonical rules regarding dependency scope, provenance, and representation to improve interoperability and compliance.
Read the full abstract ▾
All authors Alan Prado, Olivier Zendra, Philippe Boinot, Olivier Barais
Software Engineering
EviRCA: Decoupling Evidence Extraction from Reasoning for Microservice Root-Cause Analysis
Yuhao Wang, Zhen Qin, Xingliang Wang et al. (+3 more)
Root-cause analysis (RCA) is a critical yet labor-intensive task for maintaining modern microservice systems, making it an attractive target for large
language models (LLMs). Recent agentic approaches allow an LLM to iteratively explore raw telemetry by generating and executing code, asking a single model to simultaneously retrieve evidence, localize faults, and infer root causes over large volumes of heterogeneous telemetry, which leads to high computational cost, unstable behavior, and limited diagnostic accuracy. However, raw telemetry consists of numeric metrics, structured traces, and machine-generated logs that are not directly suitable for LLM processing. We present EviRCA, a framework for LLM-based RCA that decouples deterministic evidence extraction from LLM
reasoning. A system-agnostic extraction stage converts raw metrics, traces, and logs into a compact set of faithful
multimodal evidence cards, while the LLM reasons only over these structured observations through a small set of predefined read-only tools, without accessing raw telemetry or executing code. We evaluate EviRCA on OpenRCA, a
benchmark built from real, heterogeneous telemetry across three enterprise systems. EviRCA achieves a correct rate of 40.6%-43.9% across two different LLMs, substantially outperforming prior OpenRCA baselines that achieve up to 15.2%, while reducing
token consumption by 15-26x and execution time by 3-20x. Moreover, EviRCA solves hard cases requiring simultaneous
reasoning over time, components, and root causes, a setting where previous approaches reported near-zero performance. Our process-level failure analysis further shows that the bottleneck lies in judging the evidence that the extraction stage has already surfaced, rather than searching for it, suggesting that the effectiveness of LLM-based RCA depends heavily on the quality of evidence extraction.
Read the full abstract ▾
All authors Yuhao Wang, Zhen Qin, Xingliang Wang, Guochang Li, Weize Li, Shuiguang Deng
Software Engineering
A Closed-Loop Control Architecture for Reliable Constraint Satisfaction in LLM Text Generation
Quan Zhou, Shahbaz Siddeeq, Mika Saari et al. (+1 more)
Software systems increasingly embed a large
language model in features that must satisfy a numeric output constraint, that is, a requirement expressible as a number or an interval and checkable by code, such as a target word count or a target readability grade band. Because such a model is non-deterministic, is configured through natural-
language instructions rather than a typed interface, and satisfies a stated requirement only approximately, a single
prompt neither reliably meets the target nor preserves the source content. This paper presents and evaluates a closed-loop control architecture for this problem. It has five stages: generate, evaluate, adjust, archive, and analyze. The model is called only to write and to edit text, while deterministic code compares a composite readability value against a target band, rejects any edit that drops source entities, numbers, or keywords, and makes every accept decision. Over 114 single-shot generation jobs and 240 closed-loop runs on four commercial models, single-shot prompting met the target in 21.1 to 31.6 percent of cases and the closed loop in 92.5 to 98.8 percent, within two edit rounds on average and at a recall-based fidelity of 0.92 to 0.93; the two models common to both settings show the same effect. Because the controller optimizes the value on which success is scored, the result establishes reproducible control over a declared, computable metric and not validated human difficulty. The transferable practice is to declare the acceptance condition as code, bound the model to local edits, and gate every edit on a content check.
Read the full abstract ▾
All authors Quan Zhou, Shahbaz Siddeeq, Mika Saari, Pekka Abrahamsson
Software Engineering
BA-TRACE: Boundary-Aware Trace Reconstruction for Scenario-Based Evaluation of Mixed AUTOSAR Adaptive and ROS 2 Vehicular Embedded Systems
Shunsuke Ito, Ryudai Iwakami, Hiroyuki Hanyu et al. (+2 more)
Modern vehicular embedded systems increasingly combine ROS 2-based autonomous-driving stacks with AUTOSAR Adaptive Platform (AUTOSAR AP). Such mixed stacks make scenario-based evaluation hard to interpret because execution paths cross DDS-SOME/IP middleware boundaries between ROS 2 and AUTOSAR AP. Existing simulators and tracing tools execute scenarios or collect platform-local traces but cannot reconstruct cross-domain data flows. This paper presents BA-TRACE, a boundary-aware trace reconstruction framework for scenario-based evaluation of mixed AUTOSAR AP and ROS 2 vehicular embedded systems. BA-TRACE combines ROS 2 trace events, AUTOSAR ara::log events, ARXML-derived structural dependencies, and bridge-level instrumentation to reconstruct an end-to-end execution graph across the DDS-SOME/IP boundary. A case study with an AWSIM/OpenSCENARIO-based object-detection and braking scenario shows that BA-TRACE reconstructs the expected cross-platform path and exposes boundary-specific latency such as point-cloud transfer overhead. The reconstructed
topology is used as evidence of traceability, not as proof of behavioral correctness or safety.
Read the full abstract ▾
All authors Shunsuke Ito, Ryudai Iwakami, Hiroyuki Hanyu, Tasuku Ishigooka, Takuya Azumi
Software Engineering · Cross-listed from cs.AI
FINSKILLOPS: A Self-Evolving Multi-Agent System for SEC Filing QA
Yanzhang Ma, Zhenghan Tai, Hanwei Wu et al. (+25 more)
Financial QA systems are typically improved before deployment through better
retrieval, prompting, or
agent coordination, leaving their reliability behavior fixed thereafter. In practice, new SEC-filing questions repeatedly expose heterogeneous errors in period, entity, evidence use, and calculation. Existing self-improvement methods can turn failures into new behaviors, but offer limited control over where a correction should apply or which previously correct answers it may break. We therefore frame post-deployment improvement as controlled behavioral maintenance: recurring failures should become scoped skill patches, and each patch should earn deployment with- out introducing regressions. We instantiate this view in FINSKILLOPS, a multi-
agent system for SEC filing QA. FINSKILLOPS derives reusable skills from evidence-grounded, typed failure diagnoses and governs them through targeted validation, protected-case regression checks, negative controls, and versioned replacement or retirement. Across six financial QA
benchmarks, a single frozen skill registry achieves the highest verdict-weighted correctness and reference consistency among the evaluated systems. Evolved skills raise correctness from 3.70 to 4.55 on our enhanced
benchmark. In a separate 12-round operational study, only six of 33 proposed skills are promoted, while the monitoring non-correct rate falls from 20.0% to 12.5%. These results establish controlled skill scope, admission, and lifecycle management as the foundation for reliable self-improvement.
Read the full abstract ▾
All authors Yanzhang Ma, Zhenghan Tai, Hanwei Wu, Sizhe Guan, Jianliang Lei, Hailin He, Chaolong Jiang, Jijun Chi, Tung Sum Thomas Kwok, Bohuai Xiao, Jingrui Tian, Xinlu Wu, Xingao Zhan, Peng Lu, Muzhi Li, Yihong Wu, Liheng Ma, Sicheng Lyu, Tianshuo Yan, Junhao Zhu, Yaqian Xu, Lei Ding, Yufei Cui, Ziquan Liu, Boyu Han, Hengli Liu, Ling Zhou, Xinyu Wang
Software Engineering · Cross-listed from cs.LG
The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability
Michael Hernandez and Tian Zhao
Complexity measured from generated code is failure-dependent: a difficult
prompt can yield a short failing program and be assigned low output complexity. We introduce a six-dimension
prompt-side structural-complexity index scored before generation and kept separate from correctness. We select 5,000 Python
prompts across six bands of a preliminary single-rater rubric. Four out-of-panel LLM raters rescore the locked
prompts, giving 19,997 score rows; composite inter-rater reliability is ICC = 0.872 on the 4,998
prompts with all four ratings. We evaluate 21 models per
prompt, yielding 105,000 generations. In the unadjusted mean-pooled analysis, pass rate has a nonmonotone breakpoint at composite 13.75, with 79.9% at or below and 87.6% above. This is not a universal failure cutoff. Task-type fixed effects shift the breakpoint to 10.75 and cut the regime gap from 7.6 to 2.1 points. A construction-frame control shifts it to 8.50 with a raw gap of -3.5 points, and neither frame alone reproduces the pooled +7.6-point change. Model-specific fits include 16 upward and five downward changes. A 365-
prompt audit-clean extension matches the original five-model estimates at bins 15 and 16 but adds only 14
prompts above bin 16. Among zero-pass generations with computable Lizard complexity, 28.5% pair a
prompt composite above 8 with output complexity at most 10. Human agreement is moderate and rater-dependent on a disagreement-enriched calibration set; paraphrase and cross-
language rescoring preserve score ordering. Overidentification tests reject the joint restrictions on the six dimensions, so we treat the composite as an index and make no causal interpretation of the 2SLS estimates. The contribution is a pre-generation measurement framework and a bounded observational analysis of reliability regimes.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CL
Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction
Yuanzhe Jia and Ali Anaissi
Modern applications generate massive volumes of raw telemetry data, but translating those noisy, heterogeneous event streams into actionable business insights remains a fundamental challenge. Data engineers and analysts expend substantial effort reconciling semantic discrepancies, hand-crafting parsing logics, and maintaining fragile mappings between raw data and business KPIs. In this paper, we present an end-to-end framework that fully automates the construction of a business semantic layer from application raw logs. Our approach introduces a two-stage semantic abstraction: first, high-level business features are identified via LLM inference augmented with domain-specific industry knowledge; second, fine-grained business nodes are derived through a structured pipeline comprising data refinement, hybrid
retrieval, multi-stage filtering, semantic clustering, and canonical naming. Evaluation on production-scale telemetry demonstrates that our system improves human-assessed semantic quality from 50 to 80+ on a 100-point scale, reduces maintenance effort by 80%, filters out 74% of noise, and achieves 0.87 Cohen's kappa via an integrated LLM-as-Judge evaluation, enabling continuous, scalable quality assurance. Overall, our work distinguishes itself from prior work by addressing the novel problem of business semantic layer induction from raw telemetry, operating without labeled training data or manual rule engineering.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.PL
The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs
Paul Biberstein, Joseph Devietti and Mayur Naik
Tensor programs, as used in deep learning models, are a
prime target for optimization, as small performance improvements can have a large impact across training or inference workloads. However, such optimizations are complicated and can produce subtle bugs. Traditionally, correctness is assumed when differential testing against a reference on random inputs fails to reveal bugs. However, the inputs to these programs are massive tensors, and finding bugs can require generating extremely low likelihood inputs with precise relationships among their values. We propose a novel way to find bugs more consistently by flipping the quantifiers. Rather than generating a single input and checking all output tensor locations for equivalence, what if you could check a single output tensor location's equivalence for all inputs? We implement this idea in a system, \dirigo, by using a novel symbolic execution strategy. We demonstrate that \dirigo can find bugs effectively in a public
dataset of 6,988 AI-written CUDA kernels that are all marked correct by differential testing. Of these, \dirigo finds 600 kernels that are actually buggy, and finds 97.3% of those bugs within two minutes.
Read the full abstract ▾
Software Engineering
DeltaSelect: Affordable A/B Testing for Coding Agents
Nicholas J. Conn
Coding-
agent benchmarks are built for broad and comprehensive comparisons, not frequent development decisions. Individual runs vary, full suites are expensive, and the
benchmark harness may differ from the harness used in practice. In a resampling analysis of DeepSWE's published trials, only 19.5% of tasks (22 of 113) had a fifth-percentile Pearson correlation of at least 0.50 with full-
benchmark performance. The paper presents DeltaSelect, an open-source method that identifies tasks whose one-run results consistently track full-
benchmark performance using Pearson correlation, maps fractional verifier results to a common score using linear regression, and selects a fixed task set within a dollar budget. DeltaSelect is intended for repeated baseline-versus-candidate comparisons during development, not model rankings. In a gpt-5.6-luna low-
reasoning case study, DeltaSelect was used to revise custom skills and instructions. Across 13 evaluations, the recorded cost was USD 27.86 at rates published August 16, 2026. The adopted version cost 58.1% less than the initial version (USD 1.75 versus USD 4.18; p=0.008), while the calibrated score was higher (42.36% versus 36.46%; published-analog variance p=0.326).
Read the full abstract ▾