Software Engineering
Why Do Pull Requests Go Silent? Uncovering the Barriers to Contribution Completion in Open-Source Code Review
Md Shamimur Rahman, Farhana Akter, Md Mustakim Billah et al. (+2 more)
Pull requests (PRs) underpin pull-based software development by enabling distributed code review and collaborative contribution in open-source projects. Yet many become inactive before integration and are eventually abandoned or closed, wasting contributor and maintainer effort. Although prior work has examined PR abandonment and review delays, less is known about the contribution types, discussion-level barriers, and post-stalling collaboration patterns associated with inactivity. We investigate which PR types most often stall, why inactivity occurs from authors' and reviewers' perspectives, and how stalling relates to later contributor and reviewer engagement. We analyzed 14,234 stalled PRs and 164,562 review comments from 19 popular GitHub repositories using stale-bot workflows. An LLM-based voting classifier categorized PRs by contribution type, while quantitative analysis was combined with qualitative coding of general and inline review discussions. Feature-enhancement and issue-fixing PRs formed the largest share, together exceeding 77% of classified stalled PRs. General comments linked stalling mainly to communication and coordination breakdowns, including missing interaction, delayed feedback, and unclear follow-up. Inline comments showed that inactivity does not always reflect disengagement: many PRs were blocked by technical or dependency issues, including failing checks, configuration problems, compatibility concerns, and environment mismatches. Only 39.56% of contributors later submitted another PR, and reviewer re-engagement with the same contributors was approximately 21%. PR inactivity is a socio-technical coordination problem involving communication, technical readiness, review ownership, and automation practices. We recommend type-aware triage, clearer review feedback, explicit ownership of next actions, CI blocker management, and cause-aware stale-bot interventions.
Read the full abstract ▾
All authors Md Shamimur Rahman, Farhana Akter, Md Mustakim Billah, Zadia Codabux, Chanchal K. Roy
Software Engineering
Does Order Matter? An Empirical Investigation into the Impact of File Ordering on Code Review Effectiveness
Md Shamimur Rahman, Zadia Codabux and Chanchal K. Roy
Modern code review is central to software quality, but its effectiveness depends on reviewer expertise, change characteristics, and how tools present changes. Most platforms display modified files alphabetically by default, although our prior work shows that developers find this ordering cognitively misaligned with how they understand multi-file pull requests. Whether these ordering-related
attention patterns affect outcomes at scale remains unclear. We present a large-scale study of file ordering and review effectiveness, mining 330,343 multi-file pull requests and 756,814 file instances from 182 GitHub projects in five programming
languages. We examine whether file position is associated with later bug-fixing changes, whether pull request size moderates this relationship, and whether reviewer
attention, proxied by comments, aligns with latent bug outcomes. Results show statistically significant but modest associations among file position, pull request size, review activity, and latent bug likelihood. Latent bug rates rise from 56.7% at position 1 to 61.5% at position 30. Pull request size has a non-linear relationship with latent bug likelihood: pull requests of about ten files have the lowest risk, while very small and very large ones have elevated rates. Hurdle models show that
attention is diluted as pull request size grows: each additional modified file reduces the odds of receiving any review comment by about 8.7%. These findings reveal an
attention-effectiveness gap: visible review activity does not necessarily prevent defects. Alphabetical ordering is therefore not a neutral interface default, but a structural feature shaping
attention allocation, review coverage, and confidence in review outcomes. We propose context-aware file ordering, dependency-aware grouping, risk-aware prioritization, and per-file coverage indicators to make review
attention more visible and actionable.
Read the full abstract ▾
Software Engineering
Identity or Prompt Noise? A Calibrated Invariance Audit of LLM Code Generation
Maksim E. Eren, Ryan Barron, Eric Michalak et al. (+2 more)
Identity cues are irrelevant to a fixed programming specification, but raw counterfactual differences can arise from unequal samples and
prompt wording. We audit 30.73 million executed Python generations from seven checkpoints on HumanEval+ and MBPP+, supplemented by an exploratory 550B slice, under model-assigned gender, country, and occupation personas. Within-task randomization and false-discovery-rate control identify occupation as the most consistent structural axis: CodeBLEU dispersion exceeds its exchangeability null in 10/14 model–
benchmark cells, remains significant in 8/12 full-coverage
cells, and exceeds the country ratio in every paired
cell, although the median excess is only 0.141 points. In six high-pass-rate
cells, occupation dispersion replicates across
token similarity, length, comments, reference similarity, and complexity, while pass-rate dispersion is significant in none. Country leads raw dispersion in 12/14
cells but has a median calibrated ratio of 1.00. Gender-associated variation cannot be separated from persona wording in this design. Thus, the evidence supports small, reproducible occupation-conditioned changes in code form, not stable disadvantage to named identities or demonstrated downstream harm. We use this finding to motivate a broader commentary on the potential impacts of bias in code generation, including the possibility that models may condition their outputs on identity information available from prior conversational context.
Read the full abstract ▾
All authors Maksim E. Eren, Ryan Barron, Eric Michalak, Charles Nicholas, Manish Bhattarai
Software Engineering
A Governance-Aware Large Language Model Orchestrated Agentic Digital Twin for Transmission System Operator Control Room Decision Support
Costas Mylonas, Magda Foti and Emmanouel Varvarigos
Transmission system operators face rising complexity from renewable integration, reduced inertia, and tighter security margins. Large
language models offer natural-
language decision support, but their
hallucinations, uncontrolled tool use, and weak traceability conflict with control room requirements. This paper presents a governance-aware agentic digital twin for transmission grid control rooms. The large
language model only selects and parameterizes whitelisted analysis tools, and every proposed action passes through a governance layer that the model cannot bypass. The layer enforces four rules on every run. Only whitelisted tools execute. No run exceeds its step budget. No action with side effects executes without explicit operator approval. Every number in an answer is rendered by the layer from backend results with its unit, variable, and time. The rules are checked on a persistent audit trail for every run of a released 118-task
benchmark, which covers analytics, simulation, multi-step workflows, and twelve families of
adversarial inputs on a digital twin of the Greek transmission network. Across 590 runs of the primary model, tool selection reaches 96.5% and task success 93.7%, and all four rules hold without exception. In a separate three-repetition study across four large
language models, 1416 runs in total, the rules again hold on every run, with an approximate 95% lower bound of 99.8%. Removing the layer makes the same model execute all 45 approval-requiring runs without authorization and leaves only 39.2% of its answers with backend-supported numbers. Enforcement costs 12 to 16 milliseconds per request.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.AI
Goal-driven Variant Categorization
Daniel Calegari and Daniel Amyot
Process discovery rarely yields a single coherent process structure. For analysis, a common step is to cluster process variants based on structural similarity and then assign business meaning to the resulting groups. Since these partitions are not derived from the organization's goals, analysts must manually interpret and consolidate variants into business-meaningful categories. This judgment-intensive step becomes increasingly difficult as the number and complexity of variants grow. In this paper, we propose a goal-driven approach to variant categorization that reverses this workflow. We first author an organization's goal model that predefines the categorization axis. Each variant is transformed into a textual narrative describing its behavior, and a Large
Language Model (LLM) interprets it in the context of the goal model and assigns the variant to the most appropriate category. LLM-based semantic
reasoning connects low-level process behavior with analyst-defined business goals. We instantiate this approach end-to-end and evaluate it on three public logs differing substantially in scale and behavioral diversity. Goal-model guidance yields partitions that differ from those produced by unguided induction and respond to controlled edits to the declared alternatives, at the cost of authoring a goal model.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.RO
Behavior Trees for Robotic Systems: An Empirical Study on Practices and Experiences
Razan Ghzouli, Jennifer Horkoff, Daniel Strüber et al. (+1 more)
Over the last decade, behavior trees (BT) have become one of the dominant behavior models for coordinating missions of robotic systems. Yet empirical evidence on BT adoption in real-world contexts remains limited, especially regarding practitioners experiences and practices. Without practitioner-grounded evidence from both academic and industrial settings, the research community risks developing guidelines and tools that are plausible in principle but only partly aligned with real challenges. This scarcity of evidence leads to ad-hoc practices, impeding software reuse, maintenance, and
evolution. This paper reports a mixed-methods study combining a technical action research investigation at an automotive company with a survey of 34 robotics practitioners. Our results indicate that BTs improve team communication and the understandability of robotic decision-making logic, reflecting BTs practical value beyond mission coordination. At the same time, practitioners face multiple non-trivial design and integration decisions, complicated by the lack of adequate guidelines and tool support. These decisions span architectural and
language choices when implementing BTs and integrating them within ROS, for which we report observed patterns. Additionally, practitioners faced multi-factor granularity decisions and mixed experiences with current libraries. For the most used BT libraries, they reported implementation challenges and documentation gaps. Regarding planning
algorithms, practitioners find deciding optimal BT node ordering challenging. Views on scalability remained inconclusive, and limitations in current libraries and practices hinder broader adoption. We conclude with cross-cutting observations and implications for both practitioners adopting BTs in robotic systems and researchers aiming to advance empirical understanding of BT adoption.
Read the full abstract ▾
All authors Razan Ghzouli, Jennifer Horkoff, Daniel Strüber, Rebekka Wohlrab
Software Engineering
How Researchers Use and Verify AI Coding Assistants: Tasks and Validation Practices in Scientific Programming
Gabrielle O'Brien, Reed Milewicz and Nasir Eisty
Generative AI has entered research programming, yet there is little evidence about which tasks researchers hand to it or how they decide whether its code is correct. We draw on 527 free-text responses to a 2025 survey of researchers who write code, most of them at U.S. universities. In each response, a researcher recounts a single task from their own work, the way they used an AI tool for it, and what they did to assess the result. We coded the task and the evaluation strategies reported, and related both to programming experience, research area, and confidence ratings. Use was concentrated in five tasks: data handling, visualization, debugging, mathematical/scientific computing, and statistical analysis. Evaluation was informal and individual. Over half of accounts described running the generated code, while automated tests and review by another person were rare. Use cases and evaluation strategies varied little with programming experience, but confidence did: Less experienced programmers trusted the AI more than themselves, and experienced programmers the reverse. Evaluation confidence was not associated with the strategies reported. Its strongest correlates were confidence in the tool and in oneself. Validating AI contributions to scientific code rested largely on individual judgment, outside shared infrastructure for testing or review. Interfaces could support task-appropriate evaluation rather than leave it to the user.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CL
RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents
Shuai Bai, Jiayong Deng, Sicheng Fan et al. (+30 more)
Computer-use
agents (CUAs) have advanced along two separate lines: graphical interaction and software development through code and the command line. Real digital work requires both, interleaved rather than stacked end to end. We study hybrid CUAs that autonomously decide when to explore an interface, implement software, and run and visually verify their artifacts. We introduce RecreationWorld, a five-platform framework built around recreation: given a running reference, an
agent must discover its behavior and build a faithful implementation with no prescribed workflow. RecreationWorld provides reproducible environments on Ubuntu, macOS, Windows, Android, and Web, plus a unified harness with native GUI control and coding tools. The running reference serves as an oracle for hidden behavioral tests, providing execution-grounded rewards. We scale trajectory generation with high-quality open-source applications. Models trained on these trajectories improve across five out-of-distribution coding and hybrid computer-use
benchmarks and more frequently verify their rendered outputs, providing evidence of transfer beyond recreation. For held-out evaluation, we introduce RecreationBench, comprising 250 diverse tasks across domains and platforms. Reference-grounded programmatic and visual assertions cover action-conditioned outcomes at multiple interaction depths; each is validated on the reference and by human reviewers before the suite is frozen for automatic scoring. GPT-6 Astra leads at 58.1% overall, but passes all programmatic tests on just 2.8% of tasks.
Agents reproduce static interface structure more reliably than interactions and computed outputs, while generated applications remain smaller and more monolithic than their references. We release the
benchmark, environments, and test suites.
Read the full abstract ▾
All authors Shuai Bai, Jiayong Deng, Sicheng Fan, Yikun Fu, Chang Gao, Xuhao Hu, Mianqiu Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Keliang Li, Ning Li, Wanli Li, Dayiheng Liu, Dunjie Lu, Changwei Luo, Que Shen, Zheyuan Wang, Zijian Wang, Jie Wu, Gao Wu, Zhihui Xie, Rui Xie, Haiyang Xu, An Yang, Jiakang Yuan, Yanming Zhang, Jiajun Zhang, Xi Zhang, Zhenru Zhang, Zhuo Zhen, Mingkang Zhu, Bowen Zhou
Software Engineering
Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report
Eric Jansson, Per Strandberg, Thomas Sörensen et al. (+2 more)
This study examines tool-augmented Large
Language Model (LLM) systems for supporting Root Cause Analysis (RCA) of nightly test failures at Westermo Network Technologies AB. Nightly test executions produce heterogeneous test data and logs that practitioners currently inspect manually across multiple sources. We implemented an RCA workflow in single-
agent and orchestrated multi-
agent configurations, both with access to test metadata and logs. An exploratory industrial case study used two real failure scenarios. Six practitioners evaluated the scenario reports through a survey and focus group, and operational measurements were collected from 120 repeated executions. The evaluation covered practitioner-perceived correctness,
reasoning quality, fix realism, clarity, usefulness, and trust, as well as cost, duration, and consistency. Neither configuration showed a consistent practitioner-perceived quality advantage across the two scenarios. The single
agent system generated reports faster and at lower cost, making it the more practical baseline in this context. The potential benefits of
agent architectures require further evaluation in more complex scenarios.
Read the full abstract ▾
All authors Eric Jansson, Per Strandberg, Thomas Sörensen, Eduard Paul Enoiu, Wasif Afzal
Software Engineering · Cross-listed from cs.DL
From Code Archival to Knowledge Graph: Bridging Software Heritage, COAR Notify and Wikidata
Camillo Carlo Pellizzari di San Girolamo and Francesco Tosoni
Software is a first-class scientific object, yet validated links between source code and the scholarly record remain largely absent from the Linked Open Data (LOD) cloud, isolating archived artefacts from semantic discovery. This paper presents an end-to-end reconciliation pipeline that harvests, validates, and models publication-to-repository pairs from sources where the link between a paper and its source code is explicit and editorially verified: the software-centric journals JOSS, SoftwareX, and IPOL, together with the reproducibility reports of the SIGMOD Availability and Reproducibility Initiative (ARI). This yields a curated corpus of 4,397 $\langle$DOI, repository-URL$\rangle$ pairs. We design two distinct application profiles grounded in Wikidata classes (one for scholarly articles, one for software instances) aligned with the schema.org and CodeMeta vocabularies. This architectural separation enables rule-based reconciliation at two granularities: lightweight, inline publication references or standalone, first-class Wikidata software nodes equipped with SWHIDs, Software Heritage's content-addressed identifiers. A read-only lookup against Wikidata shows that only 82 of the harvested repositories were already modelled there; human-reviewed batches have since created 4{,}182 new software items cross-linked to their articles. We further show that payloads of the emerging COAR Notify protocol, an external effort we do not develop, map natively onto our input format, so the same backend could later serve a live enrichment stream. Our core contribution is a pair of application profiles that turn Wikidata into a connector between the scholarly record and archived source code; we openly release all code, application profiles, and harvested
datasets.
Read the full abstract ▾
Software Engineering
From Smarter to Hungrier: the Role of Energy Efficiency in Software-defined Vehicles
Ella Peltonen
Automated features and advanced driving support systems have enabled a range of life-saving and comfort applications in modern personal vehicles. Development from mostly mechanical entities towards truly software-defined devices also means that added sensor complexity, processing power, and networking capabilities provide a platform for running even more complex
algorithms and applications. As vehicles become smarter, they require even more energy to enable advanced machine learning- and artificial intelligence-based functionalities. This paper discusses the implications for sustainability and the threats to environmental ethics posed by the development of software-defined vehicles. We discuss the balance between introducing more resource-hungry software and hardware components and their effect on society and sustainability. We underscore the need for novel tools to measure, manage, and optimise energy consumption and other sustainability indicators, with design guidance and actionable recommendations.
Read the full abstract ▾
Software Engineering
Labelling Bug-Fixing Commits with Local Open-Weight Language Models
Philip König, Georg Goldenits, Caroline König et al. (+6 more)
Defect prediction depends on knowing which commits fix bugs, yet the labels that encode this are produced by routes that each introduce noise. Reused
benchmarks carry documented data-quality problems, issue-tracker links are biased and the underlying reports are frequently mistyped, and matching keywords in commit messages is a coarse heuristic. This paper examines whether commits can be labelled as bug fixes from their content alone, using open-weight
language models that run locally and therefore keep the process reproducible, inexpensive at corpus scale, usable on proprietary code, and independent of any issue tracker. Against
datasets of manually validated and curated bug fixes spanning Java, Python, and JavaScript, we compare a keyword baseline with a set of open-weight models of varying size, prompting each with the commit message and the code diff. On the manually validated corpus the keyword baseline recovers fewer than half of the fixes, whereas the open-weight models recover the large majority and outperform it repository by repository with statistical significance, and larger models do not consistently outperform smaller ones. We further show that evaluation corpora without negative examples cannot support a precision-aware comparison of such classifiers. We release the labelling pipeline together with a labelled, multi-
language corpus produced by the recommended configuration, as a reproducible silver-standard resource for building current, project-specific
datasets.
Read the full abstract ▾
All authors Philip König, Georg Goldenits, Caroline König, Sebastian Raubitzek, Fabian Obermann, Dennis Toth, David Schmidt, Edgar Weippl, Kevin Mallinger
Software Engineering
GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions
Xinyu Che, Yunfei Ge, Shihao Li et al. (+9 more)
Coding
agents can modify and test code across large software projects. Game development is a domain where
agents must implement gameplay rules. A game can end in a valid state even after violating its rules during the run. Current game-development
benchmarks replay fixed examples, score
videos, or ask another model to judge the result. However, no existing
benchmark checks game rules throughout execution across varied evaluator-selected scenarios while ensuring exactly reproducible verdicts. We introduce GameLogicBench, a
benchmark of 72 gameplay-logic tasks in Godot projects. An automated evaluator checks each game's rules at every simulation tick. Across 403 hand-designed scenarios, seeded parameter variations produce 1,451 test cases. To ensure that the evaluator measures behavior rather than implementation choice, it must accept different correct implementations for each task while rejecting mutants, implementations with one required capability removed. The tasks span isolated mechanics, multi-system interactions, and repository-scale features. Across 20 combinations of
language models and scaffolds, the best observed run solves 52.78% of tasks. Under Claude Code, all twelve models solve fewer tasks as task scope expands from isolated mechanics, through interacting systems, to repository-scale features.
Agents inspect code more often and make more tool calls on repository-scale tasks than on isolated-mechanic tasks. Most unsuccessful submissions are runnable, but implement some required game behavior incorrectly. We compared versions of our
benchmark evaluator built with and without validation using mutants. Without this validation, incorrect
agent submissions passed. A separate analysis finds
agents copying code from public repositories when network access is open. Reliable evaluation thus depends both on what the tests reject and on what external code
agents can access.
Read the full abstract ▾
All authors Xinyu Che, Yunfei Ge, Shihao Li, Yanchen Liu, Hang Yan, Xinping Lei, Yanghai Wang, Zixuan Dong, Yifan Yao, Qianqian Xie, Letian Zhu, Jiaheng Liu
Software Engineering
Cross-Platform vs Native Mobile Development: An Empirical Study of Software Quality Trade-offs
Alexandru Ilovan
Cross-platform mobile frameworks promise code reuse, shorter delivery cycles, and lower implementation effort, but their trade-offs relative to native development remain difficult to assess objectively. Many comparisons rely on simplified applications, inconsistent feature sets, or a narrow set of metrics. This paper compares five implementations of the same plant-management application: native iOS, native Android, Flutter, React Native, and Kotlin Multiplatform. The shared approaches target both Android and iOS, yielding eight executable variants. Guided by ISO/IEC 25010, the study examines time behavior, implementation footprint, source-code organization, and observable rendering responsiveness. All implementations share the same domain, backend services, functional requirements, and
benchmark contract. The supplied
dataset contains 2,000 completed runs per variant and covers authenticated and cached
retrieval, image transfer and decoding, list rendering and scrolling, local synchronization, and media upload. Backend preparation and framework-specific UI drivers are analyzed separately from the primary client workflow. Native records the lowest non-UI client subtotal on both operating systems, with Kotlin Multiplatform the closest cross-platform implementation, while operation and UI-wrapper rankings vary by task. The shared approaches contain less authored mobile source than the native applications combined, and the source inventory shows different patterns of file size, organization, and dependency use. Rather than identifying a universally superior technology, the study shows that each approach's advantages and costs depend on the quality attribute, workload, platform, and measurement boundary.
Read the full abstract ▾
Software Engineering · Cross-listed from stat.CO
Scentree: a framework for generating scenario trees for multistage stochastic programming
Cristian Pachón-García, Albert Solà Vilalta and F-.Javier Heredia
We present scentree, an open-source Python package for constructing a scenario fan and a scenario tree for multistage
stochastic programming from historical data. It combines machine learning and multivariate time series models to obtain a scenario fan that captures inter-stage dependencies in the
stochastic processes. This scenario fan is subsequently transformed into a scenario tree suitable for multistage
stochastic optimization, providing a flexible and extensible framework for uncertainty modeling. A key contribution is the automation of the complete workflow, including model selection, parameter estimation, scenario fan generation, and scenario tree construction. Scentree does not rely on assumptions about the underlying data distribution, reducing the statistical expertise required to produce a scenario tree. Furthermore, it is agnostic to the specific multistage
stochastic problem to be solved.
Read the full abstract ▾
Software Engineering
What Stops a Small Language Model From Driving a Database Agent
Cevheri Bozoglan, Yusuf Gundogdu, Abdullah Kaya et al. (+1 more)
Small open-weight
language models are assumed to fail at agentic database work because they lack the
reasoning capacity for it. We test that against a production system. Over eleven days we drove the
agent mode of an open-source SQL client with 39 open-weight models served locally and one hosted control, across six task surfaces: 8,199 runs, 110,711 ledger events, 14,008 refused tool calls. Of the 2,100 model-attributed
agent-mode losses, 1,590, or 75.7%, came from runs that had invoked at least one tool. That majority is what survives resampling models rather than runs: it holds in 99.7% of clustered resamples and in 15 of the 22 models with at least twenty losses. Within it, transport, a run that used the tools and never got a deliverable through, is the largest class at 36.2% and capability, a run that invoked no tool at all, the smallest at 17.3%; we report that ordering as a property of this corpus rather than a general finding, since clustered by model it holds in only 74.5% of resamples. Transport failures decompose into a few mechanical argument shapes. Production ledgers record refusal codes and never the model's arguments, so these were invisible for ten days; capturing them exposed five server defects, one of which demanded a field on one tool, forbade it on the sibling that composed it, then failed the run for its absence. Five server changes, touching no model,
prompt or sampling setting, moved six models by 6 to 21
cells out of 30. We also report a confound we believe affects published local-model
benchmarks, ours included: with no context cap, one 7.1 GB model was admitted at its full 262,144-
token window and held 51 GB on a 64 GB machine, producing runs indistinguishable in any ordinary log from a model timing out. The corpus, the scorer and a verifier that regenerates every figure are released.
Read the full abstract ▾
All authors Cevheri Bozoglan, Yusuf Gundogdu, Abdullah Kaya, Koray Sirin
Software Engineering · Cross-listed from cs.AI
GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development
Xiuhui Zhang, Yi Chen, Shusheng Xu et al. (+4 more)
Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessarily establish that their interacting components satisfy the specified behavioral requirements. We introduce GameASG-Bench, a
benchmark that makes behavioral testability part of the generation task for game development. Our design declares an evaluation interface specification before generation, fixing legal starting scenarios, player-level actions, stable snapshots, rejection behavior, and invariants while leaving private implementations open. Concretely, we include: (i) static L1 checks that assess source-level compliance; and (ii) browser-executed L2 checks that combine semantic observations with real input and runtime evidence. We implement this protocol as 47 browser-native game-generation tasks spanning 12 primary genres and both 2D and 3D interaction, each with executable checks and an independently verified reference implementation. Our experiments answer four key questions about end-to-end
agent performance, tool access and nominal turn budget,
reasoning effort, and harness choice. Across nine
agent stacks, the highest observed mean L2 check pass rate is 93.2%, yet the highest observed strict task success rate, requiring all L1 and applicable L2 prerequisite and core requirement checks, is only 55.3% (26/47 tasks). For DeepSeek-V4-Flash, full tool access and larger nominal turn budgets yield more strict task successes, while the strict task success rate is not monotonic in
reasoning effort. Both tested harnesses achieve 18 strict task successes, but only ten tasks succeed under both. These results expose task-level compliance gaps that high average check pass rates actually obscure.
Read the full abstract ▾
All authors Xiuhui Zhang, Yi Chen, Shusheng Xu, Fan Li, Huan Wang, Tongkai Yang, Binhang Yuan
Software Engineering · Cross-listed from cs.AI
Efficient Benchmarking in Production: A Study of an Evolving LLM Agent
Yining She and Lei Lin
Production LLM
agents are evaluated repeatedly as they evolve, but full
agent benchmarks are costly to rerun. We study efficient recurring evaluation for a production analytics
agent serving tens of thousands of monthly active users and report first-hand deployment experience. Using 574 historical runs of the production
benchmark, split chronologically into calibration and held-out periods, we compare random sampling, historical caching, fixed representative subsets, and IRT-based adaptive testing. The results show that multidimensional 2PL adaptive testing achieves the best overall score fidelity: executing 200 questions, 38.5% of a full run, yields 1.03 pp of MAE. We nevertheless deployed difficulty-stratified fixed subsets because of their operational simplicity, and show they transfer without recalibration to five other
agent families and remain stable across calibration windows as short as one day. Drawing on this deployment experience, we report practical recommendations for recurring production-
agent evaluation.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.HC
Two's a Crowd: Human and AI-Based Copresence for Developers with ADHD
Veronica Pimenova, Seth Bernstein, Shalini Madan et al. (+2 more)
Effective collaboration and communication are vital to developer productivity and well-being, yet remain constrained by human factors such as
attention, intrinsic motivation, and interpersonal accountability. These constraints are particularly vital for developers identifying with
Attention Deficit Hyperactivity Disorder (ADHD), who navigate persistent environmental barriers in modern hybrid workplace settings. While developers with ADHD frequently rely on collaborative copresence practices (such as body doubling or pair programming) to support executive function, the recent emergence of agentic AI coding assistants has begun reshaping these collaborative dynamics. To investigate how developers with ADHD engage in human and AI-based copresence practices, we conducted semi-structured interviews with 14 software engineers with ADHD. Our findings reveal that while traditional human-human copresence provides critical social support and onboarding structure, it forces developers to constantly manage professional reputation and sacrifice personal
privacy. Conversely, developers leverage emerging human-AI copresence to maintain accountability and cognitive flow without the social anxiety, performance judgment, or surveillance associated with human observation. Based on these empirical insights, we map developer copresence practices onto core dimensions of Goffman's copresence theory and Forsgren et al.'s SPACE framework of developer productivity, and provide design recommendations for AI-based tools that promote inclusive collaboration for developers with ADHD.
Read the full abstract ▾
All authors Veronica Pimenova, Seth Bernstein, Shalini Madan, Dhruv Jain, Venkatesh Potluri
Software Engineering
License Compliance in Open Source Cybersecurity Projects
Ahmed Shah, Selman Selman and Ibrahim Abualhaol
Developers of cybersecurity software often include and rely upon open source software packages in their commercial software products. Before open source code is absorbed into a proprietary product, developers must check the package license to see if the project is permissively licensed, thereby allowing for commercial-friendly inheritance and redistribution. However, there is a risk that the open source package license could be inaccurate due to being silently contaminated with restrictively licensed open source code that may prohibit the sale or confidentiality of commercial derivative work. Contamination of commercial products could lead to expensive remediation costs, damage to the company's reputation, and costly legal fees. In this article, we report on our preliminary analysis of more than 200 open source cybersecurity projects to identify the most frequently used license types and
languages and to look for evidence of permissively licensed open source projects that are likely contaminated by restrictive licensed material (i.e., containing commercial-unfriendly code). Our analysis identified restrictive license contamination cases occurring in permissively licensed open source projects. Furthermore, we found a high proportion of code that lacked copyright attribution. We expect that the results of this study will: i) provide managers and developers with an understanding of how contamination can occur, ii) provide open source communities with an understanding on how they can better protect their intellectual property by including licenses and copyright information in their code, and iii) provide entrepreneurs with an understanding of the open source cybersecurity domain in terms of licensing and contamination and how they affect decisions about cybersecurity software architectures.
Read the full abstract ▾
Software Engineering
Embedding Drift in Code Vulnerability Models Under Intended Behaviour-Preserving Transformations
Hasti Ghaneshirazi, Tahsin Reza and Ladan Tahvildari
Code edits that preserve program behaviour can shift frozen code
embeddings across classifier decision boundaries, causing correctly detected vulnerabilities to be predicted as benign. Such instability is problematic because semantically neutral changes, including comment removal, insertion of unreachable code, variable renaming, and loop rewriting, should not alter a model's security judgment. We investigate this issue using 15,000 C/C++ functions organized as 7,500 vulnerable-patched pairs from the Big-Vul
dataset. Four mutation tracks are applied to both vulnerable and benign samples: comment removal, unreachable code insertion, variable renaming, loop rewriting, and a combined setting that applies all transformations. Using frozen microsoft/codebert-base
embeddings with six downstream classifiers, we evaluate the robustness of vulnerability predictions and introduce a train-only defence that projects clean and mutated representations closer together while preserving vulnerability-class information. Under combined mutations, the baseline Vulnerable Flip Rate (VFR) ranges from 35.55% to 42.15%. The proposed defence reduces the mean common-set VFR from 22.60% to 11.32% and improves mean mutated accuracy from 55.33% to 57.10%. For Logistic Regression, VFR decreases by 17.29 percentage points, although benign-to-vulnerable flips increase by 11.95 percentage points. These results indicate that the defence improves robustness to semantics-preserving code transformations but introduces a false-positive trade-off and does not fully eliminate prediction instability.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.LG
SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?
George Ma, Benjamin Mikek, Haoyu Li et al. (+9 more)
Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering.
Benchmarks for agentic code generation check correctness with held-out test suites, which are inherently incomplete and increasingly susceptible to memorization. Formal verification avoids both problems, but existing work covers only standalone tasks whose specifications are given as input, not real issues, which touch large repositories and state intent in vague natural
language. We present Benchproofer, a pipeline that turns a coding task with a known correct patch into a formally verified one: it writes a specification for the new code, summarizes the existing functions that code calls with axioms, and admits an instance only after mechanical and
adversarial gates agree. Applying it to SWE-bench Verified yields SWE-Proof, 500 real issues whose correctness is formally verified rather than tested, and it extends to SWE-bench Pro. Evaluating Claude Opus 4.8, we find that verification catches what tests miss: a quarter of test-passing patches admit counterexamples, which a structured natural-
language specification does not fix, while a correct formal one lifts resolution from 85% to 95%. Writing that specification is the hard part: an
agent that must write its own gains nothing over an unaided baseline, and only 56% of those specifications pass our audit. The usual failure is faithfulness, a specification that constrains part of the required behavior and leaves the rest free. Specification quality still tracks the outcome, failing on 92% of unresolved instances against 51% of resolved ones, making faithful specification synthesis a concrete open problem.
Read the full abstract ▾
All authors George Ma, Benjamin Mikek, Haoyu Li, Ferhat Erata, Yuhao Zhang, Zeren Shui, Behrooz Omidvar Tehrani, Jun Huan, Murali Krishna Ramanathan, Somayeh Sojoudi, Hao Zhou, Anoop Deoras