Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Computer Science Software Engineering

cs.SE Watch Software Engineering

Tuesday 13 October · still arriving

Software Engineering · Cross-listed from cs.LG

Interaction-as-a-Judge

Wayne Chi, I-Hung Hsu, Li Chen et al. (+6 more)

Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games. While humans assess these applications interactively, current automated LLM evaluations primarily focus on code analysis and unit tests. This mismatch can cause LLM judges to miss failures that are immediately apparent to human users. We study Interaction-as-a-Judge, an evaluation paradigm where computer-use agents (CUAs) directly interact with an application. We demonstrate that Interaction-as-a-Judge can improve both evaluation and generation of interactive applications. To this end, we collect and release two complementary datasets: $6,858$ pairs of applications with human preferences and corresponding agent traces, and $222$ applications with human-verified requirement labels. Relative to standard non-interactive approaches, we find that Interaction-as-a-Judge improves evaluation in terms of predicting human preferences ($65.6\% \to 69.1\%$) and requirement verification ($77.0\% \to 81.6\%$). We also observe scaling benefits: across both preference alignment and requirement verification, higher-quality interactions improve performance. Finally, through both inference- and training-time experiments, we demonstrate that Interaction-as-a-Judge can also improve the quality of generated artifacts. At inference time, Interaction-as-a-Judge repairs $75.5\%$ of broken requirements within five attempts, compared to $55.4\%$ for code-only self-feedback. At training time, using Interaction-as-a-Judge as a reward signal improves win-rate over the base model by $6.7$pp while reducing both generated code length by $35.7\%$ and build failure rates by $7.2$pp compared to a code-only reward signal.
Read the full abstract

All authors Wayne Chi, I-Hung Hsu, Li Chen, Wei-Lin Chiang, Ion Stoica, Anastasios Nikolas Angelopoulos, Ameet Talwalkar, Chris Donahue, Cho-Jui Hsieh

Software Engineering

Mobile App Analysis in the New Era: Challenges and the Road Ahead

Jiale Wu, Tianming Liu, Yanjie Zhao et al. (+1 more)

Mobile applications (apps) have grown into complex, multi-platform systems that support diverse services and operate within varying regulatory and distribution contexts. Traditional analysis techniques have established a strong foundation for ensuring quality, security, and compliance, but face increasing challenges as mobile apps evolve. This article presents a three-layer perspective to examine these challenges. The Artifact Layer highlights how heterogeneous code bases and protection mechanisms reduce the analyzability of packaged apps. The Runtime Layer addresses growing execution complexity, from multi-modal interactions and super-app architectures to AI-driven autonomous operations. The Ecosystem Layer considers broader contexts, including emerging platforms, fragmented distribution channels, and region-specific regulatory requirements. New characteristics across these layers interact and compound, creating difficulties that existing approaches are not designed to handle. For each layer, we identify key challenges and potential research directions, including framework-aware analysis, adaptive testing, and policy-aware compliance verification. We also discuss how large language models bring new analytical capabilities to the field, while their integration into apps raises new security and privacy concerns. This article aims to clarify where established methods fall short and highlight directions for future research in this evolving field.
Read the full abstract

All authors Jiale Wu, Tianming Liu, Yanjie Zhao, Haoyu Wang

Software Engineering

State-Aware Generation of Deployable Websites from Visual Prototypes and Requirements

Ruwei Pan, Junlei Shen, Linhao Wu et al. (+7 more)

Generating deployable websites from visual prototypes and textual requirements is increasingly feasible with multimodal models, yet keeping generated projects consistent with their prototypes remains challenging. Prototypes specify layouts and visual elements, while requirements describe interaction workflows and deployment constraints. Existing UI-to-code methods and coding agents often treat layout reconstruction, interaction implementation, and project editing separately, allowing visual elements, files, workflows, dependencies, and render evidence to lose their correspondence across generation, verification, and update. We call this problem prototype-to-project state drift. We propose VisStateCoder, a state-aware framework that maintains these correspondences in a Project State Graph. The graph connects four types of state nodes through typed links and supports prototype grounding, graph-guided generation, and render-evidence-guided scoped updates that map failed checks to affected files while preserving files outside the update scope. We evaluate VisStateCoder on Vision2Web using five foundation models across Static Webpage, Interactive Frontend, and Full-Stack Website tasks. Averaged across models, VisStateCoder improves overall scores over the strongest baseline for each model by 26.1 and 22.6 points on Interactive Frontend and Full-Stack Website tasks, respectively. It also reduces verified-state drift, measured by checks that pass in one update round but fail later, compared with alternative state representations. An annotation study of 50 tasks finds that the constructed graphs achieve 95.6% edge-level F1.
Read the full abstract

All authors Ruwei Pan, Junlei Shen, Linhao Wu, Zixiong Yang, Qisheng Zhang, Yifan Li, Tianyu Ma, Yakun Zhang, Lu Zhang, Hongyu Zhang

Software Engineering

Diff Diffusion: Predicting Edit Paths with Diffusion Models for Efficient Code Editing

Xiaolong Xu, Hao Jiang, Bo Jiang et al. (+5 more)

Diffusion large language models (dLLMs) are appealing for code generation because they can update many tokens in parallel, yet many are optimized for sequence synthesis rather than edit prediction. This setting is poorly matched to modern AI-assisted IDEs, where edit prediction must infer the next code change from recent edits, the current file, and cursor-local context. Such changes are usually sparse and preserve most source tokens, so regenerating a full code segment wastes computation. We propose Diff Diffusion, a diffusion-inspired LLM framework for code editing that predicts edit paths from an original snippet to its revised form. Diff Diffusion represents denoising as structured edit operations, and combines this formulation with multi-stage training and efficient parallel decoding. By focusing computation on changed regions, Diff Diffusion supports accurate low-latency edits and multi-position parallel modification. Across public edit-completion benchmarks, edit-prediction evaluations, and online validation, Diff Diffusion matches or leads autoregressive baselines while delivering 2.78 times faster inference on public edit-completion benchmarks and a 39.0% P50 model-latency reduction in production. This work provides a new modeling perspective for AI applications that require high accuracy and low latency.
Read the full abstract

All authors Xiaolong Xu, Hao Jiang, Bo Jiang, Zhiwen Deng, Ningyuan Sun, Bingzhou Chen, Jin Chen, Chenxing Wei

Software Engineering · Cross-listed from cs.AI

Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents

Kaiming Liu, Yuxuan Liu, Jinrui Ju et al. (+4 more)

Large language model agents can follow complex human instructions to solve real-world tasks. In practice, however, initial instructions are often incomplete. In our analysis of 100 user-agent trajectories from SWE-chat, 56% of initial instructions omit requirements that later emerge as necessary, and 52% of the useful follow-up requirements could have been inferred from the project. A user states requirements from their own understanding of the task, and these stated requirements often differ from the true requirements that completing the task imposes. An agent should therefore treat the instruction as evidence rather than as the objective, inferring the true requirements from the instruction and the environment and satisfying them even when the user is not yet aware of all of them. We call this ability outcome alignment. In this work, we construct TiGanEval, a benchmark of software engineering and research tasks. In every task the user genuinely lacks part of the requirements, each requirement stays discoverable from the code and runtime behaviour of the environment, and requirements the agent infers are separated from those the user adds after observing delivered work. These properties together make outcome alignment measurable. Evaluating eight frontier models including GPT-5.6-Sol, we find outcome alignment broadly missing: the models lose 53 to 67 points of success rate when moving from the full specification to the partial instruction, and consequence feedback across four submissions does not close the gap. Furthermore, a plan-mode harness cannot compensate for the missing ability of the model itself, which places outcome alignment among the abilities the next generation of models should be developed for.
Read the full abstract

All authors Kaiming Liu, Yuxuan Liu, Jinrui Ju, Minyuan Ruan, Xuanyu Lei, Peng Li, Yang Liu

Software Engineering

StateHarness: Maintaining Evolving Project State for Long-Horizon Coding Agents

Chuan Zou, Yujun Zhou, Xiaonan Luo et al. (+1 more)

Long-horizon coding requires agents to maintain an evolving view of the project they are changing: what problem is currently being solved, what has already been learned or modified, which other artifacts may be affected, and what work remains unresolved. When this information is scattered across a growing interaction history, local edits can become disconnected from their later consequences and progress can be difficult to carry forward. We introduce StateHarness, a runtime framework that maintains evolving project state without updating model parameters. The working agent records its current problem and hypothesis, important findings and decisions, discovered repository relationships, and unfinished work. Runtime support prompts record updates, searches for callers, tests, and documentation that may be affected by a change, and maintains a compact working brief for subsequent decisions. When behavioral signals accumulate, a separate diagnostic session revisits the recorded history and surfaces overlooked evidence. Across three repository-level evaluations, StateHarness improves task completion and several measures of change consistency. On RefactorBench, full-chain completion increases from 29.4% to 35.3% and cleanup-check accuracy from 87.0% to 95.4%. On five LHTB software-engineering tasks, mean reward improves from 0.427 to 0.576, exceeding LongHorizon-Harness at substantially lower recorded execution cost. On ChainSWE, bug-fix rate increases from 35.1% to 39.5% and residual debt decreases from 60 to 43 items, although pass-to-pass regressions increase. These results show that maintaining project state can improve long-horizon coding while introducing measurable execution and regression trade-offs.
Read the full abstract

All authors Chuan Zou, Yujun Zhou, Xiaonan Luo, Xiangliang Zhang

Software Engineering

Cross-Project Bug Localization: What Works, What Doesn't, and Why

A Eashaan Rao and Sridhar Chimalakonda

Bug localization models require substantial within-project training data that newly active or small open-source projects rarely have. Cross-project learning (CPL) promises to close this gap by reusing labeled bugs from mature projects, but prior CPL studies evaluate a single model against ample-data within-project baselines that data-scarce projects cannot reach. We ask whether CPL's effectiveness is a property of the transfer setting or depends on model architecture. We evaluate three architecturally distinct models, embedding-based BLAZE, convolutional TRANP-CNN, and GNN-based adversarial COOBA, on 63 source to target pairs from 13 open-source Python projects under identical bug corpora, data splits, retrieval candidates, and baselines. The primary baseline is within-project limited training (WP-small, 20% of target labels, the same target-side share as CP-transfer). CPL outperforms WP-small most clearly for TRANP-CNN (Cohen's d = 0.95, win rate 81.0%, p < 0.001, robust to clustering pairs by target project), and less so for BLAZE (d = 0.49, win rate 60.3%, p = 0.002; not significant once pairs are clustered by target). COOBA, under our re-implementation, shows a small gain that does not survive multiple-comparison correction (d = 0.27, p = 0.027). Negative transfer affects 9.5% of TRANP-CNN pairs and 30.2% of BLAZE's and COOBA's. Cross-model direction agreement is only 44-54%: the same pair can help one architecture and harm another, although the models' gains are positively correlated (rho = 0.24-0.42). CPL outcomes are therefore architecture-dependent: practitioners should validate CPL for their chosen model, and CPL studies should report results per architecture.
Read the full abstract

Monday 12 October

Software Engineering · Cross-listed from cs.AR

When Is RTL Better? Auditing Evaluation Stability with Tool-Grounded Requirement Profiles

Bodong Hou and Zhixin Pan

RTL-generation studies increasingly report functional correctness and post-synthesis quality, yet the evaluator itself is rarely treated as an experimental variable. A candidate can appear better when a security rule, protocol obligation, target context, feasibility gate, uncertainty policy, or provenance constraint is omitted. We introduce tool-grounded requirement profiles: machine-readable contracts binding each requirement's meaning to its executable evidence and comparison rule, together with cross-requirement gates, uncertainty handling, and provenance. We audit evaluator stability by freezing candidate pairs and perturbing only their evaluation profile. In a preregistered controlled pilot with six designs, twelve pairs, three requirement families, six evaluator conditions, and 72 decisions, a generic cell-count evaluator reports unsupported improvement for 6/12 pairs (50%; Wilson 95% interval 25.4 to 74.6%; design-cluster bootstrap interval 25 to 75%). The matched evaluator detects all twelve injected regressions, whereas syntax, generic synthesis, context mismatch, and gate ablation each lose the required distinction for 12/12 pairs. In a separately frozen external validation of all 290 public RTLLM v1.1 text-only candidates, 81/215 evaluable comparisons (37.7%; task-cluster 95% interval 24.8 to 51.4%) yield unsupported scalar improvements, including 72 functional-gate reversals. Separately, 26 OpenROAD-flow-scripts runs show why timing context and activity qualification must be part of the contract. The study does not rank generators; it establishes evaluator configuration as a measurable, auditable source of conclusion instability.
Read the full abstract

Software Engineering

Why Agentic Workflows Fail: A Taxonomy Derived from Literature and Empirical Data

Xin Li, Rami Bahsoon and Tao Chen

Agentic workflows combine large language models with planning, memory, tools, execution environments, and iterative feedback. Their failures are difficult to diagnose because the same user-visible symptom may originate from model behavior, workflow orchestration, external tools, infrastructure, or their interaction. This paper presents a source-aware taxonomy of agentic workflow failures and applies it to evidence from academic literature and GitHub. The taxonomy combines source attribution with three empirical dimensions: a concrete failure-type dictionary, workflow-stage localization, and agent-capability localization. Its central structure is a two-axis localization matrix: workflow stages form the horizontal process axis and agent capabilities form the vertical functional axis. This design moves beyond taxonomies that mainly classify final failure manifestations by localizing intermediate process states inside agent operation. Our empirical comparison shows that literature-derived failures emphasize planning, reasoning, learning, perception, and knowledge-related mechanisms, whereas GitHub-derived failures concentrate on execution-stage, actuation-related, tool, API, and infrastructure failures. The dominant hotspot is iterative execution and actuation. These findings indicate that agentic workflow reliability should be treated as a systems problem rather than only a model-output problem. The taxonomy provides a basis for structured evaluation, root cause analysis, failure injection, and engineering triage.
Read the full abstract

Software Engineering

VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green

Gabija Pasiunaite, Ryan McMorrow, Brian Clark et al. (+2 more)

Software composition analysis (SCA) tools report far more vulnerable dependencies than teams fix: in Snyk Open Source production data, 94% of vulnerability findings have a known fixed version, yet only about 10% are resolved. Agentic coding tools could close this gap, but no benchmark evaluates them on vulnerability-driven upgrades in real, runnable projects across ecosystems, verified independently of the agent. We present VulnBump: 307 upgrade tasks covering 2,066 fixable vulnerabilities in 43 repositories across six package managers, matched to Snyk Open Source production vulnerability distributions. An upgrade passes only if, in a fresh container, dependencies resolve, a rescan shows the targeted vulnerabilities fixed, the project builds, and every previously passing test still passes. A non-agentic baseline that only applies Snyk's recommended version passes 46.9% of upgrades; with the same model (Claude Sonnet 5.5), Snyk's Evo Remediation Agent passes 90.2%, Claude Code + Snyk scan 77.5% and Claude Code unaided 59.6%. The agents fail in opposite ways: Claude Code mostly leaves the vulnerability in place, whereas most of Evo Remediation Agent's failures break the build or tests. An SCA scanner is a key tool for agents: Snyk's guidance raises Claude Code's PASS rate by 17.9 percentage points; without it, Claude Code often settles on a still-vulnerable version. Yet knowing the target is not enough: given the same Snyk findings, Evo Remediation Agent's purpose-built harness passes 12.7 points more than Claude Code + Snyk scan. Major framework migrations remain hard for all harnesses. Because the agents keep projects working at similar rates, evaluating remediation quality in dependency upgrades requires verifying the fix, the build and the tests together.
Read the full abstract

All authors Gabija Pasiunaite, Ryan McMorrow, Brian Clark, Liran Tal, Luke Tu

Software Engineering · Cross-listed from cs.CV

HybridDualPathNet: Cross-Fusion Attention of EfficientNetV2 and Swin Transformer for Brain MRI Classification of Alzheimer's and Parkinson's Disease

Jarpula Nirjala and Rahul Patel

Separating Alzheimer's Disease (AD), Parkinson's Disease (PD), and Healthy Controls (HC) using structural MRI remains challenging because disease signatures overlap and model performance can be strongly affected by data partitioning. We present HybridDualPathNet, a hybrid architecture combining EfficientNetV2-S for local features and Swin-Tiny for global context through a Cross-Fusion Attention (CFA) module. We evaluate the model in two experiments. First, on the Kaggle APD-3-Class dataset using an image-level split, the model achieves 98.25% accuracy. Second, on a combined Kaggle and Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset containing 17,859 slices, we use a per-source split with patient-wise separation for ADNI subjects. In this setting, the model achieves 73.37% standard accuracy and a macro AUC of 0.880. Source-stratified evaluation reveals a substantial domain gap, with 91.2% accuracy on Kaggle test images and 59.2% on ADNI test images. Temperature scaling reduces expected calibration error from 0.130 to 0.015, while split-conformal prediction achieves 89.5% coverage. Saliency analyses using Grad-CAM++, Integrated Gradients, and occlusion indicate that model activations concentrate on brain tissue. These results highlight the importance of patient-level separation, cross-dataset evaluation, and uncertainty estimation when assessing MRI classification systems. The findings also show that strong performance on a single dataset does not necessarily translate to reliable generalization across imaging cohorts.
Read the full abstract

Software Engineering · Cross-listed from cs.LG

CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum

Junjielong Xu, Huachao Zhu, Alyssa Wu et al. (+2 more)

The pass-all-test binary reward has become a standard recipe for training software engineering agents, but its sparsity eliminates difference of partially correct patches. The test pass-rate is a natural alternative. However, we find that it is not a reliable indicator of patch quality. A functionality being tested can be covered by either one challenging test or one hundred trivial tests, where the majority might be already satisfied pass-to-pass tests. We introduce CodeLift, a reinforcement learning method that dynamically calibrates pass-rate rewards, turning the test suite into an adaptive curriculum. At each iteration, CodeLift groups tests that kill the same patches and patches that pass the same tests, then gives more weight to test groups that fewer patch groups pass. This reduces the impact of majority simple tests and focuses more on what is still challenging. Within each rollout, CodeLift applies the same scoring rule to repository snapshots, assigning credit to useful actions even when later ones introduce regressions. Experiments on SWE benchmarks show that CodeLift consistently improves pass@1/2/4/8 over binary and pass-rate baselines. Analyses show that CodeLift provides dense, stable learning signals and unlocks the inherent curriculum in the test suite.
Read the full abstract

All authors Junjielong Xu, Huachao Zhu, Alyssa Wu, Zhichao Hu, Pinjia He

Software Engineering

Deleting Code Without Losing Knowledge: Regenerative Software for Python Libraries

Thomas Bartz-Beielstein

In the age of "Software 3.0", when a complete program can be rewritten in minutes by an agent, the traditional approach of relying on code as the sole source of truth becomes questionable. Fowler advocated for a set of seven primitives, which lay the framework for regenerative software, so that programs can be deleted and rebuilt without losing their knowledge. The knowledge lives in machine-readable records in a tracked directory, from which a *regeneration bundle* is built for each unit. The regeneration bundle is a single file with the module's architecture, intent, decisions, public contract, evaluations, and consumers, but no implementation, plus the one dependency module it may import. The open-source package spotregen implements these primitives and provides a graphical user interface. This report presents a first application of Fowler's primitives to the safety-critical forecasting library spotforecast2-safe. During the first experiments, the Claude Code session that orchestrated the study consumed more than 43 million tokens, almost all of them its own history re-sent with every request. This motivated the switch to an architecture that is built on local models. Deterministic checks and the maintainer judged the results. The report contains executable examples for every primitive. Each example runs spotregen against the repository or against a throw-away copy of it.
Read the full abstract

Software Engineering · Cross-listed from cs.AI

The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express

Lovedeep Gondara, Jekaterina Novikova, Sagarika Suresh et al. (+2 more)

When an LLM judge filters instruction-tuning data, its scoring format (binary verdict, 1–5 rating, 0–100 score, or comparison) is rarely justified or ablated. Holding the candidate pool, judge prompt, decoding and selection budget fixed, we score 15,406 UltraFeedback candidates under five formats with three judges, curate, fine-tune 66 models, and evaluate them with four judges. A 0–100 scale is worth only 7–12 equally used levels, two to three times a 1–5 scale rather than twenty, and a binary verdict leaves the top score tied in 77–78% of prompts, so most selections are made at random. These properties are set by the format almost independently of the judge. Kendall's $\tau_b$ hides them: it ranks binary best while $\tau_a$ and top-1 agreement rank it last, because it discounts the ties a selector must break at random. Both comparative formats inherit a slot bias whose sign differs across judges. Downstream, curated data beat random selection under every evaluation judge, robust to length and markdown adjustment, but no ordering among formats is detectable. We propose four quantities that judge-based curation pipelines should report.
Read the full abstract

All authors Lovedeep Gondara, Jekaterina Novikova, Sagarika Suresh, Jack Ji, Jithin Pradeep

Software Engineering

AIPOCH Open-Science: An Open-Source AI Research Workbench with Reproducible .science Packages

Wen Ouyang, Bocheng Huang, Qianyu Yao et al. (+76 more)

AI-assisted research leaves instructions, computations, artifact revisions and checks whose relationships matter when another researcher receives the work. We present AIPOCH Open-Science, an open-source research workbench with portable .science session packages, and specify an exchange contract for these recorded relationships. Version and assessment targets remain attributable while receiving-workspace identifiers can be remapped, eligible contents can be omitted explicitly, and imported history remains separate from a writable fork. A controlled session tested an incorrect first version, a reviewed correction, an unreviewed revision and a planned but unexecuted step. We compared complete and partial exports with source records extracted independently of the exporter, then inspected imports into separate projects within the same installation. Both conditions retained the tested version and review associations, kept the unreviewed revision distinct from earlier assessments and represented excluded content as unavailable. A documentation-rich ordinary directory also supplied evidence for all seven static inspection questions. A separate registry case connected a table to its recorded task, execution and review; offline execution of its retained script and fixed input reproduced two saved CSV tables byte for byte. The study contributes an implemented exchange contract and a controlled examination of its preservation rules, content-selection constraints and interpretation boundaries.
Read the full abstract

All authors Wen Ouyang, Bocheng Huang, Qianyu Yao, Fei Sun, Minying Deng, Haoyue Cheng, Wenwen Zhou, Lei Mou, Yusi Wei, Jiang Li, Kun Zhu, Lei Wang, Yi Feng, Jiaheng Dai, Ke Qian, Dan Wang, Yahui Wang, Chenyi Wang, Mingsheng Lyu, Yaqiang Zhang, Ping Zheng, Chunhua Tang, Xinge Ma, Xu Zhang, Youjia Fan, Can Lu, Zhijun Dai, Man Hu, Qiqi Xie, Beinan Liu, Yuanning Guo, Yunze Liu, Jinhong Jiang, Xianjun Fu, Mingwei Zhang, Yuanji Xu, Shilin Xia, Bowen Yang, Mingjun Yu, Jingdun Xie, Xuwei Chen, Xia Zou, Yuzhou Xue, Jianguo Zhou, Chunhao Cao, Donghao Tian, Lei Huang, Jinxiao Li, Shengrong Long, Zefeng Song, Honglin Li, Jiang Luo, Wenying Jiang, Shanshan Li, Lei Dai, Ning Jiang, Ying Cai, Chaohui Li, Chunyan Peng, Hetong Zhao, Angqing Li, Chunyan Yang, Chongyang Sun, Yong Li, Xinyu Shi, Wei Dai, Ming Peng, Chen Sang, Yuanyuan Han, Desong Qiu, Haixin Miao, Jian Xu, Yi Chen, Xiaoliang Shi, Tian Shen, Teng Hu, Shuhao Wang, Yingyong Hou, Huimei Wang

Software Engineering · Cross-listed from cs.CR

CWE-bench: Refocusing Agentic Security from Exploitation to Defense

Gonzalo Gonzalez-Pumariega, Ajay Jagannath, Adit Jain et al. (+3 more)

Cybersecurity evaluations of AI agents have focused largely on offense, yet a working exploit does not show that an agent can make software safe. An attacker needs to exploit only one vulnerability; a defender must find and fix all of them. We introduce CWE-bench, a private benchmark of 120 held-out repository audit-and-repair tasks spanning 73 CWE types, eight languages, and all OWASP Top 10 2025 categories. Agents are given a repository to audit and a description of its threat model, but are not told the weakness, where it occurs, or how many vulnerable sites there are. Each patch must block the exploit while preserving original functionality, as checked by a programmatic verifier. We validated the verifier against a panel of agentic judges, manually reviewing disagreements during development and refining the checks. The best agents reach 68% pass@1 and 81% pass@4. We further conduct analyses showing that (i) agents with the same pass@1 differ widely in retry gains and cost, and (ii) no single agent is best across languages or vulnerability categories, and difficulty varies far more by vulnerability category than by language. As the offensive capabilities of agents improve, CWE-bench provides a broad and rigorous measure of whether their defensive capabilities keep pace.
Read the full abstract

All authors Gonzalo Gonzalez-Pumariega, Ajay Jagannath, Adit Jain, Guram Gogia, Soumyadeep Bakshi, Nazneen Rajani

Software Engineering · Cross-listed from cs.DB

ML Engineering Agents - A Data Management Perspective

Arnab Phani, Elias Strauss, Tim Schuster et al. (+1 more)

Machine learning engineering (MLE) agents are emerging as a new paradigm for automating end-to-end ML pipeline development. Using large language models for code generation and predictive performance as a machine-gradable feedback signal, these agents iteratively synthesize, execute, and refine pipeline code. Yet current systems are largely evaluated on pre-built, single-table benchmarks that abstract away key data management challenges. This tutorial introduces the foundations of MLE agents and highlights research opportunities for the database community. We cover how agents combine program synthesis with iterative and evolutionary optimization, and discuss open challenges in efficient pipeline execution, data discovery and augmentation over heterogeneous sources, reliability and debugging, as well as fairness and compliance. We connect these challenges to existing data management techniques and conclude with a hands-on session using contemporary MLE agents.
Read the full abstract

All authors Arnab Phani, Elias Strauss, Tim Schuster, Sebastian Schelter

Software Engineering

DeepGroundRepair: From API-Grounded Generation to Shape-Aware Repair for Deep Learning Programs

Xiangyue Ma, Zhen Wang, Xuhui Lu et al. (+5 more)

Deep learning programs are fundamental to modern AI applications, and automatically generating them from natural language requirements can substantially reduce DL development effort. Large language models have demonstrated promising capabilities for this direction. However, generated DL programs often remain unreliable because they must satisfy domain specific API usage constraints and tensor shape dependencies, violations of which can cause API level syntactic issues and runtime failures. Existing retrieval augmented generation and feedback guided code repair methods do not adequately address these DL specific constraints. In this paper, we propose DeepGroundRepair, a three stage framework that establishes a generation localization repair workflow to improve the reliability of LLM-generated DL programs. Its key idea is to progressively narrow the generation and repair spaces using DL specific knowledge and runtime tensor information. Specifically, it grounds generation in task-relevant API knowledge, localizes the upstream causes of tensor shape mismatch failures, and uses runtime tensor shapes to guide parameter repair. We evaluate DeepGroundRepair on the 100 model-level DL programming tasks in DeepEval using six open-weight LLMs across TensorFlow, PyTorch, and PaddlePaddle. We compare it with four baselines: two generation-only methods and their counterparts incorporating traceback guided self debugging. The results show that, relative to these baselines, DeepGroundRepair reduces the combined number of API-level syntactic issues by an average of 5.4 per 10 program files and achieves an average relative improvement of 85.3% in code executability. Compared with the two self debugging baselines, it improves semantic consistency by an average of 18.1% and reduces token cost by an average of 28.4%.
Read the full abstract

All authors Xiangyue Ma, Zhen Wang, Xuhui Lu, Xiaoting Du, Chenglong Li, Jun Li, Haotian Li, Zheng Zheng

Software Engineering · Cross-listed from cs.CL

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?

Zhiqing Zhong, Mingyu Chen, Aoyang Fang et al. (+5 more)

As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenarios that thoroughly exercise the code, but current benchmarks provide limited insight into whether agents can construct such evidence themselves. We introduce SpecGuardBench, a benchmark of 291 tasks from 13 real-world repositories, covering 8 types of verification logic such as functional postconditions and interaction protocols. Given a natural language specification, repository context, and a reference implementation, an agent constructs a validation harness comprising executable checks and test scenarios. The checks should reject implementations that violate the specification while accepting those that satisfy it, and the scenarios should thoroughly exercise the code. Across four proprietary and open-weight coding agents, none constructs a harness scored as faithful on the benchmark's implementations for more than 63% of the tasks. The bottleneck lies less in exploring test scenarios than in constructing checks that faithfully reflect the specification. Specifically, the agents show a clear tendency to write checks that mirror the reference implementation rather than the specification, and therefore reject implementations that the specification permits.
Read the full abstract

All authors Zhiqing Zhong, Mingyu Chen, Aoyang Fang, Yuzhong Zhang, Haoye Tian, Lionel Briand, Boxi Yu, Pinjia He

Software Engineering · Cross-listed from cs.AI

Same Operation, Different Permission: Evaluating Whether Guardrails Honor Authorization

Huihao Jing, Shaojin Chen, Wenbin Hu et al. (+5 more)

As language-model agents write code, move money and change records on behalf of users, deciding whether an action is authorized has become a central safety problem. Guardrails are often placed in front of these actions, yet few evaluations ask whether their decisions follow authorization: blocking unauthorized actions while still allowing the same actions under valid permission. We introduce \bench{}, a benchmark that contrasts authorization evidence for the same operation in its language track. It combines expert-annotated natural-language scenarios with executed and verified operations, plus externally sourced native-policy requests. \bench{} checks each decision for correctness, usability and resulting effect, and tracks each request across guardrail changes. Our experiments show that many blocked operations stem from failed reviews, not correct denials. Repairing the output format restores usable decisions but changes which authorized requests are honored, and adding a second reviewer removes some errors while rejecting valid authorization. Guardrails should therefore be evaluated on both the unauthorized actions they stop and the authorized actions they let through.
Read the full abstract

All authors Huihao Jing, Shaojin Chen, Wenbin Hu, Yuxuan Liu, Haochen Shi, Zhongwei Xie, Haoran Li, Yangqiu Song

Software Engineering

Domain-Knowledge Driven Configuration Tuning of Kubernetes Workloads

Federico Imberti, Luca Cavagnis, Stefano Cereda et al. (+2 more)

Industrial software applications often require engineers to tune the software deployment platform. Configuration tuning identifies valuable configurations that minimize costs while ensuring the application performance. However, finding these configurations is difficult. The number of tunable configuration parameters in industrial applications is large, and engineers cannot assess different configurations in production because users may experience performance degradation. This work targets Kubernetes: A system for automating the deployment, scaling, and management of containerized applications. This paper proposes a domain-knowledge-driven solution for finding configurations of industrial applications in a practical time. The solution relies on a catalog of constraints encountered by domain experts in similar applications. Depending on the problem under analysis, engineers can customize the optimization problem by selecting the relevant constraints from the catalog for the application at hand. We rely on two alternative solvers to find an optimal configuration: The first is based on heuristics, while the second reuses existing optimization-based solutions. We implemented our solution within an industrial tool from a software company specializing in performance engineering and optimization. We compare our solution to a baseline, considering a set of predefined configuration values provided by engineers. Our results confirm the effectiveness of our solution: Compared to the baseline, the heuristic-based and optimization-based solutions provide significant average savings of 11591 USD/mo and 15874 USD/mo per Kubernetes, respectively, accounting for 24.03% and 32.90% of the costs, in practical time (0.71 seconds per extraction for the heuristic-based and 45.54 seconds per extraction for the optimization-based solution).
Read the full abstract

All authors Federico Imberti, Luca Cavagnis, Stefano Cereda, Giovanni Gibilisco, Claudio Menghi

Software Engineering

Software Testing Meets Bayesian Workflow: A Literature-Driven Artifact Study of Probabilistic Programs

Oleg Jarma, Maja Dall, Raúl Pardo et al. (+2 more)

Automated testing is an effective software quality assurance method, but it is rarely used for probabilistic models. This is despite the heavy use of probabilistic programming for data analytics in research and engineering, where the consequences of errors are dire. A key obstacle is the specification of testable correctness properties for probabilistic models, known as the oracle problem. Like machine learning, probabilistic programming is used precisely where no analytical specification exists: had we been able to write one, we would have written an algorithm instead of fitting a model. An alternative is partial, proxy correctness specifications. The statistics literature already offers many, but as quality guidelines that require human judgment: comparing plots, visually analyzing shapes of distributions, assessing whether parameter values are ”far” or ”close enough”. These guidelines are not systematically automated. In this work, we elicit, implement, and study testable correctness properties of probabilistic models. We extract 17 broadly applicable properties for probabilistic models from one of the most widely used textbooks in Bayesian data analysis, Statistical Rethinking by McElreath. For each property, we select and implement a test oracle, and study its utility for automatic testing, using a population of 59 probabilistic models from academic and industrial sources. We find realistic use cases (models under test) for all the properties. We observe that the vast majority of tests are not flaky in practice, despite the theoretical potential for flakiness. To the best of our knowledge, we are the first to investigate the effectiveness of mutation testing for probabilistic models, showing that standard mutation operators do not capture faults tracked by the probabilistic properties.
Read the full abstract

All authors Oleg Jarma, Maja Dall, Raúl Pardo, Mahsa Varshosaz, Andrzej Wąsowski

Software Engineering · Cross-listed from cs.AI

EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

David Nukrai, Bar Alon, Nativ Levy et al. (+6 more)

Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily evaluate development toward fixed specifications. We introduce EvoWeb, a novel benchmark comprising 320 cumulative requests over 32 web applications, grounded in real-world product features and structured by functional dependency graphs. Agents iteratively modify a persistent codebase, while a multimodal web agent tests each intermediate application in a live browser, assessing newly requested functionality, cross-request integration, and previously requested functionality. To characterize reliability throughout development, we integrate trajectory performance across user-tolerance thresholds rather than impose a fixed stopping criterion. Experiments across open- and closed-source frontier models show that EvoWeb reveals clear performance differences between models, with cross-request integration emerging as a key challenge. Further analysis shows that later requests with deeper or more numerous dependencies are solved less accurately, while runtime errors become more frequent over successive requests. EvoWeb provides a testbed for evaluating and improving coding agents in long-horizon, multi-request development.
Read the full abstract

All authors David Nukrai, Bar Alon, Nativ Levy, Ido Andrew Atad, Ido Cohen, Itamar Zimerman, Lior Wolf, Eli David, Ilan Naiman

Software Engineering · Cross-listed from cs.AR

SafeDSE: Joint Exploration of Safety Architecture Patterns, Mapping, and Scheduling for Dataflow Applications With Safety Arguments (Extended Version)

Fatemeh Bahrami and Seyed-Hosein Attarzadeh-Niaki

Embedded platforms increasingly host safety-related streaming functions next to less critical ones. Whether a safety architecture pattern, such as a doer/checker pair, protects a function depends on the cores that run its components. Their schedule must meet the throughput requirement, and mixing safety integrity levels (SILs) on one core raises development cost. This report presents SafeDSE, a design space exploration framework for synchronous dataflow applications. One declarative constraint model decides the pattern of each actor, the placement that the fault model requires, the static-order schedule, and the platform, and prices SIL promotion. Guards activate the chosen patterns in a fixed graph that holds all of them, and node potentials encode throughput. An independent verifier re-checks every solution, and a generator turns its checks into evidence of a safety argument in Goal Structuring Notation. On 9 benchmark instances under period bounds, no rule that fixes patterns before mapping was both always feasible and never more expensive under conservative pricing, while joint exploration was. In the generated arguments, 49 to 62% of the leaves cite verifier checks; the rest are claims a mapping cannot discharge. Optimality within the model was proven for all synthetic graphs with up to 16 actors. This extended version adds the full model, the verifier and generator in detail, the complete catalog and parameters, and a proof of the potential characterization. It also reports every experiment per instance, a mutation analysis of the verifier, and the negative results of the development.
Read the full abstract

Software Engineering

When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark

Junjie Chen

Agent evaluation harnesses are software systems that run test suites, generate inputs, and decide outcomes with test oracles. This paper audits such software for reliability defects that score tables cannot reveal. We introduce an oracle-integrity auditing approach with five state-integrity invariants and three known-answer controls. We evaluate it by injecting seven oracle defect classes across two task domains and a synthetic domain. All injected defect classes are detected, and the repaired oracle has no false positives. Applied to an agent benchmark, the audit reveals a reward-function defect that changes the environment state before post-run scoring. Across 4,320 episodes, replay-based correction changes database-pass labels from 4,194 to 2,883 and reduces measured success from 94.3% to 65.5%. We also study configuration-dependent measurement effects. A component ablation over 3,240 episodes shows that the agent's policy documents, tool descriptions, and tool error strings influence observed trajectory reliability, with tool descriptions showing the largest exploratory effect (-9.2 pp) and error strings the smallest (-5.3 pp). A second task domain reverses the sign of the language effect, so the result is bounded to the tested configuration. Finally, we report a portability boundary: the auditing approach cannot validate a second independently implemented benchmark because the required reference terminal state cannot be reconstructed from its recorded API-call log. The paper releases a replication package containing the auditing suite, fault-injection catalogue, frozen records, deviation logs, and offline analysis entry points without network access, API keys, or model calls.
Read the full abstract

Sunday 11 October

Software Engineering · Cross-listed from cs.AI

Small but Mighty: Test Suites as Improvement Signals for Self-Evolving Agents

Yutong Cheng, Wei Cheng, Wenchao Yu et al. (+2 more)

A self-evolving agent rewrites its own harness and must decide which rewrites to keep, so its improvement signal is the check a rewrite has to pass. Verifying a rewrite by agent rollout is expensive and subjective, while executable checks such as the repository's own tests are too sparse to catch behavior-changing rewrites, rejecting only a quarter of them. We instead take test feedback as the executable signal, and search for test suites that are both small and strong with a quality-diversity evolutionary algorithm, so that the signal is dense and informative. TeSearch treats suite size as a behavior descriptor, so that quality is pursued at every size and a small, strong suite is reachable, and learns a transition kernel during the search that denoises the LLM as an actuator on that descriptor. Across 129 real-world modules and three backbones, TeSearch delivers the highest-quality suites of any method compared, LLM-driven evolutionary search included, and the smallest of any LLM-based one. Used as the improvement signal of a self-evolving agent, it rejects 59.9% of behavior-changing rewrites against 25.3% for the repository's tests, at a 4.0% false-alarm rate and 3.5 s per decision. The code and data are available at https://github.com/ccsnow127/TeSearch.
Read the full abstract

All authors Yutong Cheng, Wei Cheng, Wenchao Yu, Dawei Zhou, Haifeng Chen

Software Engineering

BRQGen: Towards Adaptive Completeness Assessment and Follow-Up Question Generation for Bug Reports

Tevfik Emre Sungur, Akif Emre Köşüş, Uğurcan Doğruçam et al. (+1 more)

Many bug reports lack the necessary information required to reproduce issues. In early studies, large language models have been employed to generate follow-up questions to enhance incomplete reports. However, they ask a fixed number of questions regardless of whether the report is incomplete. We propose BRQGen, an iterative framework that identifies missing or ambiguous information in bug reports and generates targeted follow-up questions, only if necessary. Reports with no identified gaps are classified as sufficient. When the author responds, BRQGen evaluates whether the answers provide the requested information and generates further questions for any exposed gaps. We curated a novel dataset of 429 incomplete bug reports from Home Assistant and Brave Browser, each containing at least one maintainer follow-up question. Each question is mapped to the author's answer and assigned a label indicating whether the requested information could have been provided directly through the issue template. Earlier maintainer requests were associated with author responses, whereas delayed requests were associated with no response. Among responding authors, requests within one hour received median replies after 0.52 hours in Home Assistant and 1.45 hours in Brave, versus 9.76 and 13.49 hours after four hours. BRQGen covered 35.4% (Qwen 3.8 Flash) to 51.8% (DeepSeek V4.1 Flash) of the 1,073 ground-truth follow-up questions. DeepSeek V4.1 Flash managed to ask 45.7% of all detailed questions and 63.9% of all generic questions. Gemini 3 Flash Preview reached the highest macro-F1 for completeness assessment of 49.3%, but with a post-hoc analysis by getting only the gaps with high severities, Qwen 3.8 Flash reaches 73.6% macro-F1 score. Completeness assessment improved when at least one identified gap had high severity, and nearly all ground-truth questions were captured within the first two iterations.
Read the full abstract

All authors Tevfik Emre Sungur, Akif Emre Köşüş, Uğurcan Doğruçam, Eray Tüzün

Software Engineering · Cross-listed from cs.LG

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair

Elad Sharony, Ido Greenberg, Jialin Song et al. (+3 more)

Large language model (LLM) code-repair systems fix faulty programs by repeatedly generating and testing revisions. Each revision is costly, making the choice of which repair to pursue important for efficiency. Existing search policies rely on test pass rates and search statistics, ignoring the details of test failures and proposed edits. We introduce CodeTriage, a learned policy that prioritizes edits before their revised programs are generated and tested. Its 0.6B-parameter model reads the test output, failure diagnosis, and proposed edit, then predicts a distribution over the number of repair steps needed to reach a fix through that edit. These predictions prioritize all pending edits across programs before generating and testing the revised code. On held-out repair tasks, CodeTriage finds fixes with 21% fewer program revisions and 12% fewer output tokens than the strongest baseline on each metric. Without retraining, it achieves the highest average success across search budgets for each of 11 unseen repair LLMs. Ablations show that both the failure context and edit text, as well as comparisons across programs, improve search. We also release a corpus of 12.2 million program versions from four LLMs, organized into repair trees with proposed edits and test results.
Read the full abstract

All authors Elad Sharony, Ido Greenberg, Jialin Song, Shie Mannor, Gal Chechik, Eli Meirom

Software Engineering · Cross-listed from math.ST

Separating Statistical Inference from Experimental Runtime: An Architecture for Sequential Scientific Analysis

Alexander Stepanov and Andrej Novikov

Sequential statistical procedures are increasingly embedded in scientific systems in which observations arrive incrementally from heterogeneous sources. In such systems the statistical update is only one part of the computation: observations must also be acquired, aligned across inputs, ordered, delivered, persisted, and replayed, and results must be passed to systems that display them or act on them. Sequential methods are nevertheless missing from the general software toolkit. They run inside proprietary experimentation platforms and commercial statistical packages, or exist as offline functions over complete data, and the batch fit interface that dominates machine-learning libraries has no place for a procedure that listens, keeps state, and decides when to stop. We propose an architecture that separates statistical inference from data topology, execution runtime, run lifecycle, persistence, and external action. Its central abstraction is the Sequential Procedure: a state-transition function that maps an inferential state, a valid statistical input, and a statistical random state to a new state and a scientific output. The procedure does not define how observations are acquired, paired, scheduled, stored, or acted upon. We specify the architecture semantically rather than through an object model, derive seven conformance invariants that any implementation can be tested against (among them synchronous/asynchronous, checkpoint, replay, and batch equivalence), and describe a dependency-free reference implementation in which all seven hold bitwise. Lorden's 2-SPRT for the Kiefer-Weiss problem serves as the reference procedure; the same contract accommodates sequential estimators, Bayesian filters, change-point detectors, and stateful predictive models.
Read the full abstract

Software Engineering

Grounding Deep Learning Framework Testing in Data Flow

Jiatao Ge, Yanzhou Mu, Yinglong Zou et al. (+3 more)

Data flow underlies deep learning (DL) framework execution, including automatic differentiation and operator-fusion optimization. Bugs involving data flow can cause incorrect results or unexpected behavior. Existing testing methods represent models primarily by operator topology, omitting runtime tensor values and properties and operator input requirements. They therefore cannot distinguish data flows in models with similar topologies, even when these differences affect framework behavior. We introduce the Tensor Flow Graph (TFG) to capture runtime tensor states and operator input requirements. Its nodes represent operators and their input requirements, while its edges record tensor values and properties. Building on the TFG, we propose FlowMuT, a mutation-based testing method grounded in data flow. Mutations are represented as constrained subgraph replacements, with mutation operators derived from prior work and an analysis of mutable data flow objects. In each round, FlowMuT builds a TFG for the seed model, pairs mutation operators with TFG sites, and filters illegal candidates using data flow constraints. It uses LinUCB to select a candidate based on coverage and impact, executes the mutant, updates selection feedback, and uses each accepted mutant as the next seed. We evaluate FlowMuT on PyTorch and MindSpore using representative models from diverse application scenarios. FlowMuT submitted 55 bug reports for cases missed by all baseline methods; developers confirmed 41 reports and fixed the bugs described in 7 reports. FlowMuT achieves 43.4% higher data flow diversity than the strongest baseline and a legal mutant rate of 96.7%, compared with 86.4% without constraints. These results show that testing from a data flow perspective is practical and effective.
Read the full abstract

All authors Jiatao Ge, Yanzhou Mu, Yinglong Zou, Yibo Hu, Zhenyu Chen, Chunrong Fang