Software Engineering
LumiXAI: A Modular Full-Stack Framework for Feature Attribution
Alfio Ferrara, Lorenzo Gatta, Sergio Picascia et al. (+1 more)
Feature attribution is a central tool of model interpretability, yet the software through which it is applied remains fragmented: individual tools specialize along narrow axes, such as a single modality, a code API or a GUI, or a fixed rather than extensible method set, and rarely combine these strengths. Moreover, many explainability tools are designed primarily for domain experts, requiring programming skills or familiarity with attribution methods that can make them difficult for non-expert users to access. In this article, we present LumiXAI, a modular full-stack framework that consolidates attribution analysis into a single system. It couples classification and generative attribution with an interactive GUI supporting bidirectional exploration, a plug-in architecture for registering new models and methods, and three access tiers serving non-programmers, developers, and extenders from one backend. Its contribution is a system that operationalises established attribution methods under one interface, one interaction model, and one persistence layer, with containerised services and persistent results making analyses reproducible across machines.
Read the full abstract ▾
All authors Alfio Ferrara, Lorenzo Gatta, Sergio Picascia, Elisabetta Rocchetti
Software Engineering · Cross-listed from cs.AI
PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents
Zhi-Kai Chen, Xu-Xiang Zhong, Song-Yan Li et al. (+2 more)
LLM
agents increasingly solve tasks by invoking multiple tools, where parallel execution is essential for low latency but difficult to manage safely. Existing
agent benchmarks primarily evaluate tool selection, argument generation, and end-to-end success under mostly serial execution, largely overlooking valid parallelization and resource-constrained scheduling. This missing scheduling dimension creates a practical failure mode: serial execution is safe but slow, while resource-agnostic parallel execution is fast but prone to avoidable resource overflows. To address this gap, we introduce PeakBench, a
benchmark of executable multi-tool workflows with execution-grounded dependency annotations and measured resource profiles. A central challenge in evaluating such workflows is attribution: failures and inefficiencies may arise from incorrect dependency planning, poor resource-constrained scheduling, or both. PeakBench addresses this challenge with a two-part evaluation framework that disentangles logical planning from physical scheduling, with dedicated metrics for each dimension. Using this framework, we show that strong logical planning does not reliably translate into safe or efficient execution under resource constraints. We further show that exposing resource information can reduce avoidable overflows and improve resource utilization, making PeakBench a useful testbed for diagnosing resource-aware
agent behavior. Code is available at https://github.com/Czzzk/Staggering-the-Peaks.
Read the full abstract ▾
All authors Zhi-Kai Chen, Xu-Xiang Zhong, Song-Yan Li, De-Chuan Zhan, Han-Jia Ye
Software Engineering
Ockhamareto: Pareto-Gated Segment-Level Credit Assignment for Concise Unit-Test Generation with Reinforcement Learning
Dong Huang, Mark Harman, Jie M. Zhang et al. (+3 more)
We introduce Ockhamareto, a single-shot GRPO framework for unit-test generation and selection, based on the principles of Ockham's Razor and Pareto Optimality. Ockhamareto has two principal components: (i)~a Pareto-gated Bonus that rewards only rollouts non-dominated in~(mutation, $-$\#tests) space, and (ii)~
Token-level Segment Credit, which attributes each test's marginal mutation kills back to the
tokens of its unit-test block. On the UnLeakedTestBench~(ULT), Ockhamareto strictly Pareto-dominates the strongest RL baseline~(MIST-RL). Furthermore, it dominates on each and all optimization objectives, catching more bugs ($49.9\%$ vs $31.3\%$ mutation score at $N{=}5$), using fewer tests ($2.60$ vs $4.67$ on average), thereby achieving $3.4\times$ the per-test trade-off improvement. The advantage is found in all four
benchmarks~(HumanEval+, MBPP+, CodeContests, TestGenEval-Lite): Ockhamareto leads both mutation and coverage metrics on every one, always with the smallest suite. Ockhamareto also outperforms the state-of-the-art at all model scales, adding $+30$–$35$~pp mutation at 4B, 9B, and 27B model sizes. We also show that the knee point of the optimal trade-off between efficiency and effectiveness on the Pareto front is not correlated with obvious more easily computed proxy metrics, such as function size. This finding motivates the Pareto front computation; it is needed to identify this crucial engineering trade-off for each function under test.
Read the full abstract ▾
All authors Dong Huang, Mark Harman, Jie M. Zhang, Zhijiang Guo, Mingzhe Du, See Kiong Ng
Software Engineering · Cross-listed from cs.AI
From State to Action: OODA-Tool for Reliable Multi-Turn Tool Use
Rongfeng Guo, Yinxuan Huang, Yusen Wu et al. (+5 more)
Reliable multi-turn tool use requires an
agent to preserve an evolving task state and ensure that each action remains consistent with it. However, direct function-calling and ReAct-style policies learn state tracking and action generation within the same autoregressive trajectory. This coupling creates state-action competition: the pressure to produce the next call can overwrite or ignore information accumulated earlier in the interaction. Inspired by Boyd's Observe-Orient-Decide-Act cycle, we introduce OODA-Tool, a typed closed-loop policy designed to mitigate this competition by separating state preservation from action realization. Rather than generating an action directly from the interaction history, OODA-Tool routes each decision through controller-checked intermediate states, ensuring that the final output remains grounded in the current task state. Specifically, Observe reconstructs the task state, Orient determines whether execution is warranted, Decide forms an admissible action structure, and Act realizes the external output. We evaluate OODA-Tool against direct function-calling and ReAct policies using Qwen3 models ranging from 0.6B to 14B across multi-turn, multi-tool, and incomplete-information settings. OODA-Tool consistently improves task success across model sizes, with larger gains on smaller models and on tasks whose actions depend strongly on information accumulated across turns and prior tool results. Controlled variants, stage-level ablations, and transfer evaluations further demonstrate the robustness of these improvements.
Read the full abstract ▾
All authors Rongfeng Guo, Yinxuan Huang, Yusen Wu, Maoqing Zhong, Yunlu Chen, Meng Tang, Teng Long, Vincent Tao Hu
Software Engineering · Cross-listed from cs.DC
Mixed-Precision SEM-Based CFD Simulations on GPUs: A Taylor-Green Vortex case
Yanxiang Chen, Manuel Münsch and Roman Iakymchuk
Mixed precision is a promising approach for reducing the computational cost and energy consumption of Computation Fluid Dynamics (CFD) simulations, but its effectiveness depends strongly on where precision is reduced within the full simulation pipeline. In this work, we study Taylor-Green
vortex case using Neko, a
matrix-free CFD solver based on the spectral element method (SEM). Profiling shows that the fluid time step is not dominated by Krylov convergence alone: the velocity and pressure solvers require only a small number of iterations per step, while a substantial fraction of runtime is spent in other SEM operators and solver components. Motivated by this structure, we propose a three-level hierarchical mixed-precision control model. Two groups of configurations are evaluated in environments bounded by 64-bit floating-point (fp64) and 32-bit floating-point (fp32) precision, respectively. The fp64-bounded group identifies accuracy sensitive components and shows that SEM-focused fp32 computation is a promising direction for future optimization. The fp32-bounded group provides the main practical benefit. For the high Reynolds number case studied, selected configurations reduce both time- and energy-to-solution by about 34% relative to the fp64 baseline, while improving robustness compared with global fp32. Targeted fp16 kernel overrides are also explored, showing potential for selected operations but increased sensitivity in gradient-based quantities such as enstrophy. Overall, these results indicate that mixed-precision for
matrix-free SEM-based CFD should be treated as a simulation-level control problem rather than solely as a Krylov-solver optimization.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.AI
ReproAgent: Contract-Guided Paper-to-Code Reproduction
Xue Hu, Zewei Pan, Zhongyuan Wang et al. (+3 more)
Paper-to-code reproduction asks scientific AI
agents to turn research papers into executable repositories that preserve the paper's method, protocol and artifacts. This is difficult because the specification is split: explicit paper content such as
algorithms, metrics and artifacts is often lost across long
agent trajectories, while implicit details such as framework defaults and conventions inherited from related work are absent from the paper. We introduce ReproAgent, a four-stage Prepare–Plan–Generate–Repair pipeline built around a persistent implementation contract with two channels: an implementation-requirement channel that turns paper snippets into code obligations, and a reference-evidence channel that retrieves content and structure evidence from related repositories. Both are bound to work packages, projected into file-level contracts, and consumed across generation and repair. On PaperBench Code-Dev, ReproAgent reaches the highest mean score among same-backbone scaffolds under both Claude-Sonnet-4.5 and Gemini-3-Flash. End-to-end channel ablations and per-paper cases support the contribution of both channels. Code and experimental artifacts are publicly available.
Read the full abstract ▾
All authors Xue Hu, Zewei Pan, Zhongyuan Wang, Zhou Liu, Zeli Su, Wentao Zhang
Software Engineering
Observability and Fault Injection for LLM-Based Multi-Agent Systems in Software Engineering
Zahra Seyedghorban, Egor Klimov, Arie van Deursen et al. (+2 more)
Large
Language Model-based multi-
agent systems are increasingly explored for software engineering tasks, but they remain difficult to inspect, debug, and evaluate under controlled failures. We present llmmas-otel, a lightweight and framework-agnostic tool that combines OpenTelemetry-based distributed tracing with fault injection for LLM-based multi-
agent systems in software engineering workflows. The tool instruments
agent executions with trace-aligned telemetry across workflow phases,
agent steps, inter-
agent communication, tool calls, and LLM invocations, and supports targeted fault injection at selected interaction points. This makes it possible to compare baseline and faulty executions in a reproducible way and inspect the effects through aligned traces and run artifacts. We describe the motivation, architecture, implementation, current capabilities, and initial validation of the tool on a minimal demo workflow and a real LLM-based multi-
agent system for software development.
Read the full abstract ▾
All authors Zahra Seyedghorban, Egor Klimov, Arie van Deursen, Annibale Panichella, Burcu Kulahcioglu Ozkan
Software Engineering
Towards LLM-Enhanced Android Taint Analysis
Nicholas Miazzo, Marco Alecci, Jordan Samhi et al. (+2 more)
Taint analysis is a fundamental technique for detecting sensitive data leaks in Android apps. However, traditional static tools, such as FlowDroid, still face well-known challenges due to the complexity of accurately modeling the Android framework. In this paper, we investigate whether off-the-shelf Large
Language Models (LLMs) can effectively reason about taint flows in Android apps. Our preliminary approach relies on an agentic interaction strategy, enabling the LLM to iteratively explore code and reason about data flows. We conduct an initial evaluation on the DroidBench
benchmark against FlowDroid, where our approach outperforms the baseline: Gemini-3 Flash achieves an F1-score of 0.96, compared to 0.55 for FlowDroid. In particular, we observe improvements in challenging categories such as inter-component communication (0.95 vs. 0.17), implicit flows (0.94 vs. 0.00), and reflection (1.00 vs. 0.50), where FlowDroid typically struggles. On a small set of real-world apps, the LLM-based approach also identifies additional potential data leaks not reported by FlowDroid. These preliminary findings suggest that LLM
reasoning may effectively complement traditional static taint analysis, motivating future research on hybrid LLM-enhanced taint analysis pipelines.
Read the full abstract ▾
All authors Nicholas Miazzo, Marco Alecci, Jordan Samhi, Jacques Klein, Eleonora Losiouk
Software Engineering
Cross-Stack Validation of Language-Model Training: A Clinical Fine-Tuning Case Study
Thang Tran and Lan Dang
Neural network training has an oracle problem: a run can converge normally and yield a usable model while the software beneath it computes something other than specified. Almost all such work runs on one stack, so there is rarely anything independent to check against. We study whether independently implemented training stacks can serve as differential oracles for a whole fine-tuning pipeline, rather than the operators and inference paths that prior differential testing targets. We define a trajectory-level protocol – a shared specification, cross-check points spanning arithmetic, model loading, data rendering and the learning trajectory, and a separation of independence of the stack, the orchestration and the
language runtime – and apply it to a LoRA adaptation of Qwen3-0.6B over 168,574 clinical question-answer pairs under PyTorch and under numbat, an independent framework written in Zig, driven natively and through its C interface from six
languages. Across 42 paired evaluations spanning a full epoch the two stacks' held-out cross-
entropy differs by 0.134% on average, and four implementations end the epoch within 0.15% of one another. The comparison exposed 17 faults that single-implementation development had missed, two of them notable for software engineering. The fault with the largest effect on the trained model lay outside the numerical kernels: a mismatch in how clinical text was rendered moved held-out loss 0.15, some 500 times more than the arithmetic faults found beside it. And four faults were reachable only from a
language whose
memory model differs from the first two implementations: a scheduler migrating work across threads, a collector blind to device
memory, an ownership discipline needing a primitive the interface lacked. Implementation diversity has several axes, and the runtime is one.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.AI
SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction
Xue Hu, Zewei Pan, Zeli Su et al. (+2 more)
LLM
agents can generate paper reproduction code, yet often produce scientifically unfaithful implementations. We define this failure mode as semantic drift, where generated code silently diverges from the paper's specifications. We introduce SemanticAlign-Bench(SA-Bench), a diagnostic
benchmark covering 30 papers from ICLR, ICML and NeurIPS 2025. For each paper, we decompose its specifications into atomic and verifiable implementation claims, which we call Semantic
Alignment Units (SAUs) and evaluate repositories along four diagnostic dimensions spanning numerical, methodological, protocol and ordering drift. In total, we construct 1,491 SAUs across five ML domains and evaluate 12 generator configurations (4 models $\times$ 3 scaffolds). Even the strongest configuration (Claude+PaperCoder) achieves a mean SAU score of only 0.301 out of 1.0, with an overall mean of 0.221 across 360 evaluations. A failure taxonomy reveals that
agents attempt most requirements but implement them incorrectly, with implementation mismatch and stubs accounting for the majority of zero-scored claims. Our analysis further indicates that scaffolds optimized for executability provide limited leverage for scientific reproduction; narrowing the gap requires scaffolds that prioritize semantic specification verification. The
benchmark, annotations and evaluation pipeline are publicly available.
Read the full abstract ▾
All authors Xue Hu, Zewei Pan, Zeli Su, Zhou Liu, Wentao Zhang
Software Engineering
Mutation Testing of Simulink Cyber-Physical System Models: Challenges and Solutions in Practice
Murat Kavak, Onur Kilincceker, Serge Demeyer et al. (+2 more)
Several experience reports illustrate that mutation testing is capable of supporting a shift-left testing strategy, especially in industries where late bug discovery incurs very high costs and risks. In the context of cyber-physical systems, a shift left implies that engineers need to test the design models used to simulate, prototype, and analyse the feasibility of the system under design. In this paper, we analyse the challenges we encountered and the lessons we learned when incorporating mutation testing in the context of DUCO, a company producing ventilation systems for buildings. The engineers within DUCO have years of experience with model-based engineering centered around Simulink and StateFlow, including unit tests for their models. During a pilot project with our tool prototype MUT4SLX we learned that equivalent mutants, requirement traceability, and mutation testing for Stateflow represent particular challenges not yet reported in the academic literature.
Read the full abstract ▾
All authors Murat Kavak, Onur Kilincceker, Serge Demeyer, Kevin Vandenbroucke, Halim Abdurrahman Ceylan
Software Engineering · Cross-listed from physics.med-ph
Validation of HRV Studio: A Transparent and Quality-Control-Aware Platform for Heart Rate Variability Analysis
Cyrus Mexon Evrard Djindot, Faliang Liu, Sylvain Laborde et al. (+6 more)
Reproducibility of heart rate variability (HRV) analysis is limited by differences in preprocessing and computational conventions across software platforms. We developed HRV Studio, an open-source PyQt6-based desktop application integrating transparent HRV analysis with automated quality-control (QC) diagnostics. Validation included large-scale agreement with NeuroKit2, targeted Kubios benchmarking, spectral-method comparison, synthetic perturbation testing, recording-duration sensitivity analysis, and arrhythmia-focused QC stress testing. HRV Studio showed near-identical agreement for the widely used time-domain indices RMSSD and SDNN under matched conditions. In the primary five-minute NeuroKit2 comparison, frequency-domain median relative errors were 1.35% for LF, 0.18% for HF, and 1.41% for LF/HF, while VLF remained more convention-sensitive (37.79%). Nonlinear Poincaré indices also demonstrated high consistency. Sequence-harmonized Kubios benchmarking confirmed near-identical agreement for time-domain and nonlinear indices and strong agreement for most frequency-domain measures. Extended ten-minute analyses reproduced the same overall pattern with lower disagreement for some convention-sensitive spectral outputs. Synthetic and arrhythmia stress tests maintained 100% numerical stability while consistently triggering QC warnings. Overall, HRV Studio provides a transparent and reproducible platform for HRV research, with strong cross-platform consistency when NN sequences, preprocessing, and analytical conventions are harmonized. Stress-test results indicate computational robustness rather than clinical validation.
Read the full abstract ▾
All authors Cyrus Mexon Evrard Djindot, Faliang Liu, Sylvain Laborde, Yinjia Zhang, Jessie Chen, Ming Li, Congrong Wang, Weixiong Rao, Qinpei Zhao
Software Engineering
DeepRepoQA: Code Repository Question Answering with Deep Agent Exploration
Weihan Peng, Yuling Shi, Yingwei Ma et al. (+3 more)
Answering developer questions about a software repository is a critical yet under-explored problem in software engineering. While existing repository understanding methods have advanced the field, they predominantly rely on surface-level code
retrieval and lack the ability for deep
reasoning over multiple files, complex software architectures, and grounding answers in long-range code dependencies. To address these limitations, we propose DeepRepoQA, a novel question answering (QA) framework for repository-level code understanding. DeepRepoQA builds on an agentic framework where LLM
agents find answers through a systematic tree search over the repository structure. A Monte-Carlo Tree Search (MCTS) mechanism is employed to empower
agents to dynamically search, navigate, and inspect code, enabling effective multi-hop
reasoning over long-range code dependencies. Comprehensive experiments on the SWE-QA
benchmark demonstrate substantial performance gains over strong baselines, validating the effectiveness of systematic MCTS-guided exploration for multi-hop repository
reasoning.
Read the full abstract ▾
All authors Weihan Peng, Yuling Shi, Yingwei Ma, Longfei Yun, Beijun Shen, Xiaodong Gu
Software Engineering · Cross-listed from cs.AI
Paritok-4B: Intent-Conditioned Context Compression for Coding Agents
Jiayu Shi and Luzhuo Chen
Coding
agents re-send large file reads and tool outputs to a frontier LLM every turn, and this context dominates their
token bill. General-purpose
prompt compressors are trained on prose and suit code poorly: they paraphrase identifiers and drop the exact spans an
agent needs to edit. We present Paritok-4B, a 4B LoRA compressor for coding-
agent trajectories built on two commitments. It is extractive: it selects spans rather than rewriting them, and 96.0% of the identifiers, paths, and numbers it emits already appear in its input, holding at 96.2% on held-out SWE-bench Lite output. It is intent-conditioned: told the
agent's current task, it acts chiefly inside a retained segment, selecting which lines survive (retained lines are +0.067 more intent-relevant than removed ones, paired 95% CI [+0.056, +0.078]) rather than changing how much is retained. We distil a gpt-4.1-mini teacher over 67,074 real OpenHands trajectories into 40,606 validated examples and fine-tune Qwen3-4B. On all 300 SWE-bench Lite instances, Paritok-4B compresses
agent context to 25.7% of its size, 2.0x harder than a gpt-4.1-mini compressor (50.2%) and 2.4x harder than gpt-5 (61.9%), while retaining 86.5% of uncompressed single-shot solve quality. Fed the cat -n line-numbered input real
agents produce, it compresses slightly less (27.8%) and retains more (89.3%); there the paired test is informative, with 30 instances solved only uncompressed and 17 only compressed, an exact McNemar p=0.079, so at this sample size compressing context to roughly a quarter of its size does not significantly reduce the solve rate. The model is a 264 MB adapter that self-hosts on one 24 GB GPU with no per-
token compressor fee, which at list prices decides the economics: gpt-5 as a compressor is net-negative, costing more than the downstream
tokens it saves. Weights, data, and evaluation scripts are open (Apache 2.0).
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CL
TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis
Boshen Shi, Yize Liu, Chen Zhao et al. (+4 more)
LLMs are increasingly used to analyze spreadsheets, CSV files, and other structured data, but producing a correct-looking answer is not the same as producing a trustworthy analysis. A trustworthy result should be supported by a valid path from the user question to the relevant data evidence. This requirement creates two diagnostic questions: whether an LLM can refuse to answer or ask for clarification when such a path does not exist, and whether it can preserve the correct analysis when the same evidence is expressed in different table forms. We introduce TrustDABench, a
benchmark that operationalizes these questions as reliability and robustness. Starting from the evidence-path view, we derive 19 perturbation operators and instantiate them through an Agentic-LLM-based generation framework. TrustDABench contains 2,340 human-verified perturbed instances, and we evaluate eight representative LLMs. The results show substantial headroom: the best reliability result is only 24.21% average MRS, achieved by GPT-5.5, while the best robustness result still has 9.10% average ASR, achieved by Claude-Sonnet-5. The failures are systematic: models rarely detect conflicting evidence, often continue along executable but unsupported analysis paths, and remain sensitive to perturbations that change observation boundaries or cross-table relations. These findings suggest that stronger evidence-boundary recognition and representation-invariant
reasoning are still needed for reliable structured-data analysis.
Read the full abstract ▾
All authors Boshen Shi, Yize Liu, Chen Zhao, Ce Chi, Zhendong Wang, Xing Wang, Junlan Feng
Software Engineering · Cross-listed from cs.AI
Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping
Yiwen Zhang, Xiaodong Yan, Zhenyu Huang et al. (+6 more)
Reinforcement Learning from Verifiable Rewards (RLVR) is pivotal for enhancing LLM code generation, yet its efficacy is often hindered by insufficient test case coverage, leading to reward hacking and policy degradation. To address this, we propose RobustTests, a framework featuring a faulty-code-driven test case synthesis strategy. By leveraging "near-correct" faulty codes, RobustTests captures latent logical discrepancies and employs validator
agents with behavioral feature clustering to filter invalid or redundant test cases. Additionally, a stepwise dense reward function based on pass rates is introduced to mitigate false negatives and enhance training robustness. Using this pipeline, we construct an augmented version of the CodeContests+
dataset with superior diagnostic utility. Experimental results show that RL fine-tuning of Qwen3-32B via RobustTests achieves a 3% absolute gain on LiveCodeBench, demonstrating its effectiveness in advancing LLM code generation proficiency. Codes and data are available at https://huggingface.co/
datasets/sid6/RobustTests.
Read the full abstract ▾
All authors Yiwen Zhang, Xiaodong Yan, Zhenyu Huang, Deng Zhao, Liang Jiang, Qing Cui, Zujie Wen, Zhiqiang Zhang, Jun Zhou
Software Engineering · Cross-listed from cs.AI
EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals
Mingxu Zhang, Ying Sun, Yuhan Li et al. (+4 more)
Large
language models (LLMs) are increasingly used as code
agents for scientific and engineering analysis, but their ability to analyze raw physical-layer measurements remains untested. We introduce EMRB (Electromagnetic
Reasoning Benchmark), which evaluates whether LLMs can analyze raw I/Q data by writing and running code. EMRB contains 200 problems across five difficulty levels and 27 question types, from signal detection to OFDM design, generated from 11 signal types with verified ground truth. Unlike
benchmarks built on preprocessed features or structured tables, EMRB provides only the raw capture; the quantities each question refers to must first be discovered through code. We evaluate 14 LLMs spanning proprietary, open-weight, and
reasoning-oriented families. Scores range from 24.1% to 78.9%, with the mean dropping from 84.9% on basic measurement to 21.2% on system design. We also propose ReconPilot, a structured method that separates signal reconnaissance, targeted analysis, and self-verification. Across three backbones, ReconPilot raises the overall score by 3.8 to 17.6 points and improves 13 of 15 backbone-level combinations tested. All data and code are publicly released in \href{https://github.com/mingxuZhang2/EMRB}{\textcolor{blue}{our GitHub repository}}.
Read the full abstract ▾
All authors Mingxu Zhang, Ying Sun, Yuhan Li, Yang Ji, Dazhong Shen, Ke Zhang, Shan Huang
Software Engineering
Evaluating Language Models on Cross-Language Code Functional Equivalence
Hui Sun, Anderson Uchôa, Rohit Gheyi et al. (+1 more)
Background: Large
Language Models (LLMs) have demonstrated strong performance across a variety of code-understanding tasks, leading many to believe that they can reason about program semantics. However, existing evaluations primarily focus on single-
language settings or rely on synthetically generated code, raising concerns about whether current results reflect true semantic understanding. Aims: We investigate whether LLMs can accurately judge functional equivalence across different programming
languages in human-written code, a setting that requires deeper
reasoning beyond superficial similarity. Method: We introduce PolyHuman, a
dataset of human-written programs in CPP, Java, and Python. Using this
dataset, we evaluate intra- and inter-
language equivalence detection across open-weight and proprietary LLMs, selecting GPT-o4-mini as a representative model to assess stability. We then manually analyze 81 cases of systematic disagreement in which models incorrectly judge functional equivalence, examining the code logic and the generated Chain-of-Thought
reasoning. Finally, we categorize these failures and compare them across GPT-o4-mini, Claude-Opus-4.7, and Gemini-3-Flash to determine whether they reflect model-specific issues or broader limitations of state-of-the-art LLMs. Results: We identify a difficulty-dependent breakdown in equivalence judgment (harder problems make the model increasingly prone to misclassifying non-equivalent code as equivalent), a model-specific sensitivity to programming
language for the best-performing model (particularly a more conservative behavior on Python), and a partial reliance on similarity-based cues. GPT-o4-mini also shows substantial run-to-run instability under identical settings, indicating inconsistent rather than absent capability. Conclusions: Current LLMs do not reliably capture functional equivalence within or across
languages.
Read the full abstract ▾
All authors Hui Sun, Anderson Uchôa, Rohit Gheyi, Wesley K. G. Assunção
Software Engineering
The Empire, Long Divided, Must Unite: Architectural Convergence in Three LLM Agent Harnesses
Dai Jiahong
An
agent harness is what turns a
language model into an autonomous
agent: the surrounding code that builds the model's context, mediates its tools, runs the loop, and persists state across a long-horizon run. This layer, not the model it wraps, is increasingly the binding constraint on
agent behaviour. We present a source-level, multi-case study of three open coding-
agent harnesses built from deliberately opposing philosophies: LangChain's deepagents (batteries-included), Earendil's pi (radical minimalism), and DeepSeek's dsh (everything-is-a-plugin). Reading each at a pinned commit and following its commit history, we find that the two mature harnesses have travelled in opposite directions (deepagents subtracting authored scaffolding, pi accreting durable infrastructure), yet converged toward one architectural middle form of five recurring elements: a commoditised loop, an append-only replayable session record, model quirks kept as data, progressive disclosure of context, and explicit extension seams. A third harness, read afterward as a held-out check, exhibits all five, and in one seam reuses another's implementation outright. We therefore do not claim independent invention, and decompose the convergence into parallel discovery,
diffusion, and literal reuse. Finally, one load-bearing dimension shows no convergence, and indeed no presence: external verifiability, a tamper-evident record an outside party can check without trusting the runtime. We read this absence not as an oversight but as a predictive gap, the next axis on which harnesses for provenance-sensitive domains will differ.
Read the full abstract ▾
Software Engineering · Cross-listed from cs.CR
SoK: ARCUS: On the Efficiency and Efficacy of Hardware Fuzzing
Alenkruth Krishnan Murali, Raghul Saravanan, Sai Manoj P D et al. (+1 more)
This work presents a comprehensive analysis of contemporary hardware fuzzing techniques applied across three major abstraction layers: Instruction Set Architecture (ISA), microarchitecture, and Register-Transfer Level (RTL). Our study examines key factors including input stimulus quality, mutation strategies, feedback mechanisms, target platforms, reference models, and achieved coverage. We find challenges, goals, and design trade-offs vary significantly across abstraction layers. We further identify several unmet needs in current hardware fuzzing practices, such as intelligent input generation, reliable and scalable golden reference models, expressive feedback channels, and cross-layer integration. Building on these insights, we outline future research directions, including hybrid fuzzing frameworks, AI-assisted test generation, scalable reference models, standardized evaluation metrics and
benchmarks, and human-in-the-loop automation for guided exploration and analysis. Together, they aim to unlock efficient, reliable, and comprehensive hardware verification solutions.
Read the full abstract ▾
All authors Alenkruth Krishnan Murali, Raghul Saravanan, Sai Manoj P D, Ashish Venkat