Computation and Language · Cross-listed from cs.PL
ESBMC-GraphPLC: Formal Verification of Graphical PLCopen XML Ladder Diagram Programs Using SMT-Based Model Checking
Pierre Dantas, Lucas Cordeiro and Waldir Junior
PLCopen XML defines two encoding formats for IEC 61131-3 Ladder Diagram programs: a textual encoding using <rung> elements, and a graphical encoding that represents rung logic as a directed graph of localId/refLocalId connections. ESBMC-PLC supported the textual format but parsed graphical exports from CONTROLLINO, Beremiz, and OpenPLC Editor into an empty GOTO intermediate representation, causing vacuous verification success. This paper presents ESBMC-GraphPLC, which closes this gap with a DFS-based graphical LD resolver. The resolver traverses the connection graph from leftPowerRail to each coil, extracts rung paths as Boolean contact conjunctions, and applies a three-tier I/O inference scheme. Ordering coils by rightPowerRail connectionPointIn sequence ensures SET coils process before RESET coils, matching IEC scan-cycle semantics. The graphical-to-IR conversion leaves the ESBMC backend unchanged. Validation on 3 graphical LD programs from CONTROLLINO/OpenPLC Editor shows all produce full GOTO IR with nondeterministic inputs and rung logic, versus the empty IR previously. All 3 verify SAFE at k=2 under 70ms. The 11 textual LD
benchmarks are fully preserved, with no regression. Two Beremiz examples with no LD content or unsupported timer semantics are reported as discovered limitations. Artifact at Zenodo (DantasCordeiro2026graphical, doi:10.5281/zenodo.20699856).
Read the full abstract ▾
Computation and Language
As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language
Jasmine Owers, Edwin Simpson and Martha Lewis
Figurative
language and negation are two areas that challenge current
language models, however, both are widely used throughout written and spoken
language. Large
language models (LLMs) are also widely used in everyday contexts where they cannot necessarily be tuned for a specific
dataset. It is therefore essential to understand the ability of LLMs to correctly interpret text that includes both negation and figurative
language. To investigate this, we develop a set of new annotations to an existing
dataset of figurative
language, and test a range of
language models on the
dataset. We find that the combination of negation and figurativeness can present a particular challenge, and that performance overall and across different negation types is particularly dependent on the
prompt style used.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
REVES: REvision and VErification–Augmented Training for Test-Time Scaling
Yuanxin Liu, Ruida Zhou, Xinyan Zhao et al. (+6 more)
Test-time scaling via sequential revision has emerged as a powerful paradigm for enhancing Large
Language Model (LLM)
reasoning. However, standard post-training methods primarily optimize single-shot objectives, creating a fundamental misalignment with multi-step inference dynamics. While recent work treats this as multi-turn reinforcement learning (RL), conventional approaches optimize over the multi-step trajectories directly, failing to further exploit the high-quality mistakes in intermediate steps that model can learn from correcting them. We propose a two-stage iterative framework that alternates between online data/
prompt augmentation and policy optimization. By converting the intermediate steps (“near-miss” answers) in the successful recovery trajectories into decoupled revision and verification
prompts, our approach concentrates training on both effective answer transformation and error identification. This approach enables efficient off-policy data generation and reduces the computational overhead of long-horizon sampling compared to standard multi-turn RL. On LiveCodeBench, using publicly available test cases as feedback, we observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training. Beyond coding, our approach matches the previously reported SOTA result on circle packing while using the smallest base model (4B) and far fewer rollouts than the much larger evolutionary search systems. Math results under ground-truth verification further confirm improved correction ability. It also generalizes to out-of-distribution constraint-satisfaction puzzles such as n\_queens and mini\_sudoku, where correctness is defined entirely by problem constraints. Code is available at https://github.com/yxliu02/REVES.git.
Read the full abstract ▾
All authors Yuanxin Liu, Ruida Zhou, Xinyan Zhao, Amr Sharaf, Hongzhou Lin, Arijit Biswas, Mohammad Ghavamzadeh, Zhaoran Wang, Mingyi Hong
Computation and Language
SAGE: Stochastic Prompt Optimization via Agent-Guided Exploration
Ziyi Zhu, Luka Smyth, Saki Shinoda et al. (+1 more)
Context engineering has emerged as a primary lever for improving AI systems without parameter updates. Recent work showing that textual gradients do not function as real gradients motivates treating automatic
prompt optimization (APO) as black-box search. We introduce SPO (
Stochastic Prompt Optimization), a framework for
stochastic search over
prompt space, and compare three strategies of increasing sophistication: error-informed random search, a genetic
algorithm with evolutionary operators, and SAGE (SPO via
Agent-Guided Exploration), a multi-
agent pipeline with diagnostic code execution. Across three
benchmarks, no single strategy dominates; effectiveness depends on the interaction of landscape structure with error type. We further deploy SAGE on a mental-health chatbot under a continuous optimization paradigm, where it compounds eight cycles of individually-noisy A/B tests into a statistically robust gain in next-day retention. We argue that coupling qualitative diagnosis with quantitative validation is what makes agentic optimization effective for open-ended task-oriented dialogue.
Read the full abstract ▾
All authors Ziyi Zhu, Luka Smyth, Saki Shinoda, Jinghong Chen
Computation and Language
Learning Robust Pair Confidence for Multimodal Emotion-Cause Pair Extraction
Zhuangzhuang Pan, Ning Dong, Yingna Su et al. (+1 more)
Multimodal emotion-cause pair extraction (MECPE) requires reliable pair confidence over candidate pairs. Existing pair scorers commonly use pair-level cross
entropy over valid candidates, which treats links mostly independently. This leaves the relative confidence geometry among competing causes under-constrained, allowing gold pairs to stay close to hard negatives or rely on incidental non-gold context. We study this vulnerability as pair-confidence brittleness and propose RPCL (Robust Pair Confidence Learning), a training-only framework for pair-confidence learning. RPCL encourages pair confidence to be both discriminative and stable: gold pairs are separated from row-wise hard negatives through a confidence-difference margin constraint, and clean pair predictions are aligned with predictions from a corrupted view where non-gold contextual utterance representations are partially corrupted. The original clean pair scorer and decoding pipeline are used unchanged at inference time. On ECF, MECAD, and MEC4, RPCL improves the three-seed mean Pair F1 over a matched base model by 2.58 to 2.83 percentage points in the full text-audio-
video setting, and improves mean Pair AUPRC on all three
datasets. Diagnostic analysis further shows larger gold-negative confidence gaps and lower margin-violation severity. These results suggest that explicitly shaping pair confidence is an effective training strategy for MECPE.
Read the full abstract ▾
All authors Zhuangzhuang Pan, Ning Dong, Yingna Su, Yan Xia
Computation and Language
Improving Medical Communication using Rubric-Guided Counterfactual Recommendations
Adrian Cosma, Nicoleta-Nina Basoc, Andrei Niculae et al. (+2 more)
Text-based telemedicine increasingly relies on lightweight patient feedback, however, such feedback primarily reflects perceived communication quality rather than medical accuracy. We introduce an LM-guided counterfactual recommendation pipeline that discovers and refines interpretable communication features such as tone, personalization, actionability and completeness in addressing patient concerns, without interfering with the medical content. These features are used together with patient-doctor interaction metadata to estimate positive feedback. At inference time, the system searches over low-cost ordinal feature changes and recommends minimal communication changes predicted to increase the probability of positive feedback, while independent auditor models test whether these gains generalize beyond the selection model. Across interactions, recommendations yield a mean +6.41% gain in predicted positive feedback probability under independent auditors, and are non-negative for 93.31% of recommendations. These results suggest that small, interpretable communication changes can capture most predicted gains while preserving the doctor's control over medical
reasoning and final wording.
Read the full abstract ▾
All authors Adrian Cosma, Nicoleta-Nina Basoc, Andrei Niculae, Cosmin Dumitrache, Emilian Radoi
Computation and Language
Efficient Financial Language Understanding via Distillation with Synthetic Data
Wen-Fong Huang and Edwin Simpson
Large instruction-following models are powerful but costly to deploy, particularly in finance, where labelled data are limited by confidentiality and expert annotation cost. We present an efficient framework for financial sentiment analysis through
distillation with synthetic data, transferring knowledge from a large instruction-tuned teacher to compact student models. The framework is designed for low-resource conditions, where a small set of real examples are collected and labelled by hand. The framework then clusters the examples and uses the clusters to select seeds for generating synthetic examples via structured few-shot prompting. Experiments show that clustering-based seed selection yields more representative synthetic data than random sampling, enabling compact models to achieve strong performance with minimal supervision. Notably, on a more complex and noisy text domain, the compact model trained on the complete synthetic-seed corpus even outperforms the teacher model, while remaining competitive on formal text. The framework provides a practical route toward resource-efficient domain adaptation in financial NLP with minimal human labelling effort.
Read the full abstract ▾
Computation and Language
Diffusion-MF: Approximate Structured Diffusion for Sequence Labelling
Nicolas Floquet, Joseph Le Roux and Nadi Tomeh
We introduce
Diffusion-MF, a discrete diffu- sion sequence labeller that places a linear-chain conditional random field (LCRF) inside the denoising loop. Unlike prior
diffusion labellers, it performs structured inference at every step; parallel Mean-Field makes this efficient. Across
multilingual POS, CoNLL-2003 NER, and joint Chinese
Segmentation and POS,
Diffusion-MF achieves the best primary result across all experimental settings.
Read the full abstract ▾
Computation and Language
Aligning Implied Statements for Implicit Hate Speech Generalizability with Context-Bounded Semi-hard Negative Mining
Wicaksono Leksono Muhamad and Yunita Sari
Classifying implicit hate
speech remains a challenge, as intent is often masked through insinuation and context rather than explicit slurs. Prior supervised contrastive approaches improve in-domain detection but can overfit surface cues and struggle to transfer across
datasets. We propose ImpSH, a triplet-based framework that aligns posts with implied statements when available and uses context-bounded semi-hard negatives to focus learning on near confusions. We also examine AugSH, which forms positives via data augmentation. In controlled evaluations on IHC, SBIC, and DynaHate with BERT and HateBERT, ImpSH is a viable alternative to standard supervised contrastive baselines and often improves cross-domain performance under matched preprocessing and tuning budgets. Representation analysis using
alignment and uniformity indicates tighter positive pairs with balanced global spread, and qualitative nearest-neighbor case studies illustrate typical false negatives under domain shift. These results demonstrate that aligning posts with their implied statements via context-bounded mining provides a more stable, bijective-like mapping to related insinuations, overcoming the
volatility inherent in traditional clustering-based representation learning.
Read the full abstract ▾
Computation and Language
ScholarSum: Student-Teacher Abstractive Summarization via Knowledge Graph Reasoning and Reflective Refinement
Bohou Zhang, Xiaoyu Tao, Mingyue Cheng et al. (+2 more)
Abstractive summarization plays a crucial role in enabling efficient understanding of scientific literature, yet it inherently demands both linguistic fluency and factual faithfulness. Existing approaches often fail to reconcile these two requirements. Extractive methods rely on rigid sentence splicing that disrupts macro-level logical coherence, while large
language model (LLM)-based generative approaches, despite mastering linguistic fluency, exhibit limited factual consistency. In this work, we propose ScholarSum, a hierarchical reflective graph-based framework that emulates a student-teacher writing process for fluent and faithful scientific summarization. ScholarSum first organizes the document into a hierarchical knowledge graph by segmenting it into semantically coherent units, whose multi-layered community structure captures global logic and macro-level themes. Guided by this global structure, the student generates an initial draft, which is subsequently refined through fine-grained evidence
retrieval. To ensure factual consistency, a teacher-like reviewer then iteratively examines the draft, identifies unsupported content, and
prompts targeted re-
retrieval and rewriting until the summary meets rigorous quality standards. Extensive experiments demonstrate that ScholarSum significantly outperforms previous baselines in terms of both completeness and faithfulness. Our code is available at https://github.com/Xiaoyu-Tao/ScholarSum.
Read the full abstract ▾
All authors Bohou Zhang, Xiaoyu Tao, Mingyue Cheng, Huijie Liu, Qi Liu
Computation and Language
Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
Xiaoyue Xu, Sikui Zhang, Xiaorong Wang et al. (+2 more)
Long-context
reasoning is an essential capability for large
language models, particularly when they are deployed as autonomous
agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context
reasoning. Our recipe targets three complementary task families –
retrieval, multi-evidence synthesis, and
reasoning – for which we construct and curate eight
datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context
benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an
agent-tuned model with our data recipe improves
GAIA by +4.8 and BrowseComp by +7.0 points. We will release our
datasets to facilitate future research.
Read the full abstract ▾
All authors Xiaoyue Xu, Sikui Zhang, Xiaorong Wang, Xu Han, Chaojun Xiao
Computation and Language · Cross-listed from cs.LG
GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents
Zhe Ren, Yibo Yang, Yimeng Chen et al. (+7 more)
Memory benchmarks for LLM
agents largely assume single-user settings, leaving shared assistants for hospitals, workplaces, campuses, and households understudied. In these deployments, multiple principals write to a common
memory pool and query it under different roles, scopes, and relationships, so
memory quality requires governance as well as recall. We introduce GateMem, a
benchmark for multi-principal shared-
memory agents. GateMem jointly evaluates utility for legitimate long-horizon requests with state updates, access control across contextual authorization boundaries, and
agent-facing active forgetting after explicit deletion requests. It spans medical, office, education, and household domains, with long-form multi-party episodes, incremental
memory injection, hidden checkpoints, structured judging, and leak-target annotations. Across diverse baselines and backbone models, no method simultaneously achieves strong utility, robust access control, and reliable forgetting. Long-context prompting often yields the best governance score at high
token cost, while
retrieval-based and external-
memory methods reduce cost yet still leak unauthorized or deleted information. These results show current
memory agents remain far from reliable shared institutional deployment.
Read the full abstract ▾
All authors Zhe Ren, Yibo Yang, Yimeng Chen, Zijun Zhao, Benshuo Fu, Zhihao Shu, Bingjie Zhang, Yangyang Xu, Dandan Guo, Shuicheng Yan
Computation and Language
Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports
Qingyu Lu, Ruochen Li, Liang Ding et al. (+3 more)
Reliable evaluation of generated radiology reports requires strict clinical accuracy, as omitted critical findings or mischaracterized radiographic observations can directly affect patient care. Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar. Although Large
Language Models (LLMs) possess rich medical knowledge, they likewise struggle to draw a reliable boundary between clinically significant errors and harmless variation. We study this boundary using ReEvalMed
benchmark as testbed and evaluate metric-level clinical significance from detecting true clinical errors ("Discrimination") and tolerating insignificant variations ("Robustness"). Across 8 LLM evaluators under one-pass and two-pass settings, we identify a widespread discrimination bias: models effectively detect errors but also over-penalize harmless rephrasings. To mitigate this, we synthesize 4k report pairs and train lightweight interpretable metrics on Qwen3-8B and MedGemma-4B. Our trained metric sharpens the clinical significance boundary, surpassing 32B-scale medical LLMs and remaining competitive with proprietary models. Crucially, the more costly two-pass setting fails to consistently improve overall performance and mainly trades discrimination for robustness. These findings suggest one-pass trained metrics as the practical choice for cost-sensitive deployment, with two-pass inference reserved for settings where D-R balance is critical. We will release the
dataset and metric.
Read the full abstract ▾
All authors Qingyu Lu, Ruochen Li, Liang Ding, Yufei Xia, Youxiang Zhu, Dacheng Tao
Computation and Language · Cross-listed from cs.CV
HandwritingAgent: Language-Driven Handwriting Synthesis in Scalable Vector Space
Jaward Sesay, Yue Yu and Börje F. Karlsson
Teaching machines to emulate natural handwriting styles remains an open challenge, as it requires synthesizing stroke sequences that dynamically vary in shape, texture, pressure and script - not only across individuals, but also within a single person's handwriting. Attempts at this challenge have largely explored deep learning methods in both online and offline settings. However, these approaches are often constrained by style-specific architectural choices, heavy reliance on large
datasets, high compute costs, and a lack of flexible control over writing styles through natural
language. To this end, we introduce HandwritingAgent, a
language-driven
agent that can synthesize natural handwriting sequences directly in Scalable Vector Graphics (SVG) format with no need for style-specific training. The
agent leverages a large
reasoning model to geometrically analyse and autoregressively generate target handwritten glyphs as stroke sequences in a discrete grid canvas environment. Generation is conditioned on texts provided in either conversational or non-conversational mode, along with a reference handwriting-style image. Experiments on diverse handwriting tasks spanning imitation, recognition, multi-lingual handwriting synthesis, and generation of complex handwritten maths and science expressions indicate substantial improvement in performance, with HandwritingAgent matching or surpassing state-of-the-art generative handwriting models, while providing a more efficient, controllable, and generalizable synthesis method.
Read the full abstract ▾
Computation and Language
RedactionBench
Sean Brynjólfsson, Shashvat Jayakrishnan, Esha Sali et al. (+2 more)
Large
Language Models are increasingly applied to sensitive domains that require redaction of personally identifiable information (PII). While redacting PII is a data cleaning prerequisite, existing
benchmarks conflate extraction mechanics with
privacy semantics. A public phone number is not equivalent to a phone number in a medical record. Whether information constitutes a violation depends heavily on who holds it, why, and in what context, fundamentally differentiating redaction from simple entity recognition. Grounded in contextual integrity, we introduce RedactionBench, a manually annotated
benchmark comprising 200 diverse documents across 11 domains, mostly seeded from real-world sources. We also introduce R-Score, a novel character-level metric that treats semantically similar redactions equally and nullifies shallow formatting choices, such as varying masking styles for phone numbers. Evaluations across Named Entity Recognition models, entity extraction Small
Language Models, and frontier models equipped with agentic tools demonstrate that contextual redaction remains an unsolved problem. A human evaluation with over 80 users on RedactionBench reveals a stark dichotomy in
privacy perceptions. Annotators show consensus with target labels for mandatory redactions (89.4 percent) and safe text preservations (94.1 percent), but fail to agree on contextual redactions (47.7 percent). This variance demonstrates the subjective nature of contextual
privacy and motivates R-Score, which decouples contextual ambiguity from strict precision. We compare 35 models across families and report their performance in redacting PII. Finally, we release RedactionBench to establish a baseline for future
privacy-preserving systems, hoping to inspire efficient model design and standardized evaluations.
Read the full abstract ▾
All authors Sean Brynjólfsson, Shashvat Jayakrishnan, Esha Sali, Diptanshu Purwar, Madhav Aggarwal
Computation and Language
Lost in a Single Vector: Improving Long-Document Retrieval with Chunk Evidence Aggregation
Shanshan Lyu, Yiwei Wang, Yujun Cai et al. (+2 more)
Dense
retrieval ranks one query vector against one document vector. On long documents, this interface can fail when a short but decisive span is weakened during document encoding before ranking. We study this failure mode as document-side early compression and introduce the Evidence Dilution Index (EDI) to measure how far a document-level representation falls below the strongest chunk-level evidence within the same gold document. Guided by this view, we propose DICE (Document Inference via Chunk Evidence), a training-free document-side strategy that splits documents into chunks, encodes them independently with a frozen model, and aggregates them back into a single vector while preserving the standard one-query-one-document interface. On LongEmbed, DICE improves
retrieval across four backbones, with the largest gains on slices beyond 4k
tokens: for Dream, Passkey >4k rises from 30.0 to 90.0 and Needle >4k from 23.3 to 74.0. Across 12,779 filtered samples, DICE yields lower EDI than the single-vector baseline in 92.8% of cases. These results establish document-level encoding as a practical and underexplored lever for long-document
retrieval.
Read the full abstract ▾
All authors Shanshan Lyu, Yiwei Wang, Yujun Cai, Jiafeng Guo, Shenghua Liu
Computation and Language · Cross-listed from cs.CV
SAMA: Semantic Anchor-aligned Augmentation for Unified Low-Resource Multimodal Information Extraction
Quanjiang Guo, Chong Mu, Jiazhou Pan et al. (+4 more)
Multimodal Information Extraction (MIE)-covering tasks such as
Multimodal Named Entity Recognition (MNER), Relation Extraction (MRE), and Event Extraction (MEE)-is essential for understanding multimedia content but remains constrained by severe data scarcity. Although data augmentation is a promising remedy, existing approaches are impeded by coarse cross-modal
alignment and fragmented, task-specific designs that fail to exploit shared semantic knowledge. To overcome these limitations, we introduce Semantic Anchor-aligned
Multimodal Augmentation (SAMA), a unified framework for generating high-fidelity, task-aware synthetic data. SAMA constructs structured semantic anchors from ground-truth labels to guide a Collaborative Multi-Experts
Multimodal Large
Language Model (CME-MLLM), which integrates a Universal Adapter for shared semantics with Task-Specific Adapters to produce diverse yet constraint-compliant textual samples. For image synthesis, SAMA employs an Anchor-Preserving
Diffusion mechanism that uses anchor-weighted
prompts and latent conditioning to maintain critical semantic anchors while diversifying visual contexts. To eliminate the need for manual verification, SAMA further introduces a Dual-Constraint Filtering module that selects synthetic samples based on both cross-modal consistency and anchor fidelity. Extensive experiments across
benchmark datasets for MNER, MRE, and MEE demonstrate that SAMA consistently outperforms state-of-the-art augmentation baselines under both fully supervised and low-resource settings, underscoring its versatility, robustness, and effectiveness.
Read the full abstract ▾
All authors Quanjiang Guo, Chong Mu, Jiazhou Pan, Ming Jia, Ling Tian, Hui Gao, Zhao Kang
Computation and Language
Output Vector Editing for Memorization Mitigation in Large Language Models
Ahmad Dawar Hakimi, Kaiwei Lei, Isabelle Augenstein et al. (+1 more)
Large
language models can reproduce training text, creating
privacy, copyright, and security risks. We propose Output Vector Editing (OVE), a post-hoc method for suppressing text continuations that a model reproduces from a given prefix. Under superposition, a
neuron can contribute to multiple features. Yet some
neuron-level mitigation methods suppress localized
neurons by zeroing their activations, removing each
neuron's entire contribution. OVE instead edits their output vectors, combining (i) two-stage localization of MLP layers and
neurons with (ii) full-vocabulary least-squares updates. Its multi-anchor variant, OVE-MA, adaptively selects continuation positions expected to provide stronger suppression relative to the required weight change. We evaluate recovery and locality across four models (360M–7B parameters), using autoregressively validated continuations and recovery tests with greedy decoding, beam search, and sampling. On 634 OLMo-7B pairs, OVE-MA with at most five output-vector updates per request leaves 24.45% recoverable in these tests, compared with 32.02% for the closest prior post-hoc editing method, REVS, while using approximately 75 times fewer update operations. These results show that adaptive position selection can concentrate effective suppression into a small number of weight updates. OVE edits weights without gradients and requires no additional intervention during inference.
Read the full abstract ▾
All authors Ahmad Dawar Hakimi, Kaiwei Lei, Isabelle Augenstein, Hinrich Schütze
Computation and Language
LegalWorld: A Life-Cycle Interactive Environment for Legal Agents
Songhan Zuo, Shengbin Yue, Tao Chiang et al. (+4 more)
Civil litigation is inherently a life-cycle process: what a lawyer drafts on day one constrains what unfolds at trial months later. Yet existing legal
benchmarks evaluate isolated subtasks, and prior legal-
agent simulators reinitialize each scenario from shared ground truth, leaving cross-stage causal dependencies unmodeled. We present LegalWorld, a life-cycle interactive environment that models Chinese civil litigation as a causally connected state chain of five stages (seven sub-scenarios), grounded in 75,309 paired Chinese civil judgments. We pair it with reusable infrastructure (local
memory, global case
memory, a Skill/Tool library) that keeps each dispute consistent across its full life cycle. Building on this environment, we construct LongJud-Bench to evaluate
agent capability across all five connected stages. 18,992 ratings from 217 legal-background evaluators confirm that LegalWorld trajectories are procedurally faithful and role-consistent; and a capability-level cross-model evaluation reveals sharp divergences that aggregate scores cannot expose, with no single backbone leading across consultation, drafting, and courtroom advocacy. Detailed resources will be released publicly.
Read the full abstract ▾
All authors Songhan Zuo, Shengbin Yue, Tao Chiang, Guanying Li, Yun Song, Xuanjing Huang, Zhongyu Wei
Computation and Language
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
Tolga Şakar
Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern
language models split words by corpus statistics, fragmenting semantically loaded suffixes and – in the case of WordPiece and rule-based analyzers – failing to decode their output back to the original text. This paper presents Morpheus, a neural morpheme-boundary model for Turkish that is at once a lossless, morphology-aware tokenizer and a word-
embedding producer. A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships during training and exact segments at inference, with no string normalization, so $\mathrm{decode}(\mathrm{encode}(w)) = w$ holds by construction. Because the model is neural, the same forward pass that tokenizes also emits a structured word
embedding. Among reversible tokenizers – the only ones valid for generation – Morpheus attains the lowest bits-per-character ($1.425$), roughly doubles the gold morphological
alignment of the subword family (MorphScore macro-F1 $0.61$ vs.\ ${\sim}0.32$), and uses ${\sim}19\%$ less GPU
memory than 64K-vocabulary subword tokenizers. As an embedder, frozen Morpheus vectors lead on lexical
retrieval (root-family MAP $0.85$) and same-root verification (ROC-AUC $1.00$), surpassing the
multilingual retriever BGE-M3 and BERTurk; on context- and inflection-dependent tasks (NER, case/number probing) the heavier contextual encoders remain ahead – a trade-off we attribute to Morpheus's root-centric geometry. Code: https://github.com/lonewolf-rd/TurkishMorpheus; model: https://huggingface.co/lonewolflab/Morpheus-TR-50K; interactive demo: https://huggingface.co/spaces/lonewolflab/morpheus-tr-demo.
Read the full abstract ▾
Computation and Language
LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
Han Chen, Ming Li, Chenguang Wang et al. (+4 more)
Existing work on LLM-based educational assessment has focused largely on item difficulty, but difficulty alone does not indicate whether an item meaningfully distinguishes higher- from lower-proficiency students. Item discrimination captures this complementary and fundamental psychometric property. We investigate whether LLMs can predict human item discrimination from assessment content. We evaluate 42 proprietary and open-weight LLMs using two complementary approaches. Direct discrimination prediction asks models to explicitly predict an item's discrimination value, while response-based proxy estimation treats LLM answers as synthetic responses and applies a Classical Test Theory (CTT)-inspired item-rest calculation. Direct predictions show weak
alignment with human item discrimination. The response-based proxy provides a stronger but still limited ranking signal, reaching a CEFR-stratified rank correlation of 0.231. Further analysis shows that this correlation comes mainly from differences across models rather than proficiency
prompts that reliably simulate students at different ability levels. Current LLMs therefore contain some discrimination-relevant information, but they do not yet reliably model the ability-conditioned human response behavior that gives item discrimination its psychometric meaning.
Read the full abstract ▾
All authors Han Chen, Ming Li, Chenguang Wang, Yijun Liang, Dawei Zhou, Hong jiao, Tianyi Zhou
Computation and Language
TW-LegalBench: Measuring Taiwanese Legal Understanding
Fei-Yueh Chen, Chun Huang Lin, Chan Wei Hsu et al. (+4 more)
Large
language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal
reasoning remains underexplored. We present TW-LegalBench that utilizes Taiwanese legal system's rich official corpus open to the public to fill the gap in evaluating LLMs on Taiwanese law, among common-law
benchmarks that focus on English sources and civil-law
benchmarks focusing on sources of Simplified Chinese. TW-LegalBench comprises three task types: (1) over 16,000 multiple-choice questions (MCQs) across five years of official examinations in 18 professional domains; (2) 117 open-ended essay questions (OEQs) from examinations for legal professionals with official scoring rubrics; and (3) more than 14,000 legal judgment prediction (LJP) instances covering hundreds of crime categories. We evaluate 13 LLMs using accuracy for MCQs, a decomposed LLM-as-Judge framework based on the scoring rubric points for OEQs, and metrics for sentencing accuracy and statute citation for LJP. Our results reveal that top-performing models exceed the passing threshold for qualified lawyers (passing rate: 11%) but fall short of that for judges and prosecutors (passing rate: 1~2%). For LJP, while models demonstrate reasonable verdict type accuracy and sentence prediction capability, they struggle to cite exact legal articles. These findings highlight that reliable legal text generation remains challenging for LLMs, even though their performance on qualification examinations approaches human level.
Read the full abstract ▾
All authors Fei-Yueh Chen, Chun Huang Lin, Chan Wei Hsu, Kuan Hsuan Yeh, Zih-Ching Chen, Kuan-Ming Chen, Patrick Chung-Chia Huang
Computation and Language · Cross-listed from cs.LG
Attention as Frustrated Synchronization
Joshua Nunley
A network of oscillators that synchronizes perfectly computes nothing further, so an
attention architecture built from synchronization must locate its computation in structured departures from agreement. We introduce the Frustrated Synchronization Network (FSN), whose
token states are phases on a
torus and whose entire value pathway is one learned complex coupling kernel over harmonics and a one-step delay. Each component of the kernel is a frustration in the sense of the synchronization literature. The complex phases are static Kuramoto-Sakaguchi frustration angles, the signed harmonics are repulsive Daido components, and the delay term, which couples each
token to the successors of the
tokens it attends to, is algebraically identical to Kuramoto-Sakaguchi coupling whose frustration angle is the data's own transition, so next-
token prediction is implemented as synchronization frustrated by the data. At matched one-million-parameter and training budgets on character-level text and code, the FSN's validation loss is below a tuned RoPE-SwiGLU
transformer's at every epoch measured, and the comparison survives training the baseline to convergence: every thirty-epoch enwik8 seed finishes below the
transformer's converged fifty-epoch loss of 1.611, and the FSN's completed fifty-epoch runs converge to 1.5953 +/- 0.0014. A variant with every feed-forward block replaced by mean-field coupling to learned collective modes, leaving no multilayer perceptron in the stack, tracks the
transformer. On natural text the unfrustrated base layer falls behind the converged
transformer at every copy depth, worst on long-range copy events; the kernel reverses the deficit at every depth of four and beyond. Headline comparisons are at the one-million-parameter scale; a scale ladder is complete through four million parameters with the advantage persisting, and remaining arms are marked as in progress.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Jaeho Lee, Nick Merrill and Ezra Karger
Forecasting benchmarks for general-purpose AI systems usually inherit the constraints of the real world: outcomes resolve slowly, tail events are rare, and counterfactual questions are difficult to score. We introduce ForecastBench-Sim, a simulated-world
forecasting benchmark built on game rollouts from Freeciv, a turn-based strategy game modelled on the Civilization series. Forecasters receive a fixed world report (a structured snapshot of the current game state) and answer questions about hidden future states; the
benchmark then continues the simulation and scores forecasts. Because the world is simulated, the same setup can generate continuous or binary
forecasting questions at arbitrary time horizons, paired intervention worlds for conditional or causal questions, and resolved examples of rare or disruptive outcomes. We describe the
benchmark pipeline, question families, scoring protocol, and release artifacts, and report validation slices from model evaluations and an anonymized human pilot. ForecastBench-Sim is intended to complement real-world
forecasting benchmarks by providing controlled, immediately resolvable tasks for studying probabilistic
reasoning under dynamic world states.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.MA
EARS: Explanatory Abstention for Reliable Sub-Agent Modeling in Large-scale Multi-Agent Systems
Shuang Xie, Yunan Lu, Han Li et al. (+1 more)
In large-scale enterprise settings, centralized multi-
agent systems (MAS) are increasingly adopted, in which a coordinator delegates user requests to lightweight, domain-specialized sub-
agents. While this architecture improves modularity, scalability, and cost efficiency, its reliability depends not only on accurate routing but also on sub-
agents' ability to calibrate their responses to capability constraints. In particular, sub-
agents built on smaller fine-tuned models often struggle with such calibration, leading them to over-answer ambiguous, underspecified, misrouted, or unsupported requests and produce hallucinated outputs instead of actionable feedback. To address this challenge, we present EARS (Explanatory Abstention for Reliable Sub-
Agent Modeling), a production-oriented framework that reframes sub-
agent abstention as an inter-
agent communication protocol: a sub-
agent does not merely abstain, but exposes an actionable failure state to the coordinator. EARS curates human-
agent interaction data using an ensemble of calibrated LLM-as-a-Judge models, producing structured abstention labels and rationales under a taxonomy of sub-
agent failure modes. These data are used to fine-tune sub-
agents to detect failure conditions and return rationales for coordinator-level clarification, rerouting, or fallback. We evaluate EARS in a large-scale production e-commerce assistant supporting enterprise business intelligence workflows. EARS improves the overall response pass rate from 68.5% to 78.9%, demonstrating that sub-
agent-side explanatory abstention improves MAS reliability.
Read the full abstract ▾
All authors Shuang Xie, Yunan Lu, Han Li, Lingyun Wang
Computation and Language
RegMix-D: Dynamic Data Mixing via Proxy Training Trajectories
Kaiyan Zhao, Zhongtao Miao, Akiko Aizawa et al. (+1 more)
Data mixture selection is critical for Large
Language Model pretraining. Existing methods such as RegMix select a single static mixture by fitting a regression model on small-scale proxy runs. We propose RegMix-D, a simple extension of RegMix to dynamic mixing. Our key observation is that proxy runs produce not only endpoint losses, but also full loss trajectories, which can be used to further improve data mixture. By training regression model on these trajectories, we can predict optimal mixtures at multiple training stages. RegMix-D supports two deployment modes: an offline variant that generates a complete mixture schedule before target training, and an online variant that adapts the mixture during training using observed loss. Experiments on 25B
tokens of the Pile
dataset with a 1B parameter target model show that RegMix-D consistently improves over RegMix and DoReMi across 13 downstream tasks while remaining proxy-efficient: it surpasses RegMix even with only 128 proxy models (25% of RegMix's proxy compute budget).
Read the full abstract ▾
All authors Kaiyan Zhao, Zhongtao Miao, Akiko Aizawa, Yoshimasa Tsuruoka
Computation and Language
The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs
Naihao Deng, Yiming Feng, Chimaobi Okite et al. (+4 more)
Warning: This paper studies stereotypes and biases, and contains potentially disturbing examples, used for illustration purposes only. Our findings should not be interpreted as an argument against
alignment. Instead, this paper highlights the need for principled approaches to more advanced
alignment.
Alignment aims to ensure that large
language models (LLMs) behave safely and reliably, including by avoiding unsafe inferences. However, we show that such safety-oriented behaviors can misfire: models may reject warranted conclusions even when they are explicitly supported by context. We call this failure mode misfired
alignment, where
alignment-induced changes cause LLMs to override explicit evidence. To quantify this phenomenon, specifically on stereotype-related
alignment, we introduce VETO, a
benchmark consisting of 2,032 BBQ-derived contrastive pairs, and define a new metric, Misfired
Alignment Rate (MAR), which measures on a 0 to 100 scale how often a model fails on a stereotype-related question but succeeds on its contrastive counterpart. We
benchmark 25 LLMs on VETO, and show that all LLMs, including the most recent ones, exhibit non-trivial (4.7 to 18.9%) MARs while all human participants achieve 0.0% MAR. Controlled priming experiments further show that
alignment-induced cues can substantially amplify MAR across LLMs, indicating that these failures are not merely artifacts of individual examples but can be induced by safety-related framing. Mechanistic analyses on open-weight LLMs reveal late-layer suppression of evidence-supported answers, and comparisons between instruct and base LLMs suggest that this suppression emerges after instruction training. These findings show that current
alignment methods can overgeneralize surface-level safety cues, to the point of overriding objective evidence, motivating more work on
alignment objectives that better preserve contextual grounding.
Read the full abstract ▾
All authors Naihao Deng, Yiming Feng, Chimaobi Okite, Kaijian Zou, Lu Wang, Rada Mihalcea, Yulong Chen
Computation and Language · Cross-listed from cs.MA
Gender Bias in LLM Hiring Decisions: Evidence from a Japanese Context and Evaluation of Mitigation Strategies
Serena A. Hoffstedde, Machiko Hirota, Akshara Nadayanur Sathis Kanna et al. (+4 more)
Large
language models (LLMs) are increasingly deployed in hiring workflows, yet most research on gender bias in LLM hiring decisions has focused on English-
language, Western-format resumes. This study examines whether pro-female gender bias extends to a Japanese corporate context and evaluates two practical mitigation strategies. Using a counterfactual resume design with 60 Japanese rirekisho-format resumes, 12 name pairs selected on linguistically grounded gender-signal criteria, and five state-of-the-art LLMs (Claude Sonnet 4.6, GPT-4o, DeepSeek-V3, Gemini 2.5 Flash, Llama 3.3 70B), we conducted 43,200 API calls across baseline,
prompt instruction, and
privacy filter conditions. A crossed random-effects linear mixed model confirms a significant pro-female bias across all five models, replicating Western findings in a non-Western context. A
prompt-level gender-neutrality instruction produces no meaningful reduction in bias. A name-reliance analysis formally identifies the candidate name as the primary gender channel: removing the name from the
prompt reduces the female effect by nearly its full magnitude. An unexpected incompatibility between the
privacy filter and GPT-4o's content safety filter, resulting in a 42% refusal rate, highlights a practical deployment challenge for name anonymization in LLM-assisted recruitment pipelines.
Read the full abstract ▾
All authors Serena A. Hoffstedde, Machiko Hirota, Akshara Nadayanur Sathis Kanna, Rihito Kotani, Ujwal Kumar, Gabriele Trovato, Phan Xuan Tan
Computation and Language
PEC-Home: Interpretation of Progressively Elliptical Commands in Smart Homes
Yingyu Shan, Zeming Liu, Silin Li et al. (+4 more)
Recent advancements in Large
Language Models (LLMs) have empowered home assistants with natural
language interaction capabilities. However, current assistants overlook the progressive omission that occurs in human dialogue as shared context accumulates, leading to more elliptical expressions for efficient communication. Thus, current assistants still struggle to interpret such elliptical expressions accurately, which limits their effectiveness in real-world applications. In practical smart home scenarios, assistants face two major challenges caused by elliptical commands: (1) referential ambiguity caused by different environmental expectations among multiple users; and (2) intention ambiguity resulting from user preferences that evolve over time or change with the environment. To address these challenges, we introduce PEC-Home, the first simulated home
dataset specifically designed for interpreting progressively elliptical commands in smart homes. Extensive experiments on various LLMs, including GPT-4o, show that existing home assistants struggle to execute user-intended operations based solely on elliptical commands. Even when equipped with tools for storing and retrieving user dialogue history, execution accuracy remains below that achieved with complete commands.}.
Read the full abstract ▾
All authors Yingyu Shan, Zeming Liu, Silin Li, Boao Qian, Jiashu Yao, Yuhang Guo, Haifeng Wang
Computation and Language
PragReST: Self-Reinforcing Counterfactual Reasoning for Pragmatic Language Understanding
Jihyung Park, Minchao Huang, Leqi Liu et al. (+1 more)
Natural
language understanding often depends on meanings that are implied rather than explicitly stated, requiring pragmatic
reasoning. Despite strong performance on math and logical
reasoning, large
language models (LLMs) still struggle with making pragmatic inferences, often choosing literal interpretations. To improve LLM pragmatic
reasoning, we introduce PragReST, a self-supervised framework that constructs pragmatic QA data, generates counterfactual
reasoning traces, and trains models to internalize them through supervised fine-tuning and reinforcement learning, without human-labeled training data or
distillation from a stronger teacher. Across four pragmatic
benchmarks (PragMega, Ludwig, MetoQA, and AltPrag), PragReST improves over backbone models, task-specific pragmatic tuning baselines, and non-counterfactual variants of the same pipeline. On accuracy-based
benchmarks, PragReST improves over the instruct backbone by 5.37 and 5.50% (absolute) for Qwen3-8B and Qwen3-14B, respectively. Our error analysis and ablations underscore the importance of counterfactual
reasoning: PragReST primarily reduces errors caused by failures to contrast observed utterances with plausible alternatives, and removing counterfactual
reasoning substantially reduces performance. Moreover, our training preserves out-of-domain performance on general-knowledge and mathematical
reasoning benchmarks.
Read the full abstract ▾
All authors Jihyung Park, Minchao Huang, Leqi Liu, Elias Stengel-Eskin