Computation and Language
One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinement
Yixiao Zhou, Dongzhou Cheng, zhiliang wu et al. (+3 more)
Large
Language Models (LLMs) often fail to utilize their latent
reasoning capabilities due to a distributional mismatch between ambiguous human inquiries and the structured logic required for machine activation. Existing
alignment methods either incur prohibitive $O(N)$ costs by fine-tuning each model individually or rely on static
prompts that fail to resolve query-level structural complexity. In this paper, we propose ReQueR (Reinforcement Query Refinement), a modular framework that treats
reasoning elicitation as an inference-time
alignment task. We train a specialized Refiner policy via Reinforcement Learning to rewrite raw queries into explicit logical decompositions, treating frozen LLMs as the environment. Rooted in the classical Zone of Proximal Development from educational psychology, we introduce the Adaptive Solver Hierarchy, a curriculum mechanism that stabilizes training by dynamically aligning environmental difficulty with the Refiner's evolving competence. ReQueR yields consistent absolute gains of 1.7%–7.2% across diverse architectures and
benchmarks, outperforming strong baselines by 2.1% on average. Crucially, it provides a promising paradigm for one-to-many inference-time
reasoning elicitation, enabling a single Refiner trained on a small set of models to effectively unlock
reasoning in diverse unseen models. Code is available at https://github.com/newera-xiao/ReQueR.
Read the full abstract ▾
All authors Yixiao Zhou, Dongzhou Cheng, zhiliang wu, Yi Yang, Yu Cheng, Hehe Fan
Computation and Language · Cross-listed from cs.SD
Praxy Voice: Voice-Prompt Recovery + BUPS for Commercial-Class Indic TTS from a Frozen Non-Indic Base at Zero Commercial-Training-Data Cost
Venkata Pushpak Teja Menta
Commercial TTS systems produce near-native Indic audio, but the best open-source bases (Chatterbox, Indic Parler-TTS, IndicF5) trail them on measured phonological dimensions, and the most widely adopted
multilingual base (Chatterbox, 23
languages) does not even tokenise Telugu or Tamil. We ask: what is the minimum intervention that brings such a non-Indic-native base to commercial-class output on Telugu, Tamil, and Hindi, without training a new acoustic decoder and without any commercial TTS training data? We combine three pieces: (1) BUPS, a Brahmic Unified Phoneme Space that deterministically romanises seven Indic scripts to ISO-15919 so Chatterbox's Latin tokeniser can process them; (2) a LoRA adapter on only the text-
token predictor (Chatterbox's t3), trained on ~1,220h of licensed Indic audio with a Hindi-proxy
language_id; (3) a voice-
prompt recovery recipe – an 8-11s same-
language reference clip plus three sampling overrides (exaggeration 0.7, temperature 0.6, min_p 0.1; "Config B") – that recovers commercial-class acoustic output with no acoustic-decoder training. On Hindi, the LoRA regresses accuracy and we instead use vanilla Chatterbox + Config B, giving a two-branch deployment. Evaluated on 10-utterance pilot sets with the companion PSP
benchmark, Praxy Voice matches or slightly leads commercial baselines: 26.7% retroflex collapse on Telugu (vs Sarvam Bulbul 33.3%), 71% Tamil-zha collapse (vs commercial trio's 86%), 0.025 LLM-WER on Hindi (tied with Cartesia Sonic-3). For intra-sentential code-mix we add a third branch (IndicF5 + native-script transliteration) that drops code-mix LLM-WER from 0.80-0.85 to 0.14-0.27 across Hi/Te/Ta. We release R6 LoRA weights (Apache-2.0), inference code and router (MIT), and a Gradio demo.
Read the full abstract ▾
Computation and Language
Do LLMs Capture Embodied Cognition and Cultural Variation? Cross-Linguistic Evidence from Demonstratives
Yu Wang, Emmanuele Chersoni and Chu-Ren Huang
Do large
language models (LLMs) truly acquire
embodied cognition and cultural conventions from text? We introduce demonstratives, fundamental spatial expressions like "this/that" in English and "zhè/nà" in Chinese, as a novel probe for grounded knowledge. Using 6,400 responses from 320 native speakers, we establish a human baseline: English speakers reliably distinguish proximal-distal referents but struggle with perspective-taking, while Chinese speakers switch perspectives fluently but tolerate distal ambiguity. In contrast, five state-of-the-art LLMs fail to inherently understand the proximal-distal contrast and show no cultural differences, defaulting to English-centric
reasoning. Our study contributes (i) a new task, based on demonstratives, as a new lens for evaluating
embodied cognition and cultural conventions; (ii) empirical evidence of cross-cultural asymmetries in human interpretation; (iii) a new perspective on the
egocentric-sociocentric debate, showing both orientations coexist but vary across
languages; and (iv) a call to address individual variation in future model design.
Read the full abstract ▾
Computation and Language
Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer
Penghao Kuang, Haoyi Wu and Kewei Tu
Probabilistic
Transformer (PT), a white-box probabilistic model for contextual word representation, has demonstrated substantial similarity to standard
Transformers in both computational structure and downstream task performance on small models and small to medium sized
datasets. However, PT is less robust to hyperparameter choices than standard
Transformers, making it harder to scale efficiently. In this work, we follow Maximal Update Parametrization (muP) to rescale PT's parameters, so that hyperparameters optimized on small models can be transferred to larger models without additional tuning. With this approach, we successfully scale PT to models with up to 0.4B parameters. Experiments show that PT consistently outperforms standard
transformer under the same parameter budget on Masked
Language Modeling (MLM) tasks. We hope this work will contribute to the practical deployment of probabilistic models at substantially larger scales in the future.
Read the full abstract ▾
Computation and Language
Benchmarking PyCaret AutoML Against IndoBERT Fine-Tuning for Sentiment Analysis on Indonesian IKN Twitter Data
Mutia Alfi Mayzaroh, Dwi Fitria Ningsih, Nindi Destriani et al. (+1 more)
This paper
benchmarks a classical machine learning approach based on PyCaret AutoML against a deep learning approach based on IndoBERT fine-tuning for binary sentiment analysis of Indonesian-
language Twitter comments related to Ibu Kota Nusantara (IKN). The
dataset contains 1,472 manually labeled samples, consisting of 780 negative and 692 positive comments. In the machine learning setting, Logistic Regression, Naive Bayes, and Support Vector Machine were evaluated using 10-fold cross-validation, with Logistic Regression achieving the best performance among the classical models at 77.57% accuracy and 77.17% F1-score. In the deep learning setting, the indobenchmark/indobert-base-p1 model was fine-tuned for five epochs and achieved 89.59% test accuracy and 89.37% F1-score. The results show that IndoBERT substantially outperforms the machine learning baselines, highlighting the effectiveness of
Transformer-based contextual representations for informal Indonesian social media text.
Read the full abstract ▾
All authors Mutia Alfi Mayzaroh, Dwi Fitria Ningsih, Nindi Destriani, Martin C.T. Manullang
Computation and Language
Wiki Dumps to Training Corpora: South Slavic Case
Mihailo Škorić and Cosimo Palma
This paper presents a pipeline designed to transform raw Wikimedia dumps into quality textual corpora for seven South Slavic
languages. The work is divided into two major phases. The first involves extracting and cleaning text from raw dumps of Wikipedia, Wikisource, Wikibooks, Wikinews, and Wikiquote. This step requires careful handling of raw wiki markup to isolate, first of all, textual articles, and then usable natural
language text within them. The second phase addresses the challenge of questionable or low-quality articles, which are often generated from databases or structured knowledge bases. These articles are characterised by repetitive patterns, generic phrasing, and minimal to no original content. To mitigate their impact, a n-gram-based filtering strategy was employed to detect high levels of textual redundancy between articles and then remove such articles from the corpora entirely. The resulting
datasets aim to provide linguistically rich texts suitable for training
language models or conducting comparative research across South Slavic
languages. By combining systematic extraction with quality control, this work contributes to the creation of reliable, high-information corpora that reflect the authentic cultural contexts of
languages. While focused on the South Slavic case in the paper, the approach is mostly
language-agnostic and can be generalised to other
languages.
Read the full abstract ▾
Computation and Language
Language corpora for the Dutch medical domain
B. van Es
Background: Dutch medical corpora are scarce, limiting NLP development. Methods: We translated English
datasets, identified medical text in generic corpora, and extracted open Dutch medical resources. Results: The resulting corpus comprises +- 36 billion
tokens across the medical domain in about 105 million documents, freely available on Hugging Face. Conclusion: This work establishes the first large-scale Dutch medical
language corpus for pre-training and downstream NLP tasks.
Read the full abstract ▾
Computation and Language
The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models
Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D Khemlani et al. (+1 more)
Large
Language Models are increasingly being deployed to extract structured data from unstructured and semi-structured sources: parsing invoices, medical records, and converting PDF documents to database entries. Yet existing
benchmarks for structured output generation either focus on schema compliance alone, or evaluate value correctness within a single source domain. We introduce SOB (The Structured Output
Benchmark), a multi-source
benchmark spanning three source modalities: native text, images, and audio conversations. All models receive a text-normalized representation of their context regardless of source modality; this deliberate design isolates structured-output capability from raw
vision or
speech-processing quality, ensuring a fair, source-agnostic comparison. Our
benchmark comprises 5,000 text evaluation records derived from multi-hop QA drawn from a 25,091-record full corpus, 209 image records from OCR-processed PDFs across seven document types including multi-column layouts, dense tables, scanned historical documents, small-print text, and mathematical typesetting, and 115 audio records from the AMI corpus. Each record pairs a natural-
language question with a JSON schema that the model must follow and a ground-truth answer verified against the source context. We evaluate 21 frontier and open-weight models across three source domains and seven metrics. Our results reveal a consistent pattern: models achieve near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text, 67.2% on images, and 23.7% on audio, where longer context makes extraction substantially harder. We release the
dataset, evaluation pipeline, and all related code.
Read the full abstract ▾
All authors Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D Khemlani, Vineet Agarwal
Computation and Language · Cross-listed from cs.SE
R$^3$-SQL: Ranking Reward and Resampling for Text-to-SQL
Hojae Han, Yeonseok Jeong, Seung-won Hwang et al. (+2 more)
Modern Text-to-SQL systems generate multiple candidate SQL queries and rank them to judge a final prediction. However, existing methods face two limitations. First, they often score functionally equivalent SQL queries inconsistently despite identical execution results. Second, ranking cannot recover when the correct SQL is absent from the candidate pool. We propose R$^3$-SQL, a Text-to-SQL framework that addresses both issues through unified reward for ranking and resampling. R$^3$-SQL first groups candidates by execution result and ranks groups for consistency. To score each group, it combines a pairwise preference across groups with a pointwise utility from the best group rank and size, capturing relative preference, consistency, and candidate quality. To improve candidate recall, R$^3$-SQL introduces agentic resampling, which judges the generated candidate pool and selectively resamples when the correct SQL is likely absent. R$^3$-SQL achieves 75.03 execution accuracy on BIRD-dev, a new state of the art among methods using models with disclosed sizes, with consistent gains across five
benchmarks.
Read the full abstract ▾
All authors Hojae Han, Yeonseok Jeong, Seung-won Hwang, Zhewei Yao, Yuxiong He
Computation and Language · Cross-listed from cs.GR
Cutscene Agent: An LLM Agent Framework for Automated 3D Cutscene Generation
Lanshan He, Haozhou Pang, Qi Gan et al. (+12 more)
Cutscenes are carefully choreographed cinematic sequences embedded in
video games and interactive media, serving as the primary vehicle for narrative delivery, character development, and emotional engagement. Producing cutscenes is inherently complex: it demands seamless coordination across screenwriting, cinematography, character animation, voice acting, and technical direction, often requiring days to weeks of collaborative effort from multidisciplinary teams to produce minutes of polished content. In this work, we present Cutscene
Agent, an LLM
agent framework for automated end-to-end cutscene generation. The framework makes three contributions: (1)~a Cutscene Toolkit built on the Model Context Protocol (MCP) that establishes bidirectional integration between LLM
agents and the game engine –
agents not only invoke engine operations but continuously observe real-time scene state, enabling closed-loop generation of editable engine-native cinematic assets; (2)~a multi-
agent system where a director
agent orchestrates specialist subagents for animation, cinematography, and sound design, augmented by a visual
reasoning feedback loop for perception-driven refinement; and (3)~CutsceneBench, a hierarchical evaluation
benchmark for cutscene generation. Unlike typical tool-use
benchmarks that evaluate short, isolated function calls, cutscene generation requires long-horizon, multi-step orchestration of dozens of interdependent tool invocations with strict ordering constraints – a capability dimension that existing
benchmarks do not cover. We evaluate a range of LLMs on CutsceneBench and analyze their performance across this challenging task.
Read the full abstract ▾
All authors Lanshan He, Haozhou Pang, Qi Gan, Xin Shen, Ziwei Zhang, Yibo Liu, Gang Fang, Bo Liu, Kai Sheng, Shengfeng Zeng, Chaofan Li, Zhen Hui, Keer Zhou, Lan Zhou, Shujun Dai
Computation and Language
Faithfulness-QA: A Counterfactual Entity Substitution Dataset for Training Context-Faithful RAG Models
Li Ju, Junzhe Wang and Qi Zhang
Retrieval-Augmented Generation (RAG) models frequently produce answers grounded in parametric
memory rather than the retrieved context, undermining the core promise of
retrieval augmentation. A fundamental obstacle to fixing this unfaithfulness is the lack of training data that explicitly requires models to prefer context over internal knowledge. We introduce Faithfulness-QA, a large-scale
dataset of 99,094 samples constructed through counterfactual entity substitution. Starting from two established extractive QA
benchmarks–SQuAD and TriviaQA–we automatically identify answer-bearing named entities in each context, replace them with type-consistent alternatives drawn from a curated bank of 76,953 entities, and thereby manufacture controlled knowledge conflicts between context and parametric
memory. Rigorous quality filtering ensures 100% pass rates across four automated checks on random 200-sample audits. We release the full
dataset, the construction pipeline, and a typed entity bank covering eight named entity categories. Faithfulness-QA is designed as a training resource for
attention-based faithfulness objectives and as an evaluation
benchmark for measuring context-grounding behavior in RAG systems. Data and code are available at https://github.com/qzhangFDU/faithfulness-qa-
dataset.
Read the full abstract ▾
Computation and Language
LegalMidm: Use-Case-Driven Legal Domain Specialization for Korean Large Language Model
Youngjoon Jang, Chanhee Park, Hyeonseok Moon et al. (+5 more)
In recent years, the rapid proliferation of open-source large
language models (LLMs) has spurred efforts to turn general-purpose models into domain specialists. However, many domain-specialized LLMs are developed using
datasets and training protocols that are not aligned with the nuanced requirements of real-world applications. In the legal domain, where precision and reliability are essential, this lack of consideration limits practical utility. In this study, we propose a systematic training framework grounded in the practical needs of the legal domain, with a focus on Korean law. We introduce LegalMidm, a Korean legal-domain LLM, and present a methodology for constructing high-quality, use-case-driven legal
datasets and optimized training pipelines. Our approach emphasizes collaboration with legal professionals and rigorous data curation to ensure relevance and factual accuracy, and demonstrates effectiveness in key legal tasks.
Read the full abstract ▾
All authors Youngjoon Jang, Chanhee Park, Hyeonseok Moon, Young-kyoung Ham, Jiwon Moon, Jinhyeon Kim, JuKyung Jung, Heuiseok Lim
Computation and Language
Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMs
Jianghang Lin, Haihua Yang, Deli Yu et al. (+6 more)
Multimodal Large
Language Models (MLLMs) have shown transformative potential in medical applications, yet their performance is hindered by conventional data curation strategies that rely on coarse-grained partitioning by modality or department. Such fragmented approaches fail to capture the hierarchical and interconnected nature of clinical medical knowledge, limiting the models' ability to perform fine-grained recognition and complex
reasoning. In this paper, we propose a novel Entity-Centric Medical Data Engineering framework. We automatically extract entities from authoritative medical literature to construct a Medical Entity Tree (MET), a hierarchical structure that systematically encodes diseases, anatomical structures, modalities, and symptoms into a unified knowledge repository. Building upon the MET, we propose an advanced data engine that includes: (1) node-guided
retrieval to anchor raw data to specific medical concepts, (2) a two-stage hybrid filtering and
alignment pipeline to ensure precise visual-semantic correspondence, and (3) knowledge-aware data synthesis to generate enriched captions and targeted
reasoning VQA pairs, leveraging structural constraints. Extensive evaluations across six medical
benchmarks demonstrate that our approach significantly enhances the medical capabilities of general-purpose MLLMs, improving their ability to handle complex clinical queries and achieve state-of-the-art performance in diverse medical contexts.
Read the full abstract ▾
All authors Jianghang Lin, Haihua Yang, Deli Yu, Kai Wu, Kai Ye, Jinghao Lin, Zihan Wang, Yuhang Wu, Liujuan Cao
Computation and Language
Below-Chance Blindness: Prompted Underperformance in Small LLMs Produces Positional Bias Rather than Answer Avoidance
Jon-Paul Cacioli
Detecting sandbagging–the deliberate underperformance on capability evaluations–is an open problem in AI safety. We tested whether symptom validity testing (SVT) logic from clinical malingering detection could identify sandbagging through below-chance performance (BCB) on forced-choice items. In a pre-registered pilot at the 7-9 billion parameter instruction-tuned scale (3 models, 4 MMLU-Pro domains, 4 conditions, 500 items per
cell, 24,000 total trials), the plausibility gate failed. Zero of 12 model-domain
cells showed significant below-chance performance under sandbagging instruction. Exploratory analyses revealed three qualitatively distinct failure modes. Qwen-2.5-7B and Phi-3.5-mini largely ignored the sandbagging instruction, with 62-88% response identity with the honest baseline. Llama-3-8B complied substantially but implemented underperformance as a positional heuristic, collapsing its response distribution onto middle-alphabet options (E at 31.8%, F at 26.1%) regardless of where the correct answer fell. This produced accuracy boosts of up to 33 percentage points when the correct answer coincidentally occupied the model's preferred position. An explicit anti-task instruction ("pick the least likely answer") drove two of three models below chance, with accuracy as low as 0.024. The capability for answer-aware avoidance therefore exists but is not activated by "deliberately underperform." BCB did not fail as a logical marker of answer-aware avoidance. It was not observed in this regime because the model showing the largest behavioural shift exhibited behaviour consistent with a position-dominant response policy rather than content-aware answer avoidance. We propose that positional-distribution shift may be a more effective behavioural signature than below-chance accuracy for detecting prompted underperformance at this model scale.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
Divake Kumar, Sina Tayebati, Devashri Naik et al. (+2 more)
Vision-
language models (VLMs) are increasingly used as automated judges for
multimodal systems, yet their scores provide no indication of reliability. We study this problem through conformal prediction, a distribution-free framework that converts a judge's point score into a calibrated prediction interval using only score-
token log-probabilities, with no retraining. We present the first systematic analysis of conformal prediction for VLM-as-a-Judge across 3 judges and 14 visual task categories. Our results show that evaluation uncertainty is strongly task-dependent: intervals cover ~40% of the score range for aesthetics and natural images but expand to ~70% for chart and mathematical
reasoning, yielding a quantitative reliability map for
multimodal evaluation. We further identify a failure mode not captured by standard evaluation metrics, ranking-scoring decoupling, where judges achieve high ranking correlation while producing wide, uninformative intervals, correctly ordering responses but failing to assign reliable absolute scores. Finally, we show that interval width is driven primarily by task difficulty and annotation quality, i.e., the same judge and method yield 4.5x narrower intervals on a clean, multi-annotator captioning
benchmark. Code: https://github.com/divake/VLM-Judge-Uncertainty
Read the full abstract ▾
All authors Divake Kumar, Sina Tayebati, Devashri Naik, Ranganath Krishnan, Amit Ranjan Trivedi
Computation and Language · Cross-listed from cs.CV
DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams
Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Gaurav Najpande et al. (+4 more)
Diagram question answering (DQA) requires models to interpret structured visual representations such as charts, maps, infographics,
circuit schematics, and scientific diagrams. Recent
vision-
language models (VLMs) often achieve high answer accuracy on these tasks, yet correct answers do not guarantee that models ground their
reasoning in the diagram regions that support the prediction. Models may instead rely on textual correlations or
dataset artifacts without identifying the visual evidence required to verify the answer. This limitation prevents reliable evaluation of diagram
reasoning and reduces interpretability. We introduce DRAGON, a
benchmark for evaluating evidence-grounded visual
reasoning in diagrams. Given a diagram, a question, and the correct answer, a model must predict bounding boxes that correspond to the visual elements required to justify the answer. These evidence regions may include answer-bearing components, textual labels, legends, axes, connectors, and other supporting structures involved in the
reasoning process. The DRAGON
dataset contains 11,664 annotated question instances from six diagram QA
datasets: ChartQA,
Circuit-VQA, InfographicsVQA, MapIQ, MapWise, and AI2D, with a 2,445-instance test set carrying human-verified evidence annotations and a standardized evaluation framework. Our evaluation on recent VLMs reveals that even Claude Opus 4.6 achieves a Grounding IoU of only 23.8% on AI2D, showing that faithful visual grounding remains an open challenge. DRAGON supports future research on models that ground their predictions in visual evidence.
Read the full abstract ▾
All authors Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Gaurav Najpande, Manan Suri, Dinesh Manocha, Puneet Mathur, Vivek Gupta
Computation and Language
BARRED: Synthetic Training of Custom Policy Guardrails via Asymmetric Debate
Arnon Mazza and Elad Levi
Deploying guardrails for custom policies remains challenging, as generic safety models fail to capture task-specific requirements, while prompting LLMs suffers from inconsistent boundary-case performance and high inference costs. Training custom classifiers achieves both accuracy and efficiency, yet demands substantial labeled data that is costly to obtain. We present BARRED (Boundary
Alignment Refinement through REflection and Debate), a framework for generating faithful and diverse synthetic training data using only a task description and a small set of unlabeled examples. Our approach decomposes the domain space into dimensions to ensure comprehensive coverage, and employs multi-
agent debate to verify label correctness, yielding a high-fidelity training corpus. Experiments across diverse custom policies demonstrate that small
language models finetuned on our synthetic data consistently outperform state-of-the-art proprietary LLMs (including
reasoning models) and dedicated guardrail models. Ablation studies confirm that both dimension decomposition and debate-based verification are critical for ensuring the diversity and label fidelity required for effective fine-tuning. The BARRED framework eliminates the reliance on extensive human annotation, offering a scalable solution for accurate custom guardrails.
Read the full abstract ▾
Computation and Language
CroSearch-R1: Better Leveraging Cross-lingual Knowledge for Retrieval-Augmented Generation
Rui Qi, Fengran Mo, Sijin Lu et al. (+3 more)
A
multilingual collection may contain useful knowledge in other
languages to supplement and correct the facts in the original
language for
Retrieval-Augmented Generation (RAG). However, the vanilla approach that simply concatenates multiple pieces of knowledge from different
languages into the context may fail to improve effectiveness due to the potential disparities across
languages. To better leverage
multilingual knowledge, we propose CroSearch-R1, a search-augmented reinforcement learning framework to integrate
multilingual knowledge into the Group Relative Policy Optimization (GRPO) process. In particular, the approach adopts a multi-turn
retrieval strategy with cross-lingual knowledge integration to dynamically align the knowledge from other
languages as supplementary evidence into a unified representation space. Furthermore, we introduce a
multilingual rollout mechanism to optimize
reasoning transferability across
languages. Experimental results demonstrate that our framework effectively leverages cross-lingual complementarity and improves the effectiveness of RAG with
multilingual collections.
Read the full abstract ▾
All authors Rui Qi, Fengran Mo, Sijin Lu, Yufeng Chen, Jian-Yun Nie, Kaiyu Huang
Computation and Language · Cross-listed from cs.CR
MGTEVAL: An Interactive Platform for Systemtic Evaluation of Machine-Generated Text Detectors
Yuanfan Li, Qi Zhou, Chengzhengxu Li et al. (+5 more)
We present MGTEVAL, an extensible platform for systematic evaluation of Machine-Generated Text (MGT) detectors. Despite rapid progress in MGT detection, existing evaluations are often fragmented across
datasets, preprocessing, attacks, and metrics, making results hard to compare and reproduce. MGTEVAL organizes the workflow into four components:
Dataset Building,
Dataset Attack, Detector Training, and Performance Evaluation. It supports constructing custom
benchmarks by generating MGT with configurable LLMs, applying 12 text attacks to test sets, training detectors via a unified interface, and reporting effectiveness, robustness, and efficiency. The platform provides both command-line and Web-based interfaces for user-friendly experimentation without code rewriting.
Read the full abstract ▾
All authors Yuanfan Li, Qi Zhou, Chengzhengxu Li, Zhaohan Zhang, Chenxu Zhao, Zepu Ruan, Chao Shen, Xiaoming Liu
Computation and Language
Frictive Policy Optimization for LLMs: Epistemic Intervention, Risk-Sensitive Control, and Reflective Alignment
James Pustejovsky and Nikhil Krishnaswamy
We propose Frictive Policy Optimization (FPO), a framework for learning
language model policies that regulate not only what to say, but when and how to intervene in order to manage epistemic and normative risk. Unlike standard
alignment methods that optimize surface-level preference or task utility, FPO treats clarification, verification, challenge, redirection, and refusal as explicit control actions whose purpose is to shape the
evolution of belief, commitment, and uncertainty over time. We formalize
alignment as a risk-sensitive epistemic control problem in which intervention decisions are selected based on their expected effect on downstream epistemic quality rather than on immediate reward alone. We introduce a compact taxonomy of frictive interventions, a structured friction functional that operationalizes multiple
alignment failure modes, and a unified family of FPO methods spanning reward shaping, preference pairing, group-relative ranking, and risk-conditioned trust regions. We further propose an evaluation framework that measures epistemic competence directly through clarification behavior, calibration, contradiction repair, refusal proportionality, and information efficiency. Together, these results provide a formal and algorithmic foundation for learning
agents that are aligned not only in outcome, but in epistemic conduct.
Read the full abstract ▾
Computation and Language
FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments
Amir Saeidi, Venkatesh Mishra, Souradeep Mukhopadhyay et al. (+4 more)
Large
Language Models are being increasingly deployed as the decision-making core of autonomous
agents capable of effecting change in external environments. Yet, in conversational
benchmarks, which simulate real-world customer-centric issue resolution scenarios, these
agents frequently fail due to the cascading effects of incorrect decision-making. These challenges are particularly pronounced for open-source LLMs with smaller parameter sizes, limited context windows, and constrained inference budgets, which contribute to increased error accumulation in agentic settings. To tackle these challenges, we present the Failure-Aware Meta-Agentic (FAMA) framework. FAMA operates in two stages: first, it analyzes failure trajectories from baseline
agents to identify the most prevalent errors; second, it employs an orchestration mechanism that activates a minimal subset of specialized
agents tailored to address these failures by injecting a targeted context for the tool-use
agent before the decision-making step. Experiments across open-source LLMs demonstrate performance gains up to 27% across evaluation modes over standard baselines. These results highlight that targeted curation of context through specialized
agents to address common failures is a valuable design principle for building reliable, multi-turn tool-use LLM
agents that simulate real-world conversational scenarios.
Read the full abstract ▾
All authors Amir Saeidi, Venkatesh Mishra, Souradeep Mukhopadhyay, Gaowen Liu, Ali Payani, Jayanth Srinivasa, Chitta Baral
Computation and Language
Korean aegyo speech shows systematic F1 increase to signal childlike qualities
Ji-eun Kim and Volker Dellwo
Korean aegyo is a socially recognized childlike speaking style used predominantly in romantic interactions among adults. This study examined vowel space modification in aegyo by analyzing formant frequencies from twelve Seoul Korean speakers who produced identical scripts in aegyo and non-aegyo styles. Results show that aegyo
speech features a significant increase in F1 values across vowels and selective fronting of front vowels, leading to vowel space expansion but mainly a shift to higher F1. These findings suggest that adult speakers stylize childlike
speech by imitating the shorter vocal tract of children, mainly through global vowel lowering and partial fronting.
Read the full abstract ▾
Computation and Language
What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective
Guangzeng Han and Xiaolei Huang
Instruction-tuning
datasets often contain substantial redundancy and low-quality samples, necessitating effective data selection methods. We propose an instruction data selection framework based on weighted in-context influence (wICI), which measures how effectively each candidate example reduces instruction-following difficulty for semantically related peers. Through systematic experiments, we address three key questions: what constitutes effective instruction tuning data from an in-context perspective, whether sample difficulty correlates with in-context influence, and how in-context influence translates to instruction tuning effectiveness. Experiments across multiple models and
benchmarks demonstrate that our method consistently outperforms existing baselines under constrained data budgets, while empirically showing that sample difficulty negatively correlates with in-context influence.
Read the full abstract ▾
Computation and Language
LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization
Huyen Nguyen, Haoxuan Zhang, Yang Zhang et al. (+2 more)
Evaluating long document summaries remains the primary bottleneck in summarization research. Existing metrics correlate weakly with human judgments and produce aggregate scores without explaining deficiencies or guiding improvement, preventing effective refinement in applications requiring verifiable accuracy. We introduce LongSumEval, a unified framework bridging evaluation and generation through structured question-answering feedback. The framework operationalizes summary quality as answerability and factual
alignment of question-answer pairs, generating interpretable scores and actionable feedback that identifies coverage gaps and factual inconsistencies. This resolves the misalignment where evaluation operates independently of generation objectives. Meta-evaluation of our QA-based evaluation module across seven
benchmarks demonstrates substantially stronger agreement with human judgments compared to established metrics. Structured feedback enables significant quality improvements through self-refinement without retraining. By demonstrating that evaluation feedback can serve as executable instructions for generation, this work establishes a generalizable paradigm for aligning assessment with improvement, with direct implications for controllable text generation requiring verifiable accuracy and transparent quality control. All code and
datasets will be released in GitHub for reproducibility.
Read the full abstract ▾
All authors Huyen Nguyen, Haoxuan Zhang, Yang Zhang, Junhua Ding, Haihua Chen
Computation and Language
SCOPE:Planning for Hybrid Querying over Clinical Trial Data
Suparno Roy Chowdhury, Manan Roy Choudhury, Tejas Anvekar et al. (+5 more)
We study clinical trial table
reasoning, where answers are not directly stored in visible
cells but must be reasoned from semantic understanding through normalization, classification, extraction, or lightweight domain
reasoning. Motivated by the observation that current LLM approaches often suffer from "bad
reasoning" under implicit planning assumptions, we focus on settings in which the model must recover implicit attributes such as therapy type, added
agents, endpoint roles, or follow-up status from partially observed clinical-trial tables. We propose SCOPE (Structured Clinical hybrid Planning for Evidence
retrieval in clinical trials), a multi-LLM planner-based framework that decomposes the task into row selection, structured planning, and execution. The planner makes the source field,
reasoning rules, and output constraints explicit before answer generation, reducing ambiguity relative to direct prompting. We evaluate SCOPE on 1,500 hybrid
reasoning questions over oncology clinical-trial tables against zero-shot, few-shot, chain-of-thought, TableGPT2, Blend-SQL, and EHRAgent. Results show that explicit multi-LLM planning improves accuracy for
reasoning-based questions while offering a stronger accuracy-efficiency tradeoff than heavier agentic baselines. Our findings position clinical trial
reasoning as a distinct table understanding problem and highlight hybrid planner-based decomposition as an effective solution
Read the full abstract ▾
All authors Suparno Roy Chowdhury, Manan Roy Choudhury, Tejas Anvekar, Muhammad Ali Khan, Kaneez Zahra Rubab Khakwani, Mohamad Bassam Sonbol, Irbaz Bin Riaz, Vivek Gupta
Computation and Language · Cross-listed from cs.AI
Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling
Ocean Monjur, Shahriar Kabir Nahin and Anshuman Chhabra
Large
Language Models (LLMs) now exhibit remarkable
reasoning capabilities through test-time compute scaling (TTS), with impressive performance across math and coding
benchmarks. In parallel, research in model compression has developed
pruning methods that seek to remove redundant/detrimental parameters without sacrificing task performance. The intersection of these two research advancements lays the foundation for our work. Specific to
reasoning LLMs, prior work has shown that structured
pruning (methods which remove entire set of layer blocks), significantly degrades TTS
reasoning performance. However, in this work, we revisit this assumption and investigate whether unstructured
pruning (methods that carefully remove only certain redundant/detrimental weights) exhibits similar limitations. Surprisingly, our extensive experiments across four
reasoning benchmarks on two
reasoning LLMs: s1.1-7B and Qwen3-8B, consistently show that unstructured
pruning augments TTS performance compared to structured
pruning, and at times can even outperform the unpruned full-weight LLMs. Furthermore, we also empirically study the impact of different layer-wise sparsity allocation strategies, which are an important parametric choice for instantiating these unstructured methods. These findings challenge the conventional notion that
pruning always reduces TTS performance and in fact, suggest that carefully undertaken
pruning can retain TTS effectiveness.
Read the full abstract ▾
Computation and Language
The Dynamics of Delusion: Modeling Bidirectional False Belief Amplification in Human-Chatbot Dialogue
Ashish Mehta, Jared Moore, Jacy Reese Anthis et al. (+6 more)
There is growing concern that AI chatbots might fuel delusional beliefs in users. Some have suggested that humans and chatbots mutually reinforce false beliefs over time, but quantitative evidence is lacking. Using a unique
dataset of chat logs from individuals who exhibited delusional thinking, we developed a latent state model that captures accumulating and decaying influences between humans and chatbots. We find that a bidirectional influence model substantially outperforms a unidirectional alternative where humans are the primary driver of delusion. We find that humans exert strong but short-lived influence on chatbots, whereas chatbots exert longer-lasting influence on humans. Moreover, chatbots exert strong, stable self-influence over their own future outputs that tends to perpetuate delusions over long stretches of conversation. In fact, this chatbot self-influence constituted the dominant pathway when considering accumulated influence over time. Overall, these results indicate that humans tend to drive sharp, immediate increases in delusion, whereas chatbots sustain and propagate these effects over longer timescales. Together, these findings provide the first empirical, quantitative evidence that human-chatbot interactions can form feedback loops of delusion, decomposable into distinct pathways with dissociable temporal dynamics. By doing so, they can inform the development of safer AI systems.
Read the full abstract ▾
All authors Ashish Mehta, Jared Moore, Jacy Reese Anthis, William Agnew, Eric Lin, Peggy Yin, Desmond C. Ong, Nick Haber, Carol Dweck
Computation and Language · Cross-listed from cs.AI
Cooperate to Compete: Strategic Coordination in Multi-Agent Conquest
Abigail O'Neill, Alan Zhu, Mihran Miroyan et al. (+2 more)
Language Model (LM)-based
agents remain largely untested in mixed-motive settings where
agents must leverage short-term cooperation for long-term competitive goals (e.g., multi-party politics). We introduce Cooperate to Compete (C2C), a multi-
agent environment where players can engage in private negotiations while competing to be the first to achieve their secret objective. Players have asymmetric objectives and negotiations are non-binding, allowing alliances to form and break as players' short-term interests align and diverge. We run AI only games and conduct a user study pitting human players against AI opponents. We identify significant differences between human and AI negotiation behaviors, finding that humans favor lower-complexity deals and are significantly less reliable partners compared to LM-based
agents. We also find that humans are more aggressive negotiators, accepting deals without a counteroffer only 56.3% of the time compared to 67.6% for LM-based
agents. Through targeted prompting inspired by these findings, we modify
agents' negotiation behavior and improve win rates from 22.2% to 32.7%. We run over 1,100 games with over 16,000 private conversations totaling 15.2 million
tokens and over 150,000 player actions. Our results establish C2C as a testbed for studying and building LM-based
agents that can navigate the sophisticated coordination required for real-world deployments. The game, code, and
dataset may be found at https://negotiationgame.io/c2c.
Read the full abstract ▾
All authors Abigail O'Neill, Alan Zhu, Mihran Miroyan, Narges Norouzi, Joseph E. Gonzalez