Computation and Language · Cross-listed from cs.AI
Self-Guided Plan Extraction for Instruction-Following Tasks with Goal-Conditional Reinforcement Learning
Zoya Volovikova, Nikita Sorokin, Dmitriy Lukashevskiy et al. (+2 more)
We introduce SuperIgor, a framework for instruction-following tasks. Unlike prior methods that rely on predefined subtasks, SuperIgor enables a
language model to generate and refine high-level plans through a self-learning mechanism, reducing the need for manual
dataset annotation. Our approach involves iterative co-training: an RL
agent is trained to follow the generated plans, while the
language model adapts and modifies these plans based on RL feedback and preferences. This creates a feedback loop where both the
agent and the planner improve jointly. We validate our framework in environments with rich dynamics and stochasticity. Results show that SuperIgor
agents adhere to instructions more strictly than baseline methods, while also demonstrating strong generalization to previously unseen instructions.
Read the full abstract ▾
All authors Zoya Volovikova, Nikita Sorokin, Dmitriy Lukashevskiy, Aleksandr Panov, Alexey Skrynnik
Computation and Language · Cross-listed from cs.IR
Self-Aware Vector Embeddings for Retrieval-Augmented Generation: A Neuroscience-Inspired Framework for Temporal, Confidence-Weighted, and Relational Knowledge
Naizhong Xu
Modern
retrieval-augmented generation (RAG) systems treat vector
embeddings as static, context-free artifacts: an
embedding has no notion of when it was created, how trustworthy its source is, or which other
embeddings depend on it. This flattening of knowledge has a measurable cost: recent work on VersionRAG reports that conventional RAG achieves only 58% accuracy on versioned technical queries, because
retrieval returns semantically similar but temporally invalid content. We propose SmartVector, a framework that augments dense
embeddings with three explicit properties – temporal awareness, confidence decay, and relational awareness – and a five-stage lifecycle modeled on hippocampal-neocortical
memory consolidation. A
retrieval pipeline replaces pure cosine similarity with a four-signal score that mixes semantic relevance, temporal validity, live confidence, and graph-relational importance. A background consolidation
agent detects contradictions, builds dependency edges, and propagates updates along those edges as graph-neural-network-style messages. Confidence is governed by a closed-form function combining an Ebbinghaus-style exponential decay, user-feedback reconsolidation, and logarithmic access reinforcement. We formalize the model, relate it to temporal knowledge graph
embedding, agentic
memory architectures, and uncertainty-aware RAG, and present a reference implementation. On a reproducible synthetic versioned-policy
benchmark of 258 vectors and 138 queries, SmartVector roughly doubles top-1 accuracy over plain cosine RAG (62.0% vs. 31.0% on a held-out split), drops stale-answer rate from 35.0% to 13.3%, cuts Expected Calibration Error by nearly 2x (0.244 vs. 0.470), reduces re-
embedding cost per single-word edit by 77%, and is robust across contradiction-injection rates from 0% to 75%.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.MA
Trust, Lies, and Long Memories: Emergent Social Dynamics and Reputation in Multi-Round Avalon with LLM Agents
Suveen Ellawela
We study emergent social dynamics in LLM
agents playing The Resistance: Avalon, a hidden-role deception game. Unlike prior work on single-game performance, our
agents play repeated games while retaining
memory of previous interactions, including who played which roles and how they behaved, enabling us to study how social dynamics evolve. Across 188 games, two key phenomena emerge. First, reputation dynamics emerge organically when
agents retain cross-game
memory:
agents reference past behavior in statements like "I am wary of repeating last game's mistake of over-trusting early success." These reputations are role-conditional: the same
agent is described as "straightforward" when playing good but "subtle" when playing evil, and high-reputation players receive 46% more team inclusions. Second, higher
reasoning effort supports more strategic deception: evil players more often pass early missions to build trust before sabotaging later ones, 75% in high-effort games vs 36% in low-effort games. Together, these findings show that repeated interaction with
memory gives rise to measurable reputation and deception dynamics among LLM
agents.
Read the full abstract ▾
Computation and Language
Ask Only When Needed: Proactive Retrieval from Memory and Skills for Experience-Driven Lifelong Agents
Yuxuan Cai, Wei Li, Jie Zhou et al. (+4 more)
Online lifelong learning
agents must decide not only how to act but also when to consult prior experience to continually improve on long-horizon tasks. Existing methods typically retrieve
memories passively, such as at task initialization or after each step, and therefore miss knowledge gaps that arise during interaction. We propose ProactAgent, an experience-driven lifelong learning framework for proactive
retrieval over a structured Experience Base. ProactAgent continually improves through ExpOnEvo, which jointly updates policies and refines
memory, organizing past interactions into factual, episodic, and skill repositories. It further introduces ProactRL, which treats
retrieval as an explicit policy action and learns when and what to retrieve. By comparing paired continuations from identical interaction prefixes with and without
retrieval, ProactRL provides step-level process rewards that encourage
retrieval only when it improves task outcomes or efficiency. Experiments on SciWorld, AlfWorld, and StuLife show that ProactAgent consistently outperforms all baselines, achieving up to 32% relative improvement in success rate and over 33% reduction in interaction rounds. Our code will be publicly available at GitHub.
Read the full abstract ▾
All authors Yuxuan Cai, Wei Li, Jie Zhou, Qin Chen, Xin Li, Bo Zhang, Liang He
Computation and Language
Where Reasoning Breaks: Logic-Aware Path Selection by Controlling Logical Connectives in LLMs Reasoning Chains
Seunghyun Park and Yuanyuan Lei
While LLMs demonstrate impressive
reasoning capabilities, they remain fragile in multi-step logical deduction, where a single transition error can propagate through the entire
reasoning chain, leading to unstable performance. In this work, we identify logical connectives as primary points of this structural fragility. Through empirical analysis, we show that connective
tokens function as high
entropy forking points, at which models frequently struggle to determine the correct logical direction. Motivated by this observation, we hypothesize that intervening in logical connective selection can guide LLMs toward more correct logical direction, thereby improving the overall
reasoning chain. To validate this hypothesis, we propose a multi-layered framework that intervenes specifically at these logic-critical junctions in the
reasoning process. Our framework includes (1) Gradient-based Logical Steering to guide LLMs internal representations towards valid
reasoning subspaces, (2) Localized Branching to resolve ambiguity via targeted look-ahead search, and (3) Targeted Transition Preference Optimization, a
surgical reinforcement learning objective that selectively optimizes single-
token preferences at logical pivots. Crucially, by concentrating intervention solely on logic-critical transitions, our framework achieves a favorable accuracy–efficiency trade-off compared to global inference time scaling methods like beam search and self-consistency.
Read the full abstract ▾
Computation and Language
LLM StructCore: Schema-Guided Reasoning Condensation and Deterministic Compilation
Serhii Zabolotnii
Automatically filling Case Report Forms (CRFs) from clinical notes is challenging due to noisy
language, strict output contracts, and the high cost of false positives. We describe our CL4Health 2026 submission for Dyspnea CRF filling (134 items) using a contract-driven two-stage design grounded in Schema-Guided
Reasoning (SGR). The key task property is extreme sparsity: the majority of fields are unknown, and official scoring penalizes both empty values and unsupported predictions. We shift from a single-step "LLM predicts 134 fields" approach to a decomposition where (i) Stage 1 produces a stable SGR-style JSON summary with exactly 9 domain keys, and (ii) Stage 2 is a fully deterministic, 0-LLM compiler that parses the Stage 1 summary, canonicalizes item names, normalizes predictions to the official controlled vocabulary, applies evidence-gated false-positive filters, and expands the output into the required 134-item format. On the dev80 split, the best teacher configuration achieves macro-F1 0.6543 (EN) and 0.6905 (IT); on the hidden test200, the submitted English variant scores 0.63 on Codabench. The pipeline is
language-agnostic: Italian results match or exceed English with no
language-specific engineering.
Read the full abstract ▾
Computation and Language
LayerTracer: A Joint Task-Particle and Vulnerable-Layer Analysis framework for Arbitrary Large Language Model Architectures
Yuhang Wu, Qinyuan Liu, Qiuyang Zhao et al. (+1 more)
Currently, Large
Language Models (LLMs) feature a diversified architectural landscape, including traditional
Transformer, GateDeltaNet, and Mamba. However, the evolutionary laws of hierarchical representations, task knowledge formation positions, and network robustness bottleneck mechanisms in various LLM architectures remain unclear, posing core challenges for hybrid architecture design and model optimization. This paper proposes LayerTracer, an architecture-agnostic end-to-end analysis framework compatible with any LLM architecture. By extracting hidden states layer-by-layer and mapping them to vocabulary probability distributions, it achieves joint analysis of task particle localization and layer vulnerability quantification. We define the task particle as the key layer where the target
token probability first rises significantly, representing the model's task execution starting point, and the vulnerable layer is defined as the layer with the maximum Jensen-Shannon (JS) divergence between output distributions before and after mask perturbation, reflecting its sensitivity to disturbances. Experiments on models of different parameter scales show that task particles mainly appear in the deep layers of the model regardless of parameter size, while larger-parameter models exhibit stronger hierarchical robustness. LayerTracer provides a scientific basis for layer division, module ratio, and gating switching of hybrid architectures, effectively optimizing model performance. It accurately locates task-effective layers and stability bottlenecks, offering universal support for LLM structure design and interpretability research.
Read the full abstract ▾
All authors Yuhang Wu, Qinyuan Liu, Qiuyang Zhao, Qingwei Chong
Computation and Language
Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection
Yassine Turki, Vinko Sabolčec, Bettina Messmer et al. (+1 more)
As Large
Language Models (LLMs) scale, data curation has shifted from maximizing volume to optimizing the signal-to-noise ratio by performing quality filtering. However, for many
languages, native high quality data is insufficient to train robust quality classifiers. This work investigates the idea that quality markers in
embedding space may show cross-lingual consistency, which would allow high-resource
languages to subsidize the filtering of low-resource ones. We evaluate various filtering strategies, including cross-lingual transfer, third quartile sampling (Q3), and retention rate tuning. Our results demonstrate that massive
multilingual pooling frequently outperforms monolingual baselines in both rank stability and aggregate accuracy for a 1B model trained on 103B
tokens, delivering gains for high resource
languages (1.2% increase in aggregate normalized accuracy for French) and matching or exceeding monolingual baselines for low-resource
languages. However, we find that scale alone does not guarantee stability. Furthermore, for high-resource
languages like French, we show that refining the decision boundary through third quartile sampling (Q3) or tuning the retention rate is necessary to fully leverage the
multilingual signal.
Read the full abstract ▾
All authors Yassine Turki, Vinko Sabolčec, Bettina Messmer, Martin Jaggi
Computation and Language
Enhancing Research Idea Generation through Combinatorial Innovation and Multi-Agent Iterative Search Strategies
Shuai Chen and Chengzhi Zhang
Scientific progress depends on the continual generation of innovative re-search ideas. However, the rapid growth of scientific literature has greatly increased the cost of knowledge filtering, making it harder for researchers to identify novel directions. Although existing large
language model (LLM)-based methods show promise in research idea generation, the ideas they produce are often repetitive and lack depth. To address this issue, this study proposes a multi-
agent iterative planning search strategy inspired by com-binatorial innovation theory. The framework combines iterative knowledge search with an LLM-based multi-
agent system to generate, evaluate, and re-fine research ideas through repeated interaction, with the goal of improving idea diversity and novelty. Experiments in the natural
language processing domain show that the proposed method outperforms state-of-the-art base-lines in both diversity and novelty. Further comparison with ideas derived from top-tier machine learning conference papers indicates that the quality of the generated ideas falls between that of accepted and rejected papers. These results suggest that the proposed framework is a promising approach for supporting high-quality research idea generation. The source code and
dataset used in this paper are publicly available on Github repository: https://github.com/ChenShuai00/MAGenIdeas. The demo is available at https://huggingface.co/spaces/cshuai20/MAGenIdeas.
Read the full abstract ▾
Computation and Language
Aligning Stuttered-Speech Research with End-User Needs: Scoping Review, Survey, and Guidelines
Hawau Olamide Toyin, Mutiah Apampa, Toluwani Aremu et al. (+6 more)
Atypical
speech is receiving greater
attention in
speech technology research, but much of this work unfolds with limited interdisciplinary dialogue. For stuttered
speech in particular, it is widely recognised that current
speech recognition systems fall short in practice, and current evaluation methods and research priorities are not systematically grounded in end-user experiences and needs. In this work, we analyse these gaps through 1) a scoping review of papers that deal with stuttered
speech and 2) a survey of 70 stakeholders, including adults who stutter and
speech-
language pathologists. By analysing these two perspectives, we propose a taxonomy of stuttered-
speech research, identify where current research directions diverge from the needs articulated by stakeholders, and conclude by outlining concrete guidelines and directions towards addressing the real needs of the stuttering community.
Read the full abstract ▾
All authors Hawau Olamide Toyin, Mutiah Apampa, Toluwani Aremu, Humaid Alblooshi, Ana Rita Valente, Gonçalo Leal, Zhengjun Yue, Zeerak Talat, Hanan Aldarmaki
Computation and Language
Effects of Cross-lingual Evidence in Multilingual Medical Question Answering
Anar Yeginbergen, Maite Oronoz and Rodrigo Agerri
This paper investigates
Multilingual Medical Question Answering across high-resource (English, Spanish, French, Italian) and low-resource (Basque, Kazakh)
languages. We evaluate three types of external evidence sources across models of varying size: curated repositories of specialized medical knowledge, web-retrieved content, and explanations from LLM's parametric knowledge. Moreover, we conduct experiments with
multilingual, monolingual and cross-lingual
retrieval. Our results demonstrate that larger models consistently achieve superior performance in English across baseline evaluations. When incorporating external knowledge, web-retrieved data in English proves most beneficial for high-resource
languages. Conversely, for low-resource
languages, the most effective strategy combines
retrieval in both English and the target
language, achieving comparable accuracy to high-resource
language results. These findings challenge the assumption that external knowledge systematically improves performance and reveal that effective strategies depend on both the source of
language resources and on model scale. Furthermore, specialized medical knowledge sources such as PubMed are limited: while they provide authoritative expert knowledge, they lack adequate
multilingual coverage
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
CHASM: Unveiling Covert Advertisements on Chinese Social Media
Jingyi Zheng, Tianyi Hu, Yule Liu et al. (+5 more)
Current
benchmarks for evaluating large
language models (LLMs) in social media moderation completely overlook a serious threat: covert advertisements, which disguise themselves as regular posts to deceive and mislead consumers into making purchases, leading to significant ethical and legal concerns. In this paper, we present the CHASM, a first-of-its-kind
dataset designed to evaluate the capability of
Multimodal Large
Language Models (MLLMs) in detecting covert advertisements on social media. CHASM is a high-quality, anonymized, manually curated
dataset consisting of 4,992 instances, based on real-world scenarios from the Chinese social media platform Rednote. The
dataset was collected and annotated under strict
privacy protection and quality control protocols. It includes many product experience sharing posts that closely resemble covert advertisements, making the
dataset particularly challenging.The results show that under both zero-shot and in-context learning settings, none of the current MLLMs are sufficiently reliable for detecting covert advertisements.Our further experiments revealed that fine-tuning open-source MLLMs on our
dataset yielded noticeable performance gains. However, significant challenges persist, such as detecting subtle cues in comments and differences in visual and textual structures.We provide in-depth error analysis and outline future research directions. We hope our study can serve as a call for the research community and platform moderators to develop more precise defenses against this emerging threat.
Read the full abstract ▾
All authors Jingyi Zheng, Tianyi Hu, Yule Liu, Zhen Sun, Zongmin Zhang, Zifan Peng, Wenhan Dong, Xinlei He
Computation and Language
Knowledge Capsules: Structured Nonparametric Memory Units for LLMs
Bin Ju, Shenfeng Weng, Danying Zhou et al. (+2 more)
Large
language models (LLMs) encode knowledge in parametric weights, making it costly to update or extend without retraining.
Retrieval-augmented generation (RAG) mitigates this limitation by appending retrieved text to the input, but operates purely through context expansion, where external knowledge competes as
tokens within the
attention mechanism. As a result, its influence is indirect and often unstable, particularly in long context and multi hop
reasoning scenarios. We propose Knowledge Capsules, structured nonparametric
memory units that represent normalized relational knowledge and can be constructed directly from document corpora using a frozen base model. Instead of injecting knowledge as text, we introduce an External Key Value Injection (KVI) framework that compiles capsules into
attention-compatible key value representations, enabling external knowledge to directly participate in the model's
attention computation. By shifting knowledge integration from context-level augmentation to
memory level interaction, the proposed framework consistently outperforms RAG and GraphRAG across multiple QA
benchmarks, with improved stability and accuracy in long context and multi hop
reasoning, while requiring no parameter updates.
Read the full abstract ▾
All authors Bin Ju, Shenfeng Weng, Danying Zhou, Rongkai Xu, Kunkai Su
Computation and Language · Cross-listed from cs.RO
MOMO: A framework for seamless physical, verbal, and graphical robot skill learning and adaptation
Markus Knauer, Edoardo Fiorini, Maximilian Mühlbauer et al. (+10 more)
Industrial
robot applications require increasingly flexible systems that non-expert users can easily adapt for varying tasks and environments. However, different adaptations benefit from different interaction modalities. We present an interactive framework that enables
robot skill adaptation through three complementary modalities: kinesthetic touch for precise spatial corrections, natural
language for high-level semantic modifications, and a graphical web interface for visualizing
geometric relations and trajectories, inspecting and adjusting parameters, and editing via-points by drag-and-drop. The framework integrates five components: energy-based human-intention detection, a tool-based LLM architecture (where the LLM selects and parameterizes predefined functions rather than generating code) for safe natural
language adaptation, Kernelized Movement Primitives (KMPs) for motion encoding, probabilistic Virtual Fixtures for guided demonstration recording, and ergodic control for surface finishing. We demonstrate that this tool-based LLM architecture generalizes skill adaptation from KMPs to ergodic control, enabling voice-commanded surface finishing. Validation on a 7-DoF torque-controlled
robot at the Automatica 2025 trade fair demonstrates the practical applicability of our approach in industrial settings.
Read the full abstract ▾
All authors Markus Knauer, Edoardo Fiorini, Maximilian Mühlbauer, Stefan Schneyer, Promwat Angsuratanawech, Florian Samuel Lay, Timo Bachmann, Samuel Bustamante, Korbinian Nottensteiner, Freek Stulp, Alin Albu-Schäffer, João Silvério, Thomas Eiband
Computation and Language · Cross-listed from cs.SE
Deja Vu at Scale: Paraphrase-Robust Detection of Duplicate Gherkin Steps in Behaviour-Driven Software Testing with Sentence-Transformer Embeddings and a 1.1M-Step Open Benchmark
Ali Hassaan Mughal, Noor Fatima and Muhammad Bilal
Context. Behaviour-Driven Development (BDD) suites in Gherkin accumulate step-text duplication with documented maintenance cost. Prior detectors either require runnable tests or are single-organisation, leaving a gap: a static, paraphrase-robust, step-level detector and a public
benchmark to calibrate it. Objective. We release (i) the largest cross-organisational BDD step corpus to date, (ii) a labelled pair-level calibration
benchmark, and (iii) a four-strategy detector with a consolidation-savings model linking clusters to ISO/IEC 25010 maintainability sub-characteristics. Method. The corpus contains 347 public GitHub repositories, 23,667 .feature files, and 1,113,616 Gherkin steps, SPDX-tagged. The detector layers exact hashing, normalised Levenshtein, sentence-
transformer cosine, and a Levenshtein-banded hybrid. Calibration uses 1,020 manually labelled step pairs under a released rubric (60-pair overlap, Fleiss kappa = 0.84). We report precision, recall, and F1 with bootstrap 95% CIs under the primary rubric and a score-free relabelling, and
benchmark against SourcererCC-style and NiCad-style lexical baselines. Results. Step-weighted exact-duplicate rate is 80.2%; median-repository rate is 58.6% (Spearman rho = 0.51). The top hybrid cluster has 20,737 occurrences across 2,245 files. Near-exact reaches F1 = 0.822 on score-free labels; semantic F1 = 0.906 under the primary rubric reflects a disclosed stratification artefact. Lexical baselines reach F1 = 0.761 and 0.799. The savings model estimates 893,357 corpus-wide eliminable step occurrences; on the median repository 62.5% of step lines are eliminable.
Read the full abstract ▾
Computation and Language
Not all ANIMALs are equal: metaphorical framing through source domains and semantic frames
Yulia Otmakhova, Matteo Guida and Lea Frermann
Metaphors are powerful framing devices, yet their source domains alone do not fully explain the specific associations they evoke. We argue that the interplay between source domains and semantic frames determines how metaphors shape understanding of complex issues, and present a computational framework that allows to derive salient discourse metaphors through their source domains and semantic frames. Applying this framework to
climate change news, we uncover not only well-known source domains but also reveal nuanced frame-level associations that distinguish how the issue is portrayed. In analyzing immigration discourse across political ideologies, we demonstrate that liberals and conservatives systematically employ different semantic frames within the same source domains, with conservatives favoring frames emphasizing uncontrollability and liberals choosing neutral or more “victimizing” semantic frames. Our work bridges conceptual metaphor theory and linguistics, providing the first NLP approach for discovery of discourse metaphors and fine-grained analysis of differences in metaphorical framing. Code, data and statistical scripts are available at https://github.com/julia-nixie/ConceptFrameMet.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
HaS: Accelerating RAG through Homology-Aware Speculative Retrieval
Peng Peng, Weiwei Lin, Wentai Wu et al. (+2 more)
Retrieval-Augmented Generation (RAG) expands the knowledge boundary of large
language models (LLMs) at inference by retrieving external documents as context. However,
retrieval becomes increasingly time-consuming as the knowledge databases grow in size. Existing acceleration strategies either compromise accuracy through approximate
retrieval, or achieve marginal gains by reusing results of strictly identical queries. We propose HaS, a homology-aware speculative
retrieval framework that performs low-latency speculative
retrieval over restricted scopes to obtain candidate documents, followed by validating whether they contain the required knowledge. The validation, grounded in the homology relation between queries, is formulated as a homologous query re-identification task: once a previously observed query is identified as a homologous re-encounter of the incoming query, the draft is deemed acceptable, allowing the system to bypass slow full-database
retrieval. Benefiting from the prevalence of homologous queries under real-world popularity patterns, HaS achieves substantial efficiency gains. Extensive experiments demonstrate that HaS reduces
retrieval latency by 23.74% and 36.99% across
datasets with only a 1-2% marginal accuracy drop. As a plug-and-play solution, HaS also significantly accelerates complex multi-hop queries in modern agentic RAG pipelines. Source code is available at: https://github.com/ErrEqualsNil/HaS.
Read the full abstract ▾
All authors Peng Peng, Weiwei Lin, Wentai Wu, Xinyang Wang, Yongheng Liu
Computation and Language
Decoding Text Spans for Efficient and Accurate Named-Entity Recognition
Andrea Maracani, Savas Ozkan, Junyi Zhu et al. (+2 more)
Named Entity Recognition (NER) is a key component in industrial information extraction pipelines, where systems must satisfy strict latency and throughput constraints in addition to strong accuracy. State-of-the-art NER accuracy is often achieved by span-based frameworks, which construct span representations from
token encodings and classify candidate spans. However, many span-based methods enumerate large numbers of candidates and process each candidate with marker-augmented inputs, substantially increasing inference cost and limiting scalability in large-scale deployments. In this work, we propose SpanDec, an efficient span-based NER framework that targets this bottleneck. Our main insight is that span representation interactions can be computed effectively at the final
transformer stage, avoiding redundant computation in earlier layers via a lightweight decoder dedicated to span representations. We further introduce a span filtering mechanism during enumeration to prune unlikely candidates before expensive processing. Across multiple
benchmarks, SpanDec matches competitive span-based baselines while improving throughput and reducing computational cost, yielding a better accuracy-efficiency trade-off suitable for high-volume serving and on-device applications.
Read the full abstract ▾
All authors Andrea Maracani, Savas Ozkan, Junyi Zhu, Sinan Mutlu, Mete Ozay
Computation and Language
DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories
Neemesh Yadav, Palakorn Achananuparp, Jing Jiang et al. (+1 more)
We introduce DialToM, an annotated Theory of Mind (ToM)
benchmark built from naturalistic human-human dialogues using a multiple-choice evaluation framework. Concurrent with recent work showing a gap between explicit mental-state inference and applied ToM in synthetic settings~\cite{gu2024simpletom}, we establish a stricter State-Driven Diagnostic Probe in which models must forecast state-consistent dialogue trajectories solely from isolated mental-state profiles without dialogue context. Our evaluation reveals a systematic
reasoning asymmetry – LLMs excel at inferring mental states (Literal ToM) but struggle to leverage them for social
forecasting (Functional ToM). Crucially, a domain expert achieves 100% accuracy on this task, proving its validity and establishing a stark human-AI capability gap. Further, a teacher-student
reasoning injection probe shows that Gemini 3 Pro – which establishes the leading baseline – possesses robust Functional ToM capabilities for context-free
forecasting that are transferable to weaker models. DialToM, its evaluation code, and
dataset are publicly available at https://github.com/Stealth-py/DialToM.
Read the full abstract ▾
All authors Neemesh Yadav, Palakorn Achananuparp, Jing Jiang, Ee-Peng Lim
Computation and Language
WebGen-R1: Incentivizing Large Language Models to Generate Functional and Aesthetic Websites with Reinforcement Learning
Juyong Jiang, Chenglin Cai, Chansung Park et al. (+4 more)
While Large
Language Models (LLMs) excel at function-level code generation, project-level tasks such as generating functional and visually aesthetic multi-page websites remain highly challenging. Existing works are often limited to single-page static websites, while agentic frameworks typically rely on multi-turn execution with proprietary models, leading to substantial
token costs, high latency, and brittle integration. Training a small LLM end-to-end with reinforcement learning (RL) is a promising alternative, yet it faces a critical bottleneck in designing reliable and computationally feasible rewards for website generation. Unlike single-file coding tasks that can be verified by unit tests, website generation requires evaluating inherently subjective aesthetics, cross-page interactions, and functional correctness. To this end, we propose WebGen-R1, an end-to-end RL framework tailored for project-level website generation. We first introduce a scaffold-driven structured generation paradigm that constrains the large open-ended action space and preserves architectural integrity. We then design a novel cascaded
multimodal reward that seamlessly couples structural guarantees with execution-grounded functional feedback and
vision-based aesthetic supervision. Extensive experiments demonstrate that our WebGen-R1 substantially transforms a 7B base model from generating nearly nonfunctional websites into producing deployable, aesthetically aligned multi-page websites. Remarkably, our WebGen-R1 not only consistently outperforms heavily scaled open-source models (up to 72B), but also rivals the state-of-the-art DeepSeek-R1 (671B) in functional success, while substantially exceeding it in valid rendering and aesthetic
alignment. These results position WebGen-R1 as a viable path for scaling small open models from function-level code generation to project-level web application generation.
Read the full abstract ▾
All authors Juyong Jiang, Chenglin Cai, Chansung Park, Jiasi Shen, Sunghun Kim, Jianguo Li, Yue Wang
Computation and Language
Graph2Counsel: Clinically Grounded Synthetic Counseling Dialogue Generation from Client Psychological Graphs
Aishik Mandal, Hiba Arnaout, Clarissa W. Ong et al. (+5 more)
Rising demand for mental health support has increased interest in using Large
Language Models (LLMs) for counseling, but adapting them to this safety-critical domain is hindered by limited real-world data due to
privacy constraints. Synthetic
datasets provide a promising alternative, but existing approaches often rely on unstructured or semi-structured text inputs and overlook structural dependencies between a client's cognitive, emotional, and behavioral states, leading to psychologically inconsistent and less realistic interactions. We introduce Graph2Counsel, a framework for generating synthetic counseling sessions grounded in Client Psychological Graphs (CPGs) that encode relationships among clients' thoughts,
emotions, and behaviors. Graph2Counsel uses a structured prompting pipeline guided by counselor strategies and CPG, and explores prompting strategies including CoT and Multi-
Agent Feedback. It produces 760 sessions from 76 CPGs across diverse client profiles. In expert evaluation, our
dataset outperforms prior
datasets on specificity, counselor competence, authenticity, conversational flow, and safety (Krippendorff's $\alpha$ = 0.70). Fine-tuning an open-source model on this
dataset improves performance on several metrics on CounselingBench and CounselBench, while matching baselines on others. We further introduce Therapy-Eval, a multi-turn evaluation framework, and demonstrate the effectiveness of our fine-tuned model in realistic therapeutic conversations. We make our code, data and fine-tuned model public.
Read the full abstract ▾
All authors Aishik Mandal, Hiba Arnaout, Clarissa W. Ong, Juliet Bockhorst, Kate Sheehan, Rachael Moldow, Tanmoy Chakraborty, Iryna Gurevych
Computation and Language · Cross-listed from cs.CV
SignDATA: Data Pipeline for Sign Language Translation
Kuanwei Chen and Tingyi Lin
Sign-
language datasets are difficult to preprocess consistently because they vary in annotation schema, clip timing, signer framing, and
privacy constraints. Existing work usually reports downstream models, while the preprocessing pipeline that converts raw
video into training-ready pose or
video artifacts remains fragmented, backend-specific, and weakly documented. We present SignDATA, a config-driven preprocessing toolkit that standardizes heterogeneous sign-
language corpora into comparable outputs for learning. The system supports two end-to-end recipes: a pose recipe that performs acquisition, manifesting, person localization, clipping, cropping, landmark extraction, normalization, and WebDataset export, and a
video recipe that replaces pose extraction with signer-cropped
video packaging. SignDATA exposes interchangeable MediaPipe and MMPose backends behind a common interface, typed job schemas, experiment-level overrides, and per-stage checkpointing with config- and manifest-aware hashes. We validate the toolkit through a research-oriented evaluation design centered on backend comparison, preprocessing ablations, and
privacy-aware
video generation on
datasets. Our contribution is a reproducible preprocessing layer for sign-
language research that makes extractor choice, normalization policy, and
privacy tradeoffs explicit, configurable, and empirically comparable.Code is available at https://github.com/balaboom123/signdata-slt.
Read the full abstract ▾
Computation and Language
Surrogate modeling for interpreting black-box LLMs in medical predictions
Changho Han, Songsoo Kim, Dong Won Kim et al. (+4 more)
Large
language models (LLMs), trained on vast
datasets, encode extensive real-world knowledge within their parameters, yet their black-box nature obscures the mechanisms and extent of this encoding. Surrogate modeling, which uses simplified models to approximate complex systems, can offer a path toward better interpretability of black-box models. We propose a surrogate modeling framework that quantitatively explains LLM-encoded knowledge. For a specific hypothesis derived from domain knowledge, this framework approximates the latent LLM knowledge space using observable elements (input-output pairs) through extensive prompting across a comprehensive range of simulated scenarios. Through proof-of-concept experiments in medical predictions, we demonstrate our framework's effectiveness in revealing the extent to which LLMs "perceive" each input variable in relation to the output. Particularly, given concerns that LLMs may perpetuate inaccuracies and societal biases embedded in their training data, our experiments using this framework quantitatively revealed both associations that contradict established medical knowledge and the persistence of scientifically refuted racial assumptions within LLM-encoded knowledge. By disclosing these issues, our framework can act as a red-flag indicator to support the safe and reliable application of these models.
Read the full abstract ▾
All authors Changho Han, Songsoo Kim, Dong Won Kim, Leo Anthony Celi, Jaewoong Kim, SungA Bae, Dukyong Yoon
Computation and Language
Multi-Perspective Evidence Synthesis and Reasoning for Unsupervised Multimodal Entity Linking
Mo Zhou, Jianwei Wang, Kai Wang et al. (+3 more)
Multimodal Entity Linking (MEL) is a fundamental task in data management that maps ambiguous mentions with diverse modalities to the
multimodal entities in a knowledge base. However, most existing MEL approaches primarily focus on optimizing instance-centric features and evidence, leaving broader forms of evidence and their intricate interdependencies insufficiently explored. Motivated by the observation that human expert decision-making process relies on multi-perspective judgment, in this work, we propose MSR-MEL, a Multi-perspective Evidence Synthesis and
Reasoning framework with Large
Language Models (LLMs) for unsupervised MEL. Specifically, we adopt a two-stage framework: (1) Offline Multi-Perspective Evidence Synthesis constructs a comprehensive set of evidence. This includes instance-centric evidence capturing the instance-centric
multimodal information of mentions and entities, group-level evidence that aggregates neighborhood information, lexical evidence based on string overlap ratio, and statistical evidence based on simple summary statistics. A core contribution of our framework is the synthesis of group-level evidence, which effectively aggregates vital neighborhood information by graph. We first construct LLM-enhanced contextualized graphs. Subsequently, different modalities are jointly aligned through an asymmetric teacher-student graph neural network. (2) Online Multi-Perspective Evidence
Reasoning leverages the power of LLM as a
reasoning module to analyze the correlation and semantics of the multi-perspective evidence to induce an effective ranking strategy for accurate entity linking without supervision. Extensive experiments on widely used MEL
benchmarks demonstrate that MSR-MEL consistently outperforms state-of-the-art unsupervised methods. The source code of this paper was available at: https://anonymous.4open.science/r/MSR-MEL-C21E/.
Read the full abstract ▾
All authors Mo Zhou, Jianwei Wang, Kai Wang, Helen Paik, Ying Zhang, Wenjie Zhang
Computation and Language · Cross-listed from cs.AI
ActuBench: A Multi-Agent LLM Pipeline for Generation and Evaluation of Actuarial Reasoning Tasks
Jan-Philipp Schmidt
We present ActuBench, a multi-
agent LLM pipeline for the automated generation and evaluation of advanced actuarial assessment items aligned with the International Actuarial Association (IAA) Education Syllabus. The pipeline separates four LLM roles by adapter: one
agent drafts items, one constructs distractors, a third independently verifies both stages and drives bounded one-shot repair loops, and a cost-optimized auxiliary
agent handles Wikipedia-note summarization and topic labelling. The items, per-model responses and complete leaderboard are published as a browsable web interface at https://actubench.de/en/, allowing readers and practitioners to inspect individual items without a repository checkout. We evaluate 50
language models from eight providers on two complementary
benchmarks – 100 empirically hardest multiple-choice items and 100 open-ended items scored by an LLM judge – and report three headline findings. First, multi-
agent verification is load-bearing: the independent verifier flags a majority of drafted items on first pass, most of which the one-shot repair loop resolves. Second, locally-hosted open-weights inference sits on the cost-performance Pareto front: a Gemma~4 model running on consumer hardware and a Cerebras-hosted 120B open-weights model dominate the near-zero-cost region, with the latter within one item of the top of the leaderboard. Third, MCQ and LLM-as-Judge rankings differ meaningfully: the MCQ scaffold inflates the performance ceiling, and Judge-mode evaluation is needed to discriminate at the frontier.
Read the full abstract ▾
Computation and Language
RADS: Reinforcement Learning-Based Sample Selection Improves Transfer Learning in Low-resource and Imbalanced Clinical Settings
Wei Han, David Martinez, Anna Khanina et al. (+2 more)
A common strategy in transfer learning is few shot fine-tuning, but its success is highly dependent on the quality of samples selected as training examples. Active learning methods such as uncertainty sampling and diversity sampling can select useful samples. However, under extremely low-resource and class-imbalanced conditions, they often favor outliers rather than truly informative samples, resulting in degraded performance. In this paper, we introduce RADS (Reinforcement Adaptive Domain Sampling), a robust sample selection strategy using reinforcement learning (RL) to identify the most informative samples. Experimental evaluations on several real world clinical
datasets show our sample selection strategy enhances model transferability while maintaining robust performance under extreme class imbalance compared to traditional methods.
Read the full abstract ▾
All authors Wei Han, David Martinez, Anna Khanina, Lawrence Cavedon, Karin Verspoor
Computation and Language
Wenhong Zhu, Ruobing Xie, Rui Wang et al. (+1 more)
Knowledge
distillation (KD) is a powerful paradigm for compressing large
language models (LLMs), whose effectiveness depends on intertwined choices of divergence direction, optimization strategy, and data regime. We break down the design of existing KD methods and present a unified view that establishes connections between them, reformulating KD as a reweighted log-likelihood objective at the
token level. We further propose Hybrid Policy
Distillation (HPD), which integrates the complementary advantages of forward and reverse KL to balance mode coverage and mode-seeking, and combines off-policy data with lightweight, approximate on-policy sampling. We validate HPD on long-generation math
reasoning as well as short-generation dialogue and code tasks, demonstrating improved optimization stability, computational efficiency, and final performance across diverse model families and scales. The code related to this work is available at https://github.com/zwhong714/Hybrid-Policy-
Distillation.
Read the full abstract ▾
All authors Wenhong Zhu, Ruobing Xie, Rui Wang, Pengfei Liu
Computation and Language
Construction of a Battery Research Knowledge Graph using a Global Open Catalog
Luca Foppiano, Sae Dieb, Malik Zain et al. (+3 more)
Battery research is a rapidly growing and highly interdisciplinary field, making it increasingly difficult to track relevant expertise and identify potential collaborators across institutional boundaries. In this work, we present a pipeline for constructing an author-centric knowledge graph of battery research built on OpenAlex, a large-scale open bibliographic catalogue. For each author, we derive a weighted research descriptors vector that combines coarse-grained OpenAlex concepts with fine-grained keyphrases extracted from titles and abstracts using KeyBERT with ChatGPT (gpt-3.5-turbo) as the backend model, selected after evaluating multiple alternatives. Vector components are weighted by research descriptor origin, authorship position, and temporal recency. The framework is applied to a corpus of 189,581 battery-related works. The resulting vectors support author-author similarity computation, community detection, and exploratory search through a browser-based interface. The knowledge graph is then serialized in RDF and linked to Wikidata identifiers, making it interoperable with external linked open data sources and extensible beyond the battery domain. Unlike prior author-centric analyses confined to institutional repositories, our approach operates at cross-institutional scale and grounds similarity in domain semantics rather than citation or co-authorship structure alone.
Read the full abstract ▾
All authors Luca Foppiano, Sae Dieb, Malik Zain, Kazuki Kasama, Keitaro Sodeyama, Mikiko Tanifuji
Computation and Language
The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models
Yilun Liu, Chunguang Zhao, Mengyao Piao et al. (+14 more)
Evaluating the
multilingual and multicultural capabilities of Large
Language Models (LLMs) is essential for their global utility. However, current
benchmarks face three critical limitations: (1) fragmented evaluation dimensions that often neglect deep cultural nuances; (2) insufficient
language coverage in subjective tasks relying on low-quality machine translation; and (3) shallow analysis that lacks diagnostic depth beyond simple rankings. To address these, we introduce GaoYao, a comprehensive
benchmark with 182.3k samples, 26
languages and 51 nations/areas. First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers (General
Multilingual, Cross-cultural, Monocultural) and nine cognitive sub-layers. Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective
benchmarks into 19
languages and synthesizing cross-cultural test sets for 34 cultures, surpassing prior coverage by up to 111%. Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs. Our findings reveal significant geographical performance disparities and distinct gaps between tasks, offering a reliable map for future work. We release the
benchmark (https://github.com/lunyiliu/GaoYao).
Read the full abstract ▾
All authors Yilun Liu, Chunguang Zhao, Mengyao Piao, Lingqi Miao, Shimin Tao, Minggui He, Chenxin Liu, Li Zhang, Hongxia Ma, Jiaxin Guo, Chen Liu, Liqun Deng, Jiansheng Wei, Xiaojun Meng, Fanyi Du, Daimeng Wei, Yanghua Xiao
Computation and Language
Markov reads Pushkin, again: A statistical journey into the poetic world of Evgenij Onegin
Angelo Maria Sabatini
This study applies symbolic time series analysis and Markov modeling to explore the phonological structure of Evgenij Onegin-as captured through a graphemic vowel/consonant (V/C) encoding-and one contemporary Italian translation. Using a binary encoding inspired by Markov's original scheme, we construct minimalist probabilistic models that capture both local V/C dependencies and large-scale sequential patterns. A compact four-state Markov chain is shown to be descriptively accurate and generative, reproducing key features of the original sequences such as autocorrelation and
memory depth. All findings are exploratory in nature and aim to highlight structural regularities while suggesting hypotheses about underlying narrative dynamics. The analysis reveals a marked asymmetry between the Russian and Italian texts: the original exhibits a gradual decline in
memory depth, whereas the translation maintains a more uniform profile. To further investigate this divergence, we introduce phonological probes-short symbolic patterns that link surface structure to narrative-relevant cues. Tracked across the unfolding text, these probes reveal subtle connections between graphemic form and thematic development, particularly in the Russian original. By revisiting Markov's original proposal of applying symbolic analysis to a literary text and pairing it with contemporary tools from computational statistics and data science, this study shows that even minimalist Markov models can support exploratory analysis of complex poetic material. When complemented by a coarse layer of linguistic annotation, such models provide a general framework for comparative poetics and demonstrate that stylized structural patterns remain accessible through simple representations grounded in linguistic form.
Read the full abstract ▾