Computation and Language
Improving LLM-based Autonomous Web Agents with Filtering
Zhitong Guo, Jing Yu Koh and Ruiyu Li
Autonomous web
agents, powered by Large
Language Models (LLMs), have garnered significant
attention for automating various web-based tasks with multi-step
reasoning and decision-making capabilities. An open research question in the development of these
agents lies in the format of the webpage input. Raw HTML source code, with its extensive and often irrelevant details, poses difficulties for LLMs with limited context windows. To address this challenge, we first reproduce baseline models such as GPT-3.5 and LLaMA-2-70B on the WebArena (Zhou et al., 2023)
benchmark, identifying common failure modes. We then propose two
retrieval strategies to filter out irrelevant context for LLM
agents. We develop DeBERTa-based and T5-based models that rank HTML elements by their relevance to the task. We fine-tune them on Mind2Web trajectory data and transfer them to WebArena. Experiments show that our DeBERTa-based model improves the success rate of the LLaMA-2-70B LLM
agent on WebArena from 1.97% to 2.96%. Moreover, we develop a zero-shot ColBERT-based retriever that is able to retrieve the ground-truth element with a recall of 0.52 on Mind2Web and 0.47 on WebArena.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Fusion Anything: A Generalized Multimodal Foundation Model
Huizi Cui, Zongbo Han, Chenggong Ding et al. (+6 more)
Making prediction with
multimodal data is widely used in diverse scenarios. Existing
multimodal fusion models, once deployed, can only handle predefined modalities (e.g.,
vision, text and audio) and single task, making it difficult to quickly adapt to new downstream applications. Therefore, a natural yet aggressive question arises - whether there exists a general
multimodal fusion model that can be applied to arbitrary modality combinations and arbitrary prediction tasks. We argue that a unified
multimodal fusion model should not depend on specific modalities and should instead encode transferable patterns of
multimodal correlation. To this end, we propose a simple and effective learning paradigm based on training on large-scale synthetic
multimodal datasets generated with Structural
Multimodal Causal Models (SMCMs), which formally characterizes the generative processes of real-world
multimodal data. Building on this framework, we propose the Fusion Anything Model (FAM), a foundation model for generalized
multimodal data fusion. By constructing large-scale synthetic
multimodal data with diverse correlation patterns, our model encodes transferable
multimodal correlations during training and activates appropriate associations through in-context examples during inference. Extensive experiments on 18 real-world
datasets spanning 12 modalities and 11 prediction tasks demonstrate that our model achieves competitive performance with specialized models without task-specific adaptation.
Read the full abstract ▾
All authors Huizi Cui, Zongbo Han, Chenggong Ding, Naichuan Xiao, Jialong Yang, Jingdong Chen, Guangyu Wang, Qinghua Hu, Changqing Zhang
Computation and Language · Cross-listed from cs.IR
Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inference
Xu Yang, Jiapeng Zhang, Yuxin Chen et al. (+4 more)
Sparse long-context inference requires efficient
token retrieval in both prefill and decode. Existing methods often use different
retrieval strategies for the two stages, preventing one
retrieval representation from being reused throughout inference. We propose Self-Indexing
Attention, a training-free framework built on a shared transform-domain sign-magnitude representation. The key signs provide a reusable
token-level index for grouped prefill selection and decode
retrieval, while the same representation remains compatible with external KV-cache compression without separate indexer metadata. This 1-bit index enables efficient
retrieval through bitwise operations widely supported by modern accelerators. At 5%
attention density, Self-Indexing
Attention remains close to dense
attention on LongBench and RULER and achieves up to 6.1x prefill and 10.3x decode
attention-operator speedups. Experiments with TurboQuant and DeepSeekV4-Flash further demonstrate compatibility with low-bit KV-cache compression and pretrained sparse-
attention indexers.
Read the full abstract ▾
All authors Xu Yang, Jiapeng Zhang, Yuxin Chen, Feiqiang Sun, Chengguang Xu, Feng Jin, Zhuo Tang
Computation and Language · Cross-listed from cs.AI
How Language Models Choose Sides: Internal Representations of Instruction Hierarchy
Enrique Balp-Straffon, Chih-Hao Hsu, Rushiraj Gadhvi et al. (+3 more)
We study how instruction-tuned LLMs arbitrate direct conflicts between system and user instructions. We introduce a
benchmark of 41 paired constraints with deterministic verifiers and evaluate eight models under matched baseline, conflict, and same-channel control conditions. Behaviourally, the models split into three regimes by System Authority Delta: hierarchy-respecting models use the system channel as an authority signal, anti-hierarchy models follow the system less often than their same-channel baseline predicts, and no-effect models show little channel sensitivity. Llama-3.1-8B is the strongest anti-hierarchy case in our suite, following the system in only 0.10 of conflict trials. We use this behavioural failure case to ask whether user-preferring arbitration reflects the absence of an internal conflictresolution signal. It does not: on Llama-3.1-8B, the conflict outcome is linearly decodable from residual-stream activations at 0.97 balanced accuracy, 17 percentage points above a metadata-only baseline, with analogous signals on Qwen2.5-7B and gpt-oss-20b. Steering with a layer-12 mean of four per-conflict logistic-regression directions raises genuine system compliance from 0.132 to 0.530, while directions selected mainly for pooled separability steer poorly. User-preferring conflict resolution can therefore coexist with a readable internal arbitration signal, and successful intervention depends on the geometry of the readout rather than probe accuracy alone
Read the full abstract ▾
All authors Enrique Balp-Straffon, Chih-Hao Hsu, Rushiraj Gadhvi, Sunishchal Dev, Callum Stuart McDougall, Anusha Mujumdar
Computation and Language
Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing
Israt Moyeen Noumi, Tarannum Ahmed Nowshin, Md. Mehedi Hasan Nipu et al. (+3 more)
Agentic security uses large-
language-model (LLM)
agents to plan, dispatch, and interpret security tools. As these systems move from demonstrations to deployed products, practitioners repeatedly encounter the same operational failures. We systematize these failures through a hands-on evaluation of ten widely used static, dynamic, cloud, orchestration, and AI red-teaming tools for unattended pipelines. We introduce a four-dimensional Integration Friction Index that separates one-time engineering cost from recurring organisational, legal, and maintenance cost. We then derive quantitative regularities that explain recurring failure modes. Modelling an agentic security system as
stochastic LLM policies wrapped by a deterministic mediator, we show that long-lived sessions lose resident evidence with phase count, while short-lived sub-
agents extend the usable horizon according to the compression ratio between raw evidence and its summary. We show that a two-stage verdict cascade multiplies scorer likelihood ratios, but provides little benefit when scorer errors correlate. We show that treating unevaluable outcomes as attack failures biases downstream measurements toward evasive and severe responses. We formulate planner-versus-worker model routing as a knapsack problem and derive a closed-form execution cap for heavy-tailed tools, eta* = alpha v/c. Finally, we show why scope and budget enforcement cannot be delegated to system
prompts:
prompts do not constrain what actually executes. Inspectra, our implemented platform, serves as a worked instantiation, with mechanisms labelled shipped, partial, or planned, including those that did not work.
Read the full abstract ▾
All authors Israt Moyeen Noumi, Tarannum Ahmed Nowshin, Md. Mehedi Hasan Nipu, Mohammad Sakib Mahmood, Md. Jakir Hossain, M. F. Mridha
Computation and Language · Cross-listed from cs.AI
The Price of Thinking: Reasoning Effort as a Model-Specific API Contract
Yeabin Moon
API buyers purchase a dated contract, not a model name alone: the contract includes the requested and served model,
reasoning-effort term or its omission, output rail, service product,
prompt, and price schedule. We study the
reasoning-effort term through a registered paired contrast of Sonnet 5 with explicit high effort against the same model with effort omitted, using 30 AIME 2026 items and five calls per item. Every paid attempt was assigned one frozen terminal category, and inference resampled items while retaining their repeated calls. Mean delivered cost was \$0.01031 per call higher under the explicit-high contract than under the omitted contract [+\$0.00204, +\$0.01974]. The corresponding accuracy contrast was +0.0133 [-0.0267, +0.0467]; we did not detect an accuracy difference, and the interval permits a gain of up to 4.67 percentage points that this design cannot rule out. Cost per correct answer was \$0.08665 under the high-effort contract and \$0.07662 under the omitted contract, as registered point estimates. A dated contract census, Models-API metadata, and preregistered raw-response probes further documented model-specific omission semantics, including within a provider; claims remained at documentation grade when raw structure was indeterminate. The request registry, parser, terminal taxonomy, statistical plan, and analysis pipeline were frozen before outcomes were examined; the resulting claims are bounded to the model, task, and collection date studied.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
The Limits of Binding in Dual Encoders
Kin Ian Lo
Dual-encoder models such as CLIP score an image-caption pair by a single inner product of two independently computed unit vectors, and fail at binding, often scoring near chance when asked to distinguish "a red car and a blue dog" from "a blue car and a red dog". We give a mathematical account of when this failure is necessary and when it is contingent. Working within the ideal-encoder framework proposed by Kang et al., we first show the relevant axioms are satisfiable, so every impossibility must enter through an added, checkable hypothesis. We then prove three such obstructions. Depth: for recursive role-binding codes the swap margin obeys an exact law $m(D) = 2b^{-D}$ in the nesting depth D, with a finite-dimension version holding up to one explicitly flagged concentration estimate; the resolvable depth grows only logarithmically in the dimension and is single-digit at CLIP scale, the nesting depth of ordinary
language. Objective: architecture-free throttle
theorems showing that the contrastive objective's entire reward for binding is bounded by the rate at which training contrasts a caption against its own swap, a rate that vanishes at web scale, and that exactly reversed binding costs only that rate times the mean binding margin; both are verified in simulation. Geometry: a tight smoothness-binding frontier: the closer the two swap-related captions must embed to a shared paraphrase anchor, the smaller the binding margin can be, with an exact constant. Measuring its text-only diagnostic across 18 deployed text encoders, every model sits at roughly 25-35% of its ceiling, and the induced per-item ceiling tracks SugarCrepe's subset difficulty at r = 0.99. Binding failure in deployed dual encoders is thus not a dimension or smoothness limit today, but an incentive and code-structure limit, with a proved depth ceiling that remains once those are fixed.
Read the full abstract ▾
Computation and Language
LLMs Get Smarter from Targeted Synthetic Multilingual Data
Ishika Agarwal, Arkajyoti Charaborty, Tanner Sorensen et al. (+2 more)
Language-specific competency (LSC) is the phenomenon of a
language model performing better or worse depending on the
language of the
prompt. In other words, a
language model outputs different (and potentially incorrect) responses to the same semantic query when prompted in different
languages. Prior work attributes this to an internal misalignment of semantic representation across
languages. Currently, there are two main approaches to address LSC in the literature: (1) routing all queries through English, improving performance, but limiting
language expressivity to English; or (2) training on
language-balanced data, equalizing model performance across
languages, but reducing overall performance. In this work, we take a data centric perspective and introduce HOTFIXR: Hardness Optimized Training data For Improving X-Lingual
Reasoning. It is a data generation framework that uses models to probe and learn a student model's
multilingual weaknesses, and generates data to mitigate them. HOTFIXR can generate
multilingual synthetic training data that can improve
multilingual performance. We evaluate on three in-distribution tasks, three out-of-distribution tasks, and four out-of-distribution
languages. On average, HOTFIXR (1) improves in-distribution performance by 6.2%, (2) reduces catastrophic forgetting (induced by fine-tuning) on OOD tasks by 3.7%, and (3) on OOD
languages by 7.1%. Overall, as many real-world applications requires
multilingual LLMs, our work contributes to the efforts of making LLMs multilingually proficient. We will release code upon acceptance.
Read the full abstract ▾
All authors Ishika Agarwal, Arkajyoti Charaborty, Tanner Sorensen, Neha Gupta, Andreas Stolcke
Computation and Language
SEER: Long-Context Reasoning via Selective Visual-Text Compression
Jiawei Xu, Zhilin Zhai, Jinrui Fang et al. (+6 more)
Long-context
reasoning remains computationally expensive for large
language models due to the quadratic complexity of
attention over text
tokens. Visual-text compression offers a promising alternative by rendering text into images and processing them with
vision-
language models, often reducing
token usage. However, existing approaches apply uniform compression regardless of query relevance, potentially sacrificing precision where detailed extraction is required. We present SEER, a framework that learns to select query-relevant images through visual scanning and retrieve textual content only where needed, combining the efficiency of visual compression with the precision of text-based
reasoning. Through supervised fine-tuning on tool-interaction trajectories, SEER learns adaptive tool invocation for selection and
retrieval. Experiments on long-context
benchmarks show that SEER improves extraction precision through selective text
retrieval while retaining average
prompt-
token savings relative to full-text baselines. On LongBench, SEER achieves 51.11% average accuracy, outperforming the visual-text baseline Glyph-9B by 2.33 points and Qwen3-8B by 3.49 points. Code can be accessed at https://github.com/jiaweixu98/SEER
Read the full abstract ▾
All authors Jiawei Xu, Zhilin Zhai, Jinrui Fang, Ruohan Xu, Mingfei Lu, Yi Zhang, Guanchu Wang, Tianlong Chen, Ying Ding
Computation and Language · Cross-listed from cs.IR
Ask to Be Sure: Informative Interactions for Confident Multi-Turn LLM Recommendation
Cedar Site Bai, Zhenyu Liao, Duanshun Li et al. (+6 more)
Recent advances in large
language models (LLMs) have enabled their use as conversational recommender systems (CRS), demonstrating strong recommendation accuracy and natural dialogue. However, guiding multi-turn interactions to elicit user preferences effectively remains challenging. Existing approaches either use separate reinforcement learning
agents with templated interactions or optimize for interactivity judged by another LLM, without measuring how much useful information is actually gained. We propose a new approach that quantifies the effectiveness of each interaction by the reduction in the assistant's uncertainty, measured via
entropy over recommendations. We apply this
entropy reduction as a reward—without relying on ground-truth recommendations, which are often unavailable in real-world scenarios—to fine-tune the LLM, enabling strategic interaction generation. Empirical results with supervised fine-tuning (SFT) and direct preference optimization (DPO) on the INSPIRED and ReDial
datasets show that our method improves both recommendation quality and conversational efficiency.
Read the full abstract ▾
All authors Cedar Site Bai, Zhenyu Liao, Duanshun Li, Sheikh Sarwar, Huiyuan Chen, Yuan Chen, Changhe Yuan, Haiyang Zhang, Qilin Qi
Computation and Language
The Null Token Knows: Reducing Message-Free Hallucination in ASR and NMT
Kirill Borodin, Vasiliy Kudryavtsev, Ivan Viakhirev et al. (+1 more)
Modern encoder-decoder systems can produce fluent text even when their input contains no recoverable message. We study this failure in ASR and NMT through the models' reserved null
tokens, asking whether the score for ending generation already carries a usable abstention signal. Across
speech recognizers and translation models, we audit native null-
token scores and scalar logit shifts. In Whisper, we additionally probe decoder states and compare supervised row edits with conventional external gates. The evaluated models often expose a useful abstention signal, but stock decoding does not reliably act on it. Raising the null-
token score can sharply suppress fabrication, but aggressive intervention also deletes valid
speech or shortens legitimate translations. These findings turn the null
token into a diagnostic lens on
hallucination and motivate evaluating abstention methods by both suppression and deletion costs, rather than by
hallucination reduction alone.
Read the full abstract ▾
All authors Kirill Borodin, Vasiliy Kudryavtsev, Ivan Viakhirev, Grach Mkrtchian
Computation and Language
Aborted but Not Forgotten: KV-Cache Retention Breaks Rollback Consistency in Language Agents
Guijia Zhang and Harry Yang
Stateful
language agents assume a rejected branch can be taken back by clearing it from the application transcript. We show this breaks when the serving session retains key/value (KV) state across the logical abort: the model can continue attending to content the application believes it discarded. We formalize the missing guarantee as rollback consistency: a complete abort must restore the state the model attends, not just the transcript. The key failure is cross-layer: a correct logical rollback need not compose with retained inference state, and the gap can remain invisible to the application. To isolate cache effects from text effects, we introduce a same-
token/different-cache audit that holds decision-step
tokens identical while varying only whether the cached prefix is stale or rebuilt from committed state. Across seven open-weight families (3.8B-36B), retained KV alone flips a typed protected effect in 25 of 63 audited
cells, while attacker
tokens are absent from the served request in all 63; rebuilding the cache closes every
cell. The channel reproduces in an end-to-end session application, on the default Hugging Face
Transformers cache-reuse path, and under LangGraph time-travel, where verified logical rollback can still leave attended KV stale. Susceptibility varies across models, but the underlying attended-state integrity violation is structural. We rule out position and length confounds, generalize across protected effects, policy structures, and a cache-isolated Mixture-of-Experts model, and show that transaction-local cache restoration closes the channel without requiring a global cache flush. All headline results are deterministic and reproducible from released artifacts.
Read the full abstract ▾
Computation and Language
Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisation
Ashima Sood, Bryan Gardiner and Joan Condell
Jointly fine-tuning an LLM on meeting-summarisation corpora of widely varying size raises a question that prior work leaves confounded: when a domain-balanced training mixture helps, is the gain due to the distribution of
tokens across domains, or merely to the volume of data seen? We disentangle these factors by constructing balanced and natural (native-proportional)
token mixtures at matched
token budgets (2-32M) over five English meeting corpora, fine-tuning Mistral-7B with QLoRA, and evaluating per domain. Balancing redistributes quality, improving the data-scarce minority domains at a low cost to the data-rich ones. The trade favours balancing whenever the minority domains matter: their share under proportional allocation is fixed at 1-2% regardless of budget, so matching balanced quality on those domains requires far more total data. We further find that
pruning low-value transcript lines removes ~15% of
tokens from the conversational corpora at no measurable cost, and that balancing by
tokens is not the same as balancing by examples. Fine-tuning one model per domain is competitive only on the data-rich domains and falls below the zero-shot model on the data-scarce ones. A two-annotator study of 741 judge-labelled facts validates our fact-level evaluation. Together these results give practitioners a basis for deciding when to balance an imbalanced multi-domain mixture, and on what unit.
Read the full abstract ▾
Computation and Language
PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming
Marianne Menglin Liu, Leonid Boytsov, Daniel W. Peterson et al. (+13 more)
We present PLSQLBench, to our knowledge the first
benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based tests. Existing LLM evaluations largely target general-purpose code generation or declarative text-to-SQL, leaving procedural database programming underexplored. PLSQLBench contains 2,865 instances: 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns. The
benchmark combines complex schema-grounded tasks over enterprise-style Spider 2 databases, simpler schema-grounded tasks derived from Spider, and MBPP-derived procedural problems, covering varying levels of database grounding and procedural complexity. Experiments with eight LLMs reveal recurring difficulties in schema grounding, PL/SQL dialect fidelity, procedural control flow, exception handling, and cross-turn consistency. Tool-augmented LLM
agents improve performance on several schema-grounded evaluations, although substantial gaps remain. These results highlight procedural database programming capabilities not directly assessed by conventional code generation or text-to-SQL
benchmarks. Our code is available at https://github.com/oracle-samples/plsqlbench.
Read the full abstract ▾
All authors Marianne Menglin Liu, Leonid Boytsov, Daniel W. Peterson, Pramuditha Perera, Rongguang Wang, Sai Ashish Somayajula, Syed Hamza Rafique, Rohit Saini, Shubham Pathak, Sujeeth Bharadwaj, Tao Sheng, Graham Horwood, Fahad Shah, Ankan Bansal, Sujith Ravi, Dan Roth
Computation and Language · Cross-listed from eess.AS
Iterative Self-Learning for Expressive Text-to-Speech Synthesis
Nicholas Sanders, Gustav Eje Henter, Simon King et al. (+1 more)
Expressive text-to-
speech (TTS) systems that use explicit conditioning labels provide direct and interpretable control over expressive attributes, in contrast to reference-based or prompting-based approaches, but require labeled data. Obtaining these labels at scale is costly and time-consuming, yet no prior semi-supervised framework addresses this specific bottleneck. Existing semi-supervised TTS methods instead target scarcity of paired
speech-text data or transcriptions. To address the scarcity of expressive labels, we propose an Iterative Self-Learning (ISL) framework for expressive TTS, built on Invert-Classify, a classifier-free method that recovers discrete expressive labels by inverting a frozen generative model. The framework iteratively pseudo-labels unlabeled
speech using the current model, retrains on the combined labeled and pseudo-labeled data, and repeats, progressively refining label quality and synthesis. We validate on two expressive tasks, word-level prominence and utterance-level
emotion, across multiple low-resource data splits. We find that iterative refinement can improve pseudo-label accuracy over single-pass baselines. Furthermore, we observe that these improvements in pseudo-labeling of expressivity translate to gains in expressive label adherence and synthesis quality, confirmed by objective metrics and human listening tests. In the most data-scarce conditions, ISL-trained models outperform single-pass pseudo-labeling and further approach fully supervised performance, demonstrating that gradient-based ISL is an effective solution to expressive label scarcity in low-resource TTS.
Read the full abstract ▾
All authors Nicholas Sanders, Gustav Eje Henter, Simon King, Korin Richmond
Computation and Language · Cross-listed from cs.IR
Large language model-assisted discovery of cohorts from scientific literature
Moritz Sturm, Lisa M. Berg, Inken Berg et al. (+6 more)
Background: Planning multi-study analyses requires identifying cohorts with the relevant participants, phenotypes, and data modalities. This process commonly relies on prior knowledge, cohort catalogues, and manual literature searches. We developed a complementary question-driven framework that searches relevant scientific literature and extracts explicit cohort names. Methods: The framework first generates multiple PubMed queries from configurable vocabularies and templates and retrieves the resulting scientific literature automatically through the PubMed API. A large
language model then screens the retrieved titles and abstracts and extracts explicit cohort names using a
prompt tailored to the research question. The extracted names are deduplicated with human review. Configurable code,
prompts, and example outputs are available at https://gitlab.rz.uni-frankfurt.de/cap_molgenlab/literature-cohort-discovery. Evaluation: As a use case, we applied the framework to youth aggression genetics. From 5,400 generated PubMed queries, the framework retrieved 5,254 unique records and identified 188 candidate cohorts. Manual screening using predefined criteria, including participant age and genetic-data availability, retained 44 eligible cohorts. Automated LLM-based name extraction was within the agreement range of human annotators. We also searched four established cohort catalogues using the same research question. Their combined results contained 27 of the 44 eligible cohorts, while 17 were not returned by any cohort catalogue search. Conclusion: The framework converts research-question-specific vocabulary into screenable cohort inventories via a large, automated literature search. It can be adapted across populations, phenotypes, data modalities, and study designs, and provides a literature-based complement to curated cohort catalogues.
Read the full abstract ▾
All authors Moritz Sturm, Lisa M. Berg, Inken Berg, Harishny Sarma, Jasmin Hartmann, Denissa Girschik, Gemma Roig, Christine M. Freitag, Andreas G. Chiocchetti
Computation and Language
When Less Is Enough: Context Selection and Prompting Strategies for Bengali News Headline Generation
Muhammad Ashad Kabir, Kawsar Ahmed and Md. Osama
Large
language models (LLMs) have shown strong performance in text generation tasks, yet their effectiveness on headline generation remains sensitive to how input context is selected and presented. In this work, we investigate Bengali news headline generation as a document-level generation task that requires effective selection and presentation of salient contextual information from long-form articles. Using Gemini-2.0-Flash, Llama-3.3-70B, and GPT-4o, we systematically study the effects of context selection, prompting strategies, and in-context learning (i.e., few-shot) on the quality of headline generation. Our experiments show that providing the full article does not necessarily improve performance; instead, using selected lead paragraphs of the article can maintain, and in some cases improve, headline generation quality. We further compare Bengali Native Prompting (BNaP) and Cross-Lingual Prompting (XLP), and examine how each interacts with context-enriched
prompt templates incorporating auxiliary contextual cues. Results demonstrate that prompting strategies substantially influence generation quality: XLP often yields stronger performance, particularly when combined with contextual enrichment, but its benefits are model-dependent. Additionally, few-shot prompting substantially improves Gemini, with most of the gain obtained from a single demonstration, whereas Llama shows limited benefit from additional examples. Overall, our findings highlight that effective Bengali news headline generation depends more on context relevance and
prompt design than on increasing input length, offering practical insights for
multilingual and low-resource LLM applications.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
Large Language Models as Implicit Sociological Models: Reconstructing Voting Behaviour from Sociodemographic Profiles
Roman Neruda, Martin Bakoš, Josef Šlerka et al. (+3 more)
Large
language models (LLMs) trained on large-scale internet corpora encode extensive statistical regularities about social identities, attitudes, and political behaviour. This paper introduces and evaluates a methodological framework that leverages these latent representations to reconstruct aggregate voting behaviour from individual-level sociodemographic profiles. We operationalize LLMs as implicit sociological models by conditioning them on demographic descriptions, eliciting probabilistic turnout and party preferences, and aggregating individual outputs via a soft voting procedure. Using the 2021 Czech parliamentary election as a validation case, we demonstrate that contemporary LLMs reproduce official election outcomes with low mean absolute error, recover known political bloc structures, and align with independently established sociodemographic gradients. The contribution of this work is methodological rather than predictive: we show how LLMs can be systematically interrogated as compressed representations of social reality, offering a novel exploratory instrument for computational social science while clearly delineating its epistemic and ethical limits.
Read the full abstract ▾
All authors Roman Neruda, Martin Bakoš, Josef Šlerka, Vít Tuček, Petra Vidnerová, Gabriela Kadlecová
Computation and Language · Cross-listed from cs.CV
Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning
Xiaoyu Zhu, Xinke Deng, Suresh Taddewadikar et al. (+4 more)
Multimodal large
language models increasingly use visual chain-of-thought (Visual CoT) to reason about spatial, temporal, and
embodied environments. By generating intermediate
reasoning images, Visual CoT provides an intuitive mechanism for visual foresight but introduces substantial inference overhead, which is particularly problematic for proactive
video reasoning. We ask whether models can learn to think visually during training while
reasoning directly at inference. We introduce Internalized Visual Thinking (IVT), a post-training framework that jointly optimizes textual prediction and next-
embedding prediction over unlabeled
videos. Given a partially observed
video, IVT predicts latent representations of future frames together with the target textual answer, encouraging the model to capture motion, object transitions, interactions, and latent intent. At inference, IVT generates the answer directly without synthesizing or re-encoding future frames. We conduct controlled studies across target representations, decoder designs, prediction horizons, data mixtures, training curricula, and predictive objectives. IVT improves over direct-answer fine-tuning on all six evaluation settings while retaining the same inference pathway. Compared with explicit Visual CoT, IVT achieves comparable or better performance and reduces average end-to-end latency by more than 5x. Together, our findings suggest that explicit pixel-space generation at inference time, as used in visual chain-of-thought, may not be necessary for effective proactive
video reasoning. Predictive world modeling can be internalized during training to produce
multimodal reasoners that are both more accurate and substantially more efficient.
Read the full abstract ▾
All authors Xiaoyu Zhu, Xinke Deng, Suresh Taddewadikar, Arnab Kumar Mondal, Zhongyu Jiang, Ian Fasel, Joerg Liebelt
Computation and Language · Cross-listed from cs.RO
Scaling Manual-Grounded Appliance Manipulation with Data Synthesis and Unified Planning
Yuxing Long, Lei Kang, Ziyan Yu et al. (+8 more)
Operating household appliances requires long-horizon planning that is state-dependent and robust to disturbances, yet existing large models fall short, as no sufficiently diverse, task-oriented
dataset exists to support such planning. To bridge this gap, we propose MAGE, a scalable data synthesis pipeline that introduces a novel Hierarchical Appliance Graph (HAG) to automatically generate part grounding, long-horizon planning, and closed-loop recovery data from appliance manuals. With MAGE, we build UseAppliance, the first large-scale
dataset for manual-grounded appliance manipulation planning, spanning 22 appliance categories with 89K+ part annotations, 53K+ manipulation tasks, and 33K+ closed-loop adjustment steps. Built on UseAppliance, we develop AppliancePlan, an end-to-end model for manual-grounded appliance manipulation planning. On RealAppliance-Bench, AppliancePlan with only 7B parameters achieves over 10x the best baseline on open-loop planning and consistently outperforms state-of-the-art models across all tasks. Real-
robot experiments on six household appliances further confirm effective sim-to-real transfer, marking an important step toward general-purpose household robotics.
Read the full abstract ▾
All authors Yuxing Long, Lei Kang, Ziyan Yu, Yuzheng Gao, Bin Cheng, Jiyao Zhang, Xiaoqi Li, Haolin Yang, Dongjiang Li, Hui Shen, Hao Dong
Computation and Language · Cross-listed from cs.IR
Query Expansion Should Be Coordinated: Dense Expands, Sparse Anchors
Chunran Zhang
Retrieval-augmented generation (RAG) systems rely on
retrieval modules to ground large
language model (LLM) outputs. LLM-based query expansion enriches
retrieval with document-like passages, but evaluations of hybrid
retrieval often fuse fixed top-L prefixes of dense and sparse rankings. Because L controls cross-channel contributions and ranking access, it can alter measured expansion gains. We therefore evaluate complete-list effectiveness and record per-channel replay stopping depths required to certify the ordered top-K. This changes the design: because both rankings determine the fused result, their query constructions should be coordinated rather than designed independently. We present DESA (Dense Expansion and Sparse Anchoring), which shares generated references across channels but specializes their integration. Orthogonal residual expansion adds new semantic directions to the dense query, whereas score-product anchoring reorders the original sparse support without admitting expansion-only matches. The same references thus play complementary roles: Dense expands; Sparse anchors. Across seven BEIR
datasets, DESA improves nDCG@10 and Recall@20 over the unexpanded query by 3.82% and 2.38%, while reducing dense and sparse replay stopping depths by 36.90% and 36.56%.
Read the full abstract ▾
Computation and Language
MicroVerse: An Instrument for Measuring Self-Authored Identity Drift in Long-Horizon Multi-Agent Language-Model Simulations
Sky Ng, Brihi Joshi, Ishan Gupta et al. (+43 more)
Long-horizon, multi-
agent language model (LM) simulations are widely proposed for studying social behavior, yet instruments to measure whether persona-conditioned
agents maintain identity fidelity under sustained pressure are lacking. We present MicroVerse, a behavioral-science instrument that measures identity drift in generative
agents.
Agents carry an immutable "soul file" (core values, moral boundaries, personality, goals) and inhabit a resource-scarce 50 x 50 environment where water is a non-respawning survival constraint. Scarcity is operationalized via a per-tick existence-cost gradient. The eight-verb action space maps directly to moral boundaries (trade, talk, attack, scavenge). Using a three-layer
memory architecture,
agents periodically revise a mutable current identity against their immutable original soul via importance-triggered reflection. To mitigate survivor bias, MicroVerse decouples measurement from behavior using uniform longitudinal engine snapshots every N ticks alongside a forced-end snapshot of all living and dead
agents. Identity drift is scored offline using a paraphrase-aware, value-anchored, multi-register diff rather than raw cosine similarity. We evaluate the instrument via a controlled seed run (n = 25) and a reflection-threshold sweep (thresholds {40, 80, 150}) to determine if drift dynamics are gate artifacts or threshold-robust properties. We report two primary findings: (1) Anti-self-deception emerges unprompted as the single largest semantic category of identity modification (27 of 111 added boundaries, 24%). (2) The system is threshold-robust; lower gates accelerate and increase revision frequency but preserve drift direction. All empirical results are strictly preliminary existence proofs and effect shapes (one model, one seed per arm, n = 25) rather than statistical significance claims.
Read the full abstract ▾
All authors Sky Ng, Brihi Joshi, Ishan Gupta, Shirley Huang, Zonglin Di, Yun Shen, Qianfeng Wen, Yifan Simon Liu, Ruoqi Gao, Yilan Fan, Zhiwei Zhang, Muhammad Ahmed Mohsin, Yucheng Lu, Xiaoyi Liu, Heming Liu, Qianyu Zhu, Hanwen Xing, Zhengyang Shan, My Chiffon Nguyen, Guanghui Min, Jianheng Hou, Yunze Xiao, Keyang Xuan, Hannah Collison, Jintao Huang, Jiatong Li, Sankalp Jajee, Yunhan Zhao, Bing Hu, Xupeng Chen, Binghang Lu, Weihang Xiao, Aravind Mohan, Bolun Sun, Yunshu Wu, Yuanda Xu, Runyu Zhang, Zheyuan Deng, Xinchen Tan, Dianzhuo Wang, Yijun Wang, Yixuan He, Koutian Wu, Cheng Cheng, Xiaomin Li, Yuexing Hao
Computation and Language · Cross-listed from cs.AI
Schema-Agnostic Graph Reasoning Agent for Hybrid Knowledge Graphs
Marius Dragic, Ruben Ifrah and Alexandre Rio
Tool-calling LLM
agents navigate unfamiliar codebases with a handful of generic primitives for listing, reading and searching files (ls, cat, grep). A knowledge graph admits the same interface: listing neighbours, reading node content and searching descriptions are the same operations on a different substrate. Building on this correspondence, we present GRA, a Graph
Reasoning Agent that explores hybrid knowledge graphs, whose nodes are either textual concepts or relational tables, with seven generic tools, discovering everything domain-specific at run time. On UFK-M (Unified Factory Knowledge Model), an industrial
benchmark of 258 analytical questions whose gold answers are produced by executing validated SQL programs, GRA beats a full-context
agent by 5.1 pp (88.4% vs. 83.3%), while reading under a third of its input
tokens. A graph-free control shows the gain comes chiefly from selective agentic access rather than graph
topology, and that the effect depends on a model able to drive tools reliably. Seeing less, the
agent answers better: selective
navigation over a structured substrate beats exhaustive context.
Read the full abstract ▾
Computation and Language
A Cognitively Motivated Multidimensional Framework for Evaluating Metaphor Explanations
Ana Naveriani, Jakob Suchan, Stefano Zoia et al. (+3 more)
Current evaluation of metaphor explanations relies mainly on holistic quality ratings, revealing little about how explanation quality is structured or where human judgments agree and diverge. We introduce a cognitively motivated framework that decomposes metaphor explanation quality into six theoretically grounded dimensions. In a dense annotation study (11,200 ratings), we find that: {\bfseries(i)} explanation quality is genuinely multidimensional; {\bfseries(ii)} annotator disagreement is systematic rather than random; and {\bfseries(iii)} the six dimensions collapse into a shared cluster and two independent axes of judgment. An exploratory feasibility study further shows that a standard automatic evaluation pipeline can recover parts of this structure, predicting the most discriminative dimensions well while its errors correlate human (dis)agreement. Together, these results suggest that multidimensional evaluation offers richer diagnostic insight than holistic ratings, and that automatic evaluators for open-ended generation tasks should be judged on how well they preserve the structure of human judgment.
Read the full abstract ▾
All authors Ana Naveriani, Jakob Suchan, Stefano Zoia, Mehul Bhatt, Antonio Lieto, Gian Luca Pozzato
Computation and Language
QuantumPhaseNet: A Gauge-Covariant Geometric and Quantum-Spectral Theory of Semantic Concept Hierarchies with Prototype Validation of a Classical Quantum-Inspired Model
Kiyotaka Kasubuchi and Kazuo Fukiya
We present QuantumPhaseNet, a gauge-covariant
geometric and
quantum-spectral extension of
Transformer representations. Context-dependent semantic states are modeled as complex amplitudes; a covariant phase rate induces a semantic wavelength used as a proxy for conceptual scale; and low-frequency graph modes define a document-level discourse direction. The theoretical part establishes local gauge invariance, unitarity of the
quantum block, boundedness and conditional stability of WavePhase
Attention, and a calibratable
hallucination-risk formulation. We also implemented a fully offline Validation Studio for the classical
quantum-inspired pipeline in Section 14.1 and evaluated the five research questions in Section 16.1 on its built-in synthetic setting (n=240, observation noise 0.22,
circuit noise 0.08, five seeds). RQ1 yielded a wavelength-hierarchy Spearman correlation of 0.852 versus 0.707 for the baseline, 87.3% direction accuracy, and AUC 0.953. RQ2 achieved discourse
alignment 0.933 versus 0.589 and 41.2 versus 16.2 paragraphs before drift. RQ3 achieved AUROC 0.881 versus cosine 0.765 and phase-shuffle 0.536. RQ4 achieved error-detection AUROC 0.854 versus
entropy 0.634, with Brier 0.150 and ECE 0.098. RQ5 did not show
quantum advantage: target probability and end-to-end cost efficiency were 25.5% and 0.107, compared with 70.7% and 0.707 for the Chebyshev classical approximation. These results provide initial synthetic evidence for the classical
quantum-inspired components, but not external validity or unconditional
quantum speedup.
Read the full abstract ▾
Computation and Language
Miyu Yamada and Yuki Arase
Hallucinations remain a major obstacle to the reliable use of large
language models (LLMs) in conditional text generation. Existing methods primarily assess the factuality of an entire generated text, providing limited insight into which output spans are hallucinated or how they relate to the input. We introduce the task of
hallucination span detection with input-side evidence
alignment, which jointly identifies hallucinated spans and aligns output
tokens with the corresponding input evidence. Our approach is based on the observation that faithful output
tokens are predictable from the input, whereas hallucinated
tokens are not. We therefore train an encoder-based model to predict masked output
tokens from the input representation, using prediction confidence for
hallucination detection while naturally producing
alignments to the input. Experiments show that the proposed method effectively detects hallucinated spans and identifies meaningful input-side evidence. Human evaluation confirms the quality of the predicted
alignments.
Read the full abstract ▾
Computation and Language
Using the Mimi codec for metalinguistic representations
Artem Saloev, Erin Pacquetet and Nicolas Ballier
In this paper, we focus on the dictionary of 2048
tokens used in Mimi semantic
token codebook, the neural codec of the Moshi
language model. We show that the ABX experiment carried out with Mimi fails to capture the mapping of the semantic
tokens to phone realisations. By realigning Mimi representations to the TIMIT corpus transcriptions, we show that the 2048
tokens IDs of the semantic codebook map to quadphone, triphone, biphone, phone and subphone realisations.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
KV-Rescue: Recovering Reasoning Language Model KV Eviction Loss via Stepwise Interleaving
Minsoo Cheong, Woosang Lim, Vincent-Daniel Yun et al. (+1 more)
KV-cache eviction caps the
memory cost of long
reasoning traces but is inherently lossy because the model decodes from a partial view of its history. Under aggressive budgets, this not only lowers accuracy but can also cause runaway degeneration, where the model produces incoherent or repetitive
tokens until reaching the length limit. We characterize much of this loss as an information gapf caused by missing context, rather than a capability gap caused by limited model capacity. An evicted 7B model and a full-context 1.5B model make complementary errors, and an oracle choice between their answers recovers 79% of the accuracy gap to the full-KV 7B model. Based on this observation, we propose KV-Rescue, a training-free inference framework that bridges the information gap introduced by KV eviction using a lightweight full-context helper. KV-Rescue interleaves
reasoning steps from the two models into a shared trajectory. An online detector uses
entropy and compressibility to terminate the generation of incoherent or repetitive base-model candidates early. Across five math
benchmarks with Qwen2.5-Math 7B and 72B, KV-Rescue recovers an average of 87% of the accuracy lost to eviction at eviction budget B=64. A decode-cost analysis further shows that preventing runaway degeneration cuts base-model
token generation by 43% on average.
Read the full abstract ▾
All authors Minsoo Cheong, Woosang Lim, Vincent-Daniel Yun, Sungjoo Yoo