Computation and Language
Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
Xinke Tong, Xuanming Zhang, Tianyi Tang et al. (+10 more)
Do Large
Language Models (LLMs) possess genuine structural
reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding numerical precision and multi-step logic over long contexts, is an ideal testbed. Existing
benchmarks fail to capture real-world industrial complexity, predominantly relying on multiple-choice questions or single-hop QA over cropped tables while ignoring intricate cross-statement dynamics and temporal de-cumulation. To bridge this gap, we introduce FinIndices, a large-scale
benchmark evaluating data-processing fidelity over uncropped financial statements (up to 32K
tokens). Utilizing an automated synthesis pipeline with
adversarial traps, FinIndices encompasses Single-Index computation and Table-Index tabulation to test complex domain, temporal, and caliber
reasoning. Our evaluation reveals two severe LLM vulnerabilities. First, a "Knowledge Bottleneck": despite memorizing formulas during pre-training, models demonstrate fragile pattern matching. Removing explicit formula hints causes performance to collapse (e.g., Gemini-3.1-Pro drops from 70.70% to 38.22% on table tasks), exposing fatal flaws in temporal de-cumulation and stock-flow caliber mismatch. Second, a "Structural Bottleneck": the intense cognitive load of generating multi-metric, multi-period tables actively drains
reasoning capacity. Under structural pressure, LLMs that flawlessly execute isolated derivations regress to shallow heuristics, such as fetching incorrect adjacent columns or substituting deep accounting adjustments with lazy literal arithmetic. Finally, Supervised Fine-Tuning (SFT) yields substantial zero-hint gains (+8.54% Single, +3.82% Table), validating that structured logic can be partially restored via data-centric
alignment.
Read the full abstract ▾
All authors Xinke Tong, Xuanming Zhang, Tianyi Tang, An Yang, Jiatu Hu, Guojie Lin, Zhenzhen Shi, Lingfeng Zeng, Boyu Yang, Bing Zhao, Hu Wei, Lin Qu, Dayiheng Liu
Computation and Language
Sympathetic Framing: Evaluating AI Alignment across Sociodemographic Groups
Haran Shani-Narkiss, Michael Fire and Oren Tsur
Large
Language Models (LLMs) are increasingly shaping how we consume information and form our worldview. This raises concerns beyond bias in AI: do LLMs grasp the emotional nuances conveyed via textual framing? In this work, we empirically evaluate how well an array of LLMs aligns with human emotional perception. Considering news headlines covering political and geopolitical conflicts, both human participants (n = 3011, a representative sample of the U.K. adult population, via a YouGov survey) and seven LLMs answered whether headlines evoked sympathy for a specified side in a conflict. We find that the correlation between AI and human evaluations varies across models, ranging from very high (0.789, GPT-5.2) to medium (0.4 ,Mistral Large 2512). Crucially, the leading models are broadly aligned with human judgments across all demographic subgroups, including age, gender, level of education, prior geopolitical knowledge, and participants' predispositions regarding the conflict, although there are statistically significant differences between groups. This research, with its robust design and large, demographically diverse
dataset, offers the most comprehensive evaluation of LLMs' comprehension of news framing to date. Findings highlight an important, often-ignored aspect of differential
alignment: even when aggregate performance is high, AI
alignment is not universal – it may correspond differently with demographic features and cultural norms. Considering or ignoring the need for differential
alignment may therefore have significant implications for the development of ethical and useful AI systems.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Open Your Model's Eyes: Video and Context-Aware Multimodal Backchannel Prediction
Min-Jae Kim, Jun-Yeong Moon, Mujeen Sung et al. (+1 more)
Backchannels, which signal listener states like empathy and understanding, are fundamental to natural human interaction. However, current approaches rely solely on audio and text. This omits crucial visual cues, such as facial expressions and gestures, as well as broader conversational contexts, which are necessary for accurate prediction. In this paper, we introduce Context-Aware
Multimodal Alignment for Backchannel Prediction (CAMA-BC), a novel framework that leverages visual information through Multi-Layer
Multimodal Alignment (MMA). Our
alignment process comprises two stages. First, Context
Alignment (MMA-CA) utilizes unlabeled dialogues with
videos to capture conversational contexts. Next, Backchannel
Alignment (MMA-BA) fine-tunes the representations specifically for backchannel prediction. Experimental results show that CAMA-BC significantly outperforms both existing methods and simple
multimodal baselines, with particular effectiveness in recognizing complex backchannels such as empathy.
Read the full abstract ▾
All authors Min-Jae Kim, Jun-Yeong Moon, Mujeen Sung, Gyeong-Moon Park
Computation and Language · Cross-listed from cs.RO
Progress Reward Modeling for Robotic Learning: A Comprehensive Survey
Jianshu Zhang, Keliang Wu, Haoran Lu et al. (+8 more)
Robotic learning takes place in dynamic environments with large behavior spaces. A terminal success signal only tells the
robot whether the task is completed. It does not explain whether the current behavior is making progress, remaining unchanged, or undoing earlier progress. For this reason, recent studies have increasingly explored progress rewards that provide feedback during task execution. However, the current literature lacks a shared framework. Existing methods use different observations, goal specifications, output signals, supervision sources, and evaluation protocols. This makes it difficult to compare them and understand what their results actually validate. In this survey, we provide a unified view of progress reward modeling for robotic learning. We organize the field in three connected steps. We first study the interface of a progress model. This defines the problem from the outside by asking what information the model receives and what form of progress signal it produces. We then move inside the model and study the methods used to construct this signal. This reveals the different assumptions and mechanisms behind progress estimation and reward generation. Finally, we examine the data and
benchmarks that support these methods. This shows how progress supervision is obtained and what different evaluations actually measure. Together, these three perspectives connect what a progress model is, how it is built, and how its quality is validated. We further summarize the main limitations of current approaches and discuss future research directions.
Read the full abstract ▾
All authors Jianshu Zhang, Keliang Wu, Haoran Lu, Anbang Liu, Ce Zhang, Weijie Yin, Chengxuan Qian, Xiyuan Yang, Zhenyu Pan, Guo Ye, Han Liu
Computation and Language · Cross-listed from cs.LG
Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning
Jian Hu, Huiying Li, Hao Zhang et al. (+8 more)
Agentic reinforcement learning requires rapid experimentation with
agents and learning
algorithms, yet large policies and long,
multimodal trajectories demand substantial distributed infrastructure. We present MOLT, a lightweight, PyTorch- and Hugging Face-native framework that brings these goals together through four contributions. MOLT combines direct loading of Hugging Face models with experimentally validated trillion-parameter scalability in approximately 9.2K lines of framework code. Unified OpenAI- and Anthropic-compatible interfaces integrate existing
agents with automatic handling of context compaction. Fully asynchronous training overlaps
agent rollouts and policy optimization, accommodating variable
agent execution times. Distributed experience storage removes centralized rollout-
memory bottlenecks for long,
multimodal trajectories. We experimentally validate the complete RL training pipeline on a one-trillion-parameter policy and demonstrate sustained learning with a 30B mixture-of-experts
agent, establishing MOLT as a lightweight foundation for large-scale agentic RL research.
Read the full abstract ▾
All authors Jian Hu, Huiying Li, Hao Zhang, Binfeng Xu, Yifan Zhang, Shaokun Zhang, Hemil Desai, Michael Demoret, Pavlo Molchanov, Jan Kautz, Yi Dong
Computation and Language · Cross-listed from cs.AI
Refusal-Gated Decoding: Preserving Refusal Behavior Under High-Temperature Sampling
Phillip Howard, Xin Su, Allen Roush et al. (+3 more)
Recent advances in truncation-based sampling have helped mitigate drawbacks of high-temperature sampling such as neural text degeneration, thereby enabling greater diversity without sacrificing coherence. However, increasing the
entropy of the
token probability distribution via high temperatures has also been shown to weaken the model's refusal response. Existing solutions for maintaining the refusal behavior of LLMs either replace the model's own refusal decision with a separate safety classifier or alter its output distribution for every
prompt. To address this gap, we propose refusal-gated decoding (RGD): an efficient sequential decoding approach which preserves a model's greedy decoding refusal response at high temperatures and samples all other
prompts from its exact direct high-temperature distribution, while incurring minimal additional latency. RGD runs a short greedy probe that reuses the
prompt's KV cache and exits as soon as it becomes incompatible with a learned set of refusal prefixes; it returns the greedy response if the probe remains compatible and otherwise discards the probe and samples from the original
prompt. Across seven models and three
benchmark datasets at T=2.0, RGD raises greedy-refusal preservation from 91.9% under direct sampling to 98.3% on average while adding only 2.2-4.3% to the median per-request latency of non-refusals across temperatures. Unlike
prompt-screening baselines which route many greedy non-refusals to greedy decoding, RGD keeps at least 98.1% of greedy non-refusals on unchanged high-temperature sampling, thereby preserving the model's natural high-temperature sampling behavior. We also propose a residual-stream variant of our method which lowers this latency overhead to at most 0.5% with comparable
prompt routing accuracy. Our work shows that unlocking greater diversity via high-temperature sampling need not erode a model's refusal behavior.
Read the full abstract ▾
All authors Phillip Howard, Xin Su, Allen Roush, Manikandan Ravikiran, Runyan Tan, Amir Abdullah
Computation and Language
Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles
Donghwan Kim
Majority voting over LLMs is widely assumed to benefit from diversity, and diversity measures are used to choose which models to combine. We ask whether five such measures track diversity or mainly re-express capability,
auditing them as predictors of majority-vote gain over the best member across 31,900 subsets of 30 LLMs on MMLU-Pro (29 on TruthfulQA) under explicit capability controls. Three findings emerge. First, latent complementarity is ubiquitous: oracle gain is positive in 100% of subsets, yet simple voting beats the strongest member in only 9.98% of all canonical size-3 subsets (18.71% with held-out best selection); the pooled size-2-4 rate is 1.27%, partly reflecting deterministic even-size voting behavior. Second, a joint-correctness proxy (strict diversity) is nearly collinear with one minus mean accuracy (size-3 Spearman rho = +0.991 / +0.988); raw diversity-gain associations are strongly capability-entangled and, with one exception, unstable under control. Third, three linear contingency-table statistics are algebraically non-separable; after capability control, the empirically stable remainder is a modest residual pairwise co-failure association in which more shared error corresponds to lower gain. This direction is robust, but its magnitude is configuration-dependent. Joint rawspace linear regressions treating strict diversity, disagreement, and double-fault as independent predictors are rank-deficient by construction.
Read the full abstract ▾
Computation and Language
Rushes: A Human Preference Dataset for Pluralistic Alignment
Michael Xu, Jorge Leandro, Sudha Rao et al. (+5 more)
We introduce Rushes, a
dataset and
benchmark for studying revealed human engagement preferences in interactive narrative environments. Rushes is collected through a game interface where users interact with AI-generated branching narratives and select one choice from a small, explicit candidate set at each decision point. Each interaction logs the full candidate set, the user's choice, and the evolving narrative context, yielding time-ordered trajectories with persistent user-level identifiers. Rushes contains 44,226 decision events from 8,167 unique users across six games, capturing sequential, personalized engagement behavior rather than static judgments. We show that user choices exhibit structured, non-random patterns, quantified by a low choice
entropy relative to a uniform baseline. We position Rushes as a diagnostic
benchmark for pluralistic
alignment and demonstrate a robust Engagement Gap: state-of-the-art LLMs, including GPT-5, fail to outperform simple baselines. While classical
Matrix Factorization (SVD) captures measurable personalized signal (37.7%), frontier LLMs (34.23%) struggle to even match the Popularity Baseline (36.4%) on event-level choice prediction. This gap suggests that single, population-level objectives, like those used in modern RLHF, appear insufficient to capture heterogeneous, context-dependent engagement signals. As a result, even highly capable models default to majority preferences rather than adapting to individual trajectories. We release Rushes to support research into pluralistic
alignment and sequential decision-making in generative systems. The full code for the platform and
dataset will be available here: https://github.com/microsoft/rushes
Read the full abstract ▾
All authors Michael Xu, Jorge Leandro, Sudha Rao, Weijia Xu, Nebojsa Jojic, Gabriel DesGarennes, Chris Quirk, Bill Dolan
Computation and Language · Cross-listed from cs.LG
GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries
Miguel P. Bento and João F. Seabra
Transformers are known to have internal continuous
symmetries that leave outputs invariant, while modifying quantization. GaugeQuant leverages this in-training by introducing a LogSumExp term to the loss that breaks the
symmetries, thus selecting a basis that minimizes activation outliers. A stop-gradient operator ensures that only rotation
matrices are updated, yielding the
language modeling objective completely unaltered. Our requires no specific calibration data, no quantization simulation, and adds negligible training overhead. With the LLaMA-2 7B model under W4A4 quantization with group size 128, perplexity drops from 8.22 to 6.73, competing with post-training methods that require frozen models and calibration
datasets. Under W4A16, perplexity drops from 11.16 to 5.45. Code is available at https://github.com/MPedraBento/gauge-quant.
Read the full abstract ▾
Computation and Language
REGARD: Regional Affective Differences in Large Language Models
Andrei Chetvergov, Alexander Evseev, Mikhail Solovev et al. (+5 more)
Large
language models trained and aligned within different linguistic and regional ecosystems may frame the same political, cultural, and geopolitical entities in different ways. Such differences are often evaluated through sentiment, favorability, or stance, reducing model attitudes to a single positive-negative axis. We introduce REGARD, a study of what drives affective framing differences across LLMs on post-Soviet entities using target-directed Valence-Arousal-Dominance profiling. We query 19 models on 500 region-specific targets, score their responses with two independent LLM judges, GPT-4o-mini and Qwen3.6-35B-A3B, and validate the measurements on a 300-item human-annotated subset. Post-hoc Ward-linkage clustering of all 19 models by affective and response-behavior profiles yields three behavioral clusters that cut across model origin, family, and parameter count. Generic-answer rate is strongly associated with lower arousal (r = -0.81) and with cluster placement: models that deflect evaluative
prompts with templated responses cluster together at low arousal regardless of origin. These findings show that VAD profiling captures emotional intensity, a dimension of affective framing that is largely invisible to conventional sentiment-based evaluation.
Read the full abstract ▾
All authors Andrei Chetvergov, Alexander Evseev, Mikhail Solovev, Timofei Sivoraksha, Stepan Ukolov, Valeriia Kuschenko, Maria Chistyakova, Sergey Bolovtsov
Computation and Language · Cross-listed from cs.AI
NVIDIA-labs OO Agents: Native Python Object-Oriented Agents
Paul Furgale, Severin Klingler, James Nolan et al. (+12 more)
Traditional
agent development is split across
prompt templates, tool schemas, callback code, and workflow graphs. We present NVIDIA Object-Oriented
Agents (NOOA), a model-agnostic Python framework for building reliable AI
agents. NOOA takes a simpler approach: an
agent is a Python object. Its methods are the actions the model can take, fields are its state, docstrings are its
prompts, and its type annotations are contracts. A method whose code body consists of "..." is completed at runtime by an LLM-driven
agent loop, while methods with normal bodies remain standard deterministic Python. This gives developers and
agents the same interface, so
agent behavior can be tested, traced, refactored, and improved just like other software. This paper makes three contributions. (1) We present the
agent-as-a-Python-object programming model and the design principles behind it. Where Python has existing abstractions, we adopt them directly.
Agent-specific capabilities–context, events, state rendering, long-term
memory, and validated LLM loops–are exposed through simple Pythonic APIs, so both developers and
agents share one familiar programming model. (2) We identify six model-facing ideas that NOOA is, to our knowledge, the first to combine on a single surface: typed input/output, pass-by-reference over live objects, code as action, programmable loop engineering, explicit object state, and model-callable harness APIs for context and events. We find the community already converging on several of these ideas–often as experimental or partial features–and present the comparison to encourage further adoption. (3) We demonstrate that current models use this interface effectively, both in targeted capability tests and on agentic and
reasoning benchmarks such as SWE-bench Verified and Terminal-Bench 2.0 and ARC-AGI-3.
Read the full abstract ▾
All authors Paul Furgale, Severin Klingler, James Nolan, Matt Staats, Gaia Di Lorenzo, Elisa Martinez Abad, Christian Schüller, Razvan Dinu, Alessio Devoto, Pascal Berard, Gal Kaplun, Elad Sarafian, Riccardo Roveri, Leon Derczynski, Ricardo Silveira Cabral
Computation and Language
Learning to Detect UI Principle Violations via Reinforcement Learning
Nishi Mehta, Swathi Alse, Himani Kumawat et al. (+2 more)
Small
language models and coding
agents increasingly generate web front-end code, yet their outputs are typically evaluated primarily for functional correctness. A generated interface may compile, render, and pass unit tests while still violating established interface quality principles, including accessibility barriers, deceptive design patterns, poor visual hierarchy, and excessive decision complexity. Existing
auditing approaches face a trade-off between cost, coverage, and scalability: expert human review provides rich judgment but is slow and expensive; frontier
vision-
language models offer broader
reasoning capabilities but remain costly to deploy at scale; and rule-based tools such as axe-core and Lighthouse are inexpensive but primarily capture mechanically checkable accessibility issues. We investigate whether a lightweight
vision-
language model can serve as an effective critic for generated interfaces. We unify 19 interface-quality principles from three complementary sources of HCI knowledge: WCAG 2.2 accessibility standards, deceptive design taxonomies, and established theories of perception, cognition, and interaction. To train this critic, we construct a verified
dataset of approximately 10,000 generated web pages by synthetically injecting known violations into clean, LLM-generated Tailwind pages. Continued reinforcement learning on a 4B
vision-
language model improves micro-F1 from 36% to 84%, with 13 of 19 principles exceeding 80% F1. The resulting critic can audit generated interfaces, filter low-quality interface training data, and provide a reward signal for design-aware code generation. We release our data-generation recipe and injection/verification
prompts to support reproducible evaluation and future work on scalable interface-quality assessment.
Read the full abstract ▾
All authors Nishi Mehta, Swathi Alse, Himani Kumawat, Yue Yu, Pratik Jayarao
Computation and Language
From Agent Failures to Text Policies: What Works and What Breaks
Jaideep Ray and Ankit Goyal
TextGrad improves
language-model systems by revising text from feedback. Its core thesis is that natural-
language feedback can act as a gradient for optimizing text components without changing model weights. Applying it to
agents is harder because feedback arrives only after a sequence of actions, making it difficult to identify which decision caused failure. We study this problem by separating the ability to follow a useful policy from the ability to learn that policy from experience. Our main finding is a clear gap between these two abilities. Human-written policies improve two frozen 7B
agents on TextWorldExpress by 5.0 success points, showing that useful policy text exists. However, policies generated from
agent trajectories do not reliably outperform fixed prompting, even with richer traces, counterfactual evidence, or iterative GEPA search. The main challenge for
agent-level TextGrad is therefore not executing textual policy updates, but reliably generating and selecting them from experience.
Read the full abstract ▾
Computation and Language
Frontier Financial Judgement: Can agents tell what might move a stock?
Joshua Harris
We introduce Frontier Financial Judgement, a challenging new
benchmark developed in collaboration with professional equity analysts to assess
agents' ability to replicate expert human judgements. Rapidly identifying new information, evaluating its implications and determining its valuation impact is one of the most time-consuming and challenging aspects of real-world equity coverage. This is becoming ever more difficult and important as AI rapidly increases the quantity of new information to process. The strongest
agent we evaluate on Frontier Financial Judgement matches all expert labels in only 52.4% of cases. We also find significant divergence in estimated false-positive rates among frontier
agents, ranging from ~1% for GPT-5.6 Sol to ~32% for Claude Sonnet 4.6. To construct the
benchmark and make it representative of real-world settings, we combine human-designed and labelled synthetic articles with live news articles and historical documents, creating 656 items for assessment. The resulting task requires
agents to distinguish genuinely new, valuation-relevant financial information from stale, immaterial or misleading news under realistic conditions. We find substantial trade-offs among
agent accuracy, cost, false positives and reliability that continue to hinder the reliable deployment of news-flow filtering in practice.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms
Yichuan Liu, Daniel Cummings and Nick Vadlamudi
Large
Language Models (LLMs) have demonstrated strong capabilities in code generation and
reasoning, yet their ability to perform temporal
reasoning over digital waveform data remains largely unexplored. Although
reasoning over digital waveforms is a critical bottleneck in design verification, existing
benchmarks primarily evaluate hardware description
language (HDL) code generation and use waveforms only as supplementary context. This paper presents WaveformQA, an open-source question-answering
benchmark for evaluating LLM temporal
reasoning over digital waveforms. The
benchmark comprises 360 questions with programmatically generated ground truths across eight categories of varying difficulty, including questions targeting multi-signal correlation and event ordering. Waveforms are generated from open-source design implementations, ensuring reproducibility and grounding the
benchmark in real hardware behavior. Evaluation of frontier LLMs reveals that while models achieve reasonable accuracy on simple queries, performance degrades due to context window limitations and
reasoning difficulties on complex temporal and multi-step questions. In addition, we show that an event-time JSON representation of waveforms improves LLM
reasoning accuracy versus the standardized value change dump (VCD) format. The open-source framework supports extending to new question categories and importing new waveform sources, enabling researchers to rapidly prototype temporal
reasoning experiments.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.DB
Demonstrating GenDB: Instance-Optimized and Customized Query Processing Code Generation via LLM Agents
Jiale Lao and Immanuel Trummer
Traditional query processing engines require continuous development and extensions to support new techniques and user requirements, and in some cases, entirely new systems must be built from scratch. However, these engines are difficult to extend due to their internal complexity, and building new systems demands significant engineering effort and cost. To address this, we demonstrate GenDB, a generative query engine that shifts query processing from manually engineered systems to query processing code generation driven by Large
Language Models (LLMs). An early prototype of GenDB uses LLM
agents to generate instance-optimized query execution code tailored to specific data, workloads, and hardware resources. This prototype suits offline code generation for repetitive, templated queries, since the upfront generation cost amortizes over many executions and correctness can be ensured through extensive fuzz testing and manual inspection. For ad-hoc queries, GenDB can work with a traditional DBMS in a hybrid architecture: the DBMS handles one-off queries, while GenDB speeds up frequent SQL templates. Our demonstration allows users to (1) visually and interactively explore how GenDB analyzes workloads, profiles hardware resources and underlying data, produces query plans, generates code based on them, and finally uses an optimizer to iteratively achieve a correct and efficient implementation; (2) use visual inspection and analysis to gain qualitative insights into why GenDB produces code that achieves significantly better performance than state-of-the-art query engines on two
benchmarks: TPC-H and a newly constructed
benchmark designed to reduce potential data leakage from LLM training data; and (3) upload their own data and queries to explore GenDB with different LLMs and query patterns.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effects
Seonglae Cho, Zekun Wu, Kleyton Da Costa et al. (+3 more)
Sparse autoencoder (SAE) features are used to interpret and steer large
language models, yet nobody has tested whether a feature's causal role is stable across SAE families. Single-
token features fire on one vocabulary item, so ground truth permits direct comparison. We analyze 3.9M features across six models and three SAE families and zero-ablate at full layer depth: they sit 4.7x tighter in decoder space and concentrate in early layers. Deleting one lowers the model's logit for that
token in 178 of 208 layer conditions, significant after multiple-comparison correction. And depth decides how the damage lands: early-layer deletions disrupt the layers that follow, late-layer deletions change the output directly. Cross-family causal differences exceed within-family scale effects: on the same base model, GemmaScope and BatchTopK features are causally anchored, LlamaScope features locally redundant. Under LlamaScope the
token returns to within 2x its pre-ablation rank 96-98% of the time. Changing only the activation function reverses the sign of that difference, so the training recipe is the remaining candidate: cross-family claims are sensitive to training methodology, not just activation function or scale.
Read the full abstract ▾
All authors Seonglae Cho, Zekun Wu, Kleyton Da Costa, Rishi Kalra, Ilham Wicaksono, Adriano Koshiyama
Computation and Language
Evaluating the Effectiveness of Persona Simulation in Opinion Prediction with GPT-4.1
Sarah Y. Li and Ziyu Yao
Persona simulation involves utilizing large
language models (LLMs) to anticipate human choices or interactions based on specific characteristic information. To further understand current limitations and future directions, we tested persona simulation in opinion prediction with GPT-4.1 (knowledge cutoff by June 2024). Using personas from nine U.S. states provided by Columbia University's Personas
dataset, GPT-4.1 accurately predicted 2024 election outcomes in eight out of the nine states, only failing in one of the swing states. We then focused on opinions related to medicine and healthcare. With the American Trends Panel Wave 123
dataset from Pew Research Center, GPT-4.1 was able to anticipate beliefs about childhood vaccines with an accuracy of up to 0.94. Furthermore, we applied GPT-4.1 to generate conversations among personas and observed that the simulated dialogues and opinions adhered well to personas' personalities and backgrounds, albeit lacking natural human-like flow. Persona simulation proves to be a promising application of artificial intelligence as long as biases are addressed. In the near future, it will be beneficial to apply it to opinion analysis and reaction prediction in diverse fields ranging from public health to lawmaking to economics.
Read the full abstract ▾
Computation and Language
LKValues: Aligning Large Language Models with Sri Lankan Societal Values
Nethmi Muthugala, Supryadi, Surangika Ranathunga et al. (+7 more)
Value
alignment of Large
Language Models (LLMs) has been shown to be culturally biased toward Western norms. This results in the mishandling of local values in
multilingual societies such as Sri Lanka that have their unique cultural dynamics. Existing
benchmarks overlook Sri Lankan-contextualized values in its official
language Sinhala, hindering culturally sensitive evaluation and fine-tuning. To bridge this gap, we propose LKValues, the first survey-grounded resource suite for Sri Lankan value
alignment. From a trilingual survey of 205 respondents, blending adapted global frameworks and LLM-elicited local constructs, we derive 40 majority-endorsed societal values. Using these values, we construct LKvaluesIT, a Sinhala-English news-derived instruction corpus containing 150k scenario-based instances, and LKvaluesBench, a value-sensitive evaluation
benchmark of 1,000 instances. We evaluate a set of proprietary and open-weight LLMs with LKvaluesBench. We fine-tune three open-weight base models (Qwen3.5-4B-Base, Qwen3.5-9B-Base, and Aya-Expanse-8B-Base). Our experiments show that newer and larger LLMs still exhibit low-resource and cultural value-
alignment gaps. LKValues fine-tuning improves Qwen-family models in English and Sinhala, reducing invalid outputs and cross-lingual disparities, though gains remain model-family dependent. These highlight LKValues efficacy in
embedding Sri Lankan values, offering a replicable pipeline for low-resource, country-specific pluralist value
alignment. The
dataset is publicly available at https://github.com/NextME14/LKValues.
Read the full abstract ▾
All authors Nethmi Muthugala, Supryadi, Surangika Ranathunga, Nisansa de Silva, Ruijie Tao, Ovindu Gunatunga, Pengyun Zhu, Shaowei Zhang, Jingting Zheng, Deyi Xiong
Computation and Language · Cross-listed from cs.AI
Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations
Hiskias Dingeto
Natural-
language autoencoders score explanations of hidden activations by reconstruction. An explanation is deemed faithful if the activation can be regenerated from it. The test is structurally insensitive to individual false claims. If flipping a claim does not change the reconstruction, the claim is never penalized. We show the test is passed in two ways, neither faithful. On a released Qwen-2.5-7B verbalizer, explanations reconstruct well above chance while ~2% of specific claims are ones the reconstruction depends on, so the score tracks gist, not specific facts. Under exact synthetic ground truth, standard training consistently develops co-adapted private codes (false wording the reconstruction depends on), and fixes that leave the target model unchanged do not help. We contribute two audit protocols, the comparison of grounding and truth and the swap to an independent evaluator, and RECAP (Readable Encodings via Co-trained Auxiliary Predictors), linear heads trained alongside the target model to keep designated content decodable. On RECAP-trained sandbox models, fresh verbalizers state the designated content truly and the codes vanish, at a +0.001-nat cost. This replicates on a pretrained Pythia-160M. The content becomes reliably decodable by a probe, though a fresh verbalizer conveys it only in part (truth 0.44-0.46 vs a near-zero control). For interpretability, high reconstruction does not certify individual claims. For AI safety, RECAP makes designated content checkable against a probe rather than asserted by prose a model can game. An independent probe ranks the verbalizer's true claims above its false ones (AUC 0.96 vs 0.82 without RECAP). Against an adversary that edits an explanation to maximize the score while lying (suppressing ~87% of its lie penalty), the RECAP probe still flags the lies (AUC 0.95) while the control probe collapses to chance (0.51).
Read the full abstract ▾
Computation and Language
Notes to Self: Can LLMs Benefit from Experiential Abstractions?
Chang Liu, Xinyu Li and Artur Dubrawski
Humans distill experience into reusable abstractions, e.g., strategies and cautionary reminders, and apply them to gradually solve problems more effectively. We study whether Large
Language Models (LLMs) can similarly benefit from such experiential abstractions. From LLMs' solution traces on the MATH training set, a stronger teacher or the LLMs themselves extract natural-
language abstractions into a retrievable library. We explore two usage modes: (1) inference-time
retrieval and (2) reinforcement learning (RL) with abstraction-augmented training
prompts. Experiential abstractions improve LLM performance on mathematical and logical
reasoning benchmarks. Self-extracted abstractions match teacher-extracted ones, and our abstraction usage framework can transfer to other
datasets and models. These findings suggest LLMs can extract and apply experiential abstractions much as humans leverage distilled experience.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Test-Time Training for Modality Order Consistency in Vision-Language Models
Aditi Gupta and Yossi Gandelsman
We find that
vision-
language models are sensitive to a specific semantically irrelevant change: the order in which the image and question are presented. Across three models and three
benchmarks, image first prompting consistently outperforms question-first prompting, revealing a repeatable modality order failure. We use this gap to design an order-consistent test-time training method. Our method substantially closes the modality-order gap across all evaluated settings. Surprisingly, it also yields consistent improvements in the stronger image-first branch over the baseline, hence bootstrapping both orderings toward mutual consistency. Activation patching localizes the ordering failure to a narrow mid-network region where representations diverge sharply between
prompt orders. We find that the test-time training method repairs this misalignment across layers. Together, our results identify modality-order sensitivity as a
circuit-level failure in VLMs and demonstrate that simple, asymmetric test-time adaptation can effectively mitigate it and even improve performance over the baseline.
Read the full abstract ▾
Computation and Language
Generative AI floods and dilutes the market for books
Tuhin Chakrabarty, Xinyue Liu, Jane C. Ginsburg et al. (+1 more)
Generative AI can produce book-length works of fiction at near-zero cost. These books are often dismissed as low-quality “slop” that buyers will ignore, and are assumed to carry little commercial weight. We test that assumption with full-text AI detection across 14,419 self-published genre-fiction books sold on Amazon from 2023 to 2026, matched to daily sales records through June 2026. None of these books disclose whether or not they contain AI-produced content. We find that books for which we detected substantial AI text ($>$ 25%) make up a large share of the catalog but a smaller share of sales. Even so, they reach commercial scale, winning a growing share of sales over time and taking more of the scarce top-rank positions once held by books with no detected AI text. Over this period, the number of books with observed sales in a quarter grew 19.2-fold, while quarterly revenue grew only 8.9-fold. The
market therefore added selling books faster than it added revenue, and revenue per selling book fell across most genres. Books with no AI text lose the most ground in genres with high AI
diffusion, and most of all where Kindle Unlimited availability is high. Among top-selling books, those with substantial AI text draw on more distinctive
language from existing books than do books with no AI text; for these books overlap rises with revenue, a gradient we do not detect for books with no AI text. Generative AI can thus reshape a creative
market through scale rather than quality. Our results bear directly on the
market-effect question at the center of the fair use defense to copyright infringement.
Read the full abstract ▾
All authors Tuhin Chakrabarty, Xinyue Liu, Jane C. Ginsburg, Paramveer Dhillon
Computation and Language
PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
Niqi Lyu, Pengtao Shi, Wei Qiu et al. (+4 more)
Large
language models (LLMs) provide strong
reasoning capabilities but are expensive to serve at scale, whereas small
language models (SLMs) are cheaper but less reliable on difficult problems. We introduce PyroDash, a cost-aware framework for
token-level SLM-LLM collaborative inference. During generation, the SLM decides whether to request assistance by emitting a control
token. A Collaborate Engine then sends the query and partial
reasoning trace to a frozen LLM for completion through a single handoff. The policy is internalized in the SLM, requiring neither a separate router, LLM retraining, nor access to LLM logits. PyroDash trains the SLM in three stages: control-
token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware
alignment with Group Relative Policy Optimization. Its reward balances answer accuracy against inference cost normalized by LLM-only inference. Across five mathematical
reasoning benchmarks, PyroDash supports different accuracy-cost operating points. With $\lambda=0.05$, it achieves 64.04 percent average accuracy, 6.36 percentage points above the LLM-only baseline, while reducing cost by 20.4 percent. With $\lambda=0.6$, it achieves 54.55 percent accuracy with a 1.90 percent LLM
token ratio and 0.012 LLM calls per example, reducing total cost from USD 49.36 to USD 1.78. These results show that learned
token-level handoffs can reduce LLM use while preserving strong
reasoning performance.
Read the full abstract ▾
All authors Niqi Lyu, Pengtao Shi, Wei Qiu, Jianlin Zhong, Sicong Xia, Jianyao Ma, Yicheng Ding
Computation and Language · Cross-listed from cs.LG
The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability
Abigail Woodring, Adrian Chan, Rana Muhammad Shahroz Khan et al. (+3 more)
Fine-tuning has been widely used to adapt large
language models (LLMs) for domain-specific tasks. Parameter efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) are frequently used to reduce computational costs. PortLLM is a training-free and data-free scheme used to adapt LLMs after continual pretraining. Although the initial PortLLM results show that LoRA patches exhibit short-term temporal portability, the long-term performance of PortLLM across several updates of continual pretraining remains underexplored. Furthermore, the intriguing effectiveness of PortLLM is not well understood from a theoretical standpoint. We address these two open questions by (1) performing an extensive empirical study of the long-term temporal portability of PortLLM patches across 10 continual pretraining steps using base models Mistral, Gemma, and Qwen; and (2) offering two theoretical analyses to explain our observation that the simple PortLLM method achieves competitive performance. We find empirically that the portability persists across longer time duration, indicating that repeated fine-tuning is not required when the base model is periodically updated. We find theoretically that near-orthogonality of high-dimensional vectors is a key justification for temporal portability. Our analyses also demonstrate a
geometric perspective of the loss landscape in facilitating the theoretical comparison of different adaptation options.
Read the full abstract ▾
All authors Abigail Woodring, Adrian Chan, Rana Muhammad Shahroz Khan, Sukwon Yun, Chau-Wai Wong, Tianlong Chen
Computation and Language
Sound Probabilistic Safety Bounds for Large Language Models
Mahdi Nazeri, Anne-Kathrin Schmuck, Sadegh Soudjani et al. (+1 more)
We propose a novel framework for computing rigorous bounds on the probability that a large
language model (LLM) generates harmful output to a given
prompt. We study a new application of the Clopper-Pearson confidence intervals to obtain probably approximately correct (PAC) bounds for this problem. As our main technical contribution, we propose an
algorithm that leverages features in the latent space to prioritize exploring branches in the auto-regressive generation tree that are more likely to produce harmful outputs. Our approach in particular enables the efficient computation of useful lower bounds, even in scenarios where the true harm probability is extremely small, and crucially, the obtained lower bounds are sound, i.e., formally proven to be less than the actual harmfulness probability: our experimental results demonstrate the effectiveness of our method by computing non-trivial lower bounds on state-of-the-art LLMs. This study newly enables the evaluation and statistical certification of LLMs.
Read the full abstract ▾
All authors Mahdi Nazeri, Anne-Kathrin Schmuck, Sadegh Soudjani, Alessandro Abate
Computation and Language · Cross-listed from cs.CV
Medical foundation models converge less under label supervision
Soroosh Tayebi Arasteh, Sebastian Ziegelmayer, Mahshad Lotfinia et al. (+4 more)
Diagnostic classifiers and imaging biomarkers are fitted on the
embeddings of medical foundation models. These models are replaced as new versions appear. This practice assumes that different models represent images alike. We tested this assumption with more than 750,000 images from 14
datasets in 5 imaging modalities, 18 public models, and 101 models trained on chest radiographs and histopathology that differ in pretraining objective, label type, size, random seed, initialization, training patients, or training recipe. Agreement was measured as the mutual k-nearest-neighbor overlap, the share of an image's nearest neighbors common to two models. Two public models shared on average 0.091 of the 10 nearest neighbors of a chest radiograph. A randomly initialized network shared 0.036 with them. In the four other modalities, they shared at most 0.300. On chest radiographs, agreement depended more on the source
dataset and the radiographic view than on the findings. It was not significantly associated with accuracy. A linear mapping between the
embeddings of two models, fitted on 4,096 unlabeled radiographs, transferred classifiers for 14 chest findings with 0.987 of their original area under the receiver operating characteristic curve. Convergence depended on the pretraining objective. Label-supervised models converged least at both model sizes in chest radiography and histopathology. On chest radiographs, two label-supervised models that differed only in their random seed shared 0.062 to 0.100 of their nearest neighbors. Two models trained with self-
distillation, masked image modeling, or contrastive pretraining shared 0.411 to 0.802. Developers can therefore choose convergence when they choose the pretraining objective. Before replacing a model, users could test on unlabeled images whether its successor returns the same nearest neighbors.
Read the full abstract ▾
All authors Soroosh Tayebi Arasteh, Sebastian Ziegelmayer, Mahshad Lotfinia, Lisa Adams, Sven Nebelung, Jakob Nikolas Kather, Daniel Truhn
Computation and Language
Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study
Andrei Chetvergov, Stepan Ukolov, Timofei Sivoraksha et al. (+5 more)
Large
language models are increasingly evaluated through the values they endorse, but such evaluations presuppose that models can identify the value expressed in a concrete situation. We study this prerequisite as controlled top-1 recognition over Schwartz's ten basic values. Our evaluation set contains 1,000 Russian situational texts, balanced across the ten values and independently labeled by two human annotators per item. We evaluate 21 instruction-tuned LLM runs under a fixed ranked-response protocol; 20 runs with reliable outputs form the semantic panel. Pooled Acc@1 is 0.683 and Acc@3 is 0.892, showing that models often locate the correct motivational region while ranking close alternatives unstably. Adjacent values account for 50.9% of semantic errors, compared with 24.4% under a checkpoint-specific null. Eight directed confusions recur across checkpoints and human-confirmed subsets. Several are strongly asymmetric, including Universalism to Benevolence, Tradition to Conformity, and Security to Power, whereas Stimulation-Hedonism forms a bidirectional boundary. Their severity is checkpoint-specific and can bias higher-order value profiles. The results motivate value-recognition evaluation that combines exact accuracy, ranked recovery, and directed error analysis.
Read the full abstract ▾
All authors Andrei Chetvergov, Stepan Ukolov, Timofei Sivoraksha, Alexander Evseev, Mikhail Solovev, Valeriia Kuschenko, Maria Chistyakova, Sergey Bolovtsov
Computation and Language · Cross-listed from cs.AI
PoTRE: Test-Time Reasoning inspired by Cognitive Heterogeneity
Anmol Kankariya and Sercan Ö. Ar{\i}k
While Large
Language Models (LLMs) excel at many tasks, they frequently struggle with complex
reasoning that requires long-horizon planning and iterative error correction. Furthermore, standard single-stream prompting proves brittle when models encounter novel abstractions or rigorous domain constraints. We introduce PoTRE (Poly-Topological
Reasoning Ensembles), a heterogeneous framework that decouples inference into four
agents: (1)
Adversarial Refinement
Agent, (2) Hierarchical strategic Planning
Agent, (3) Spectrum Search
Agent, and (4) Direct Chain
Agent. A final Task-Adaptive Aggregation Layer dynamically reconciles these perspectives – via final candidate selection, semantic synthesis, or neuro-symbolic verification – to produce a robust global solution. We evaluate PoTRE on three frontier
benchmarks: ARC-AGI-2, Humanity's Last Exam (HLE), and PRBench Finance. PoTRE achieves state-of-the-art accuracy of 49.92% on HLE, surpassing the previous best official score. We demonstrate that this architectural heterogeneity achieves improved
reasoning performance using similar or fewer inference
tokens compared to heavily scaled homogeneous baselines.
Read the full abstract ▾
Computation and Language
The Maskability Index: Predicting Task-Objective Alignment in Pretrained Language Models
Ahmad Pouramini and Mahsa Afsharzadeh
Large-scale pretrained
language models such as T5 and BERT have demonstrated strong capabilities for generating structured knowledge. However, their performance depends on how closely the prompting strategy matches the objectives used during pretraining. We introduce the Maskability Index (MI), a quantitative metric that estimates whether a knowledge relation is better suited to masked-style prompting or prefix-style prompting in few-shot generation. MI is computed from differences in DepthRank scores between masked and unmasked templates, providing a principled measure of objective-template
alignment. We evaluate MI on a diverse set of relations from the ATOMIC2020 knowledge base completion
benchmark and show that it is positively correlated with downstream generation performance. These results indicate that MI can help select appropriate prompting templates and adaptation strategies for extracting relational knowledge from pretrained
language models, especially in low-resource settings.
Read the full abstract ▾