Computation and Language
Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation
Bolei Ma, Yong Cao, Indira Sen et al. (+4 more)
Large
Language Models (LLMs) are increasingly used to simulate public opinion and other social phenomena. Most current studies constrain these simulations to multiple-choice or short-answer formats for ease of scoring and comparison, but such closed designs overlook the inherently generative nature of LLMs. In this position paper, we argue that open-endedness, using free-form text that captures topics, viewpoints, and
reasoning processes "in" LLMs, is essential for realistic social simulation. Drawing on decades of survey-methodology research and recent advances in NLP, we argue why this open-endedness is valuable in LLM social simulations, showing how it can improve measurement and design, support exploration of unanticipated views, and reduce researcher-imposed directive bias. It also captures expressiveness and individuality, aids in pretesting, and ultimately enhances methodological utility. We call for novel practices and evaluation frameworks that leverage rather than constrain the open-ended generative diversity of LLMs, creating synergies between NLP and social science.
Read the full abstract ▾
All authors Bolei Ma, Yong Cao, Indira Sen, Anna-Carolina Haensch, Frauke Kreuter, Barbara Plank, Daniel Hershcovich
Computation and Language
On the Role of Preference Variance in Preference Optimization
Jiacheng Guo, Zihao Li, Jiahao Qiu et al. (+2 more)
Direct Preference Optimization (DPO) has emerged as an important approach for learning from human preferences in aligning large
language models (LLMs). However, collecting human preference data is costly and inefficient, motivating methods to reduce the required annotations. In this work, we investigate the impact of preference variance (PVar), which measures the variance in model preferences when comparing pairs of responses, on the effectiveness of DPO training. We provide a theoretical insight by establishing an upper bound on the DPO gradient norm for any given
prompt, showing it is controlled by the PVar of that
prompt. This implies that
prompts with low PVar can only produce small gradient updates, making them less valuable for learning. We validate this finding by fine-tuning LLMs with preferences generated by a reward model, evaluating on two
benchmarks (AlpacaEval 2.0 and Arena-Hard). Experimental results demonstrate that
prompts with higher PVar outperform randomly selected
prompts or those with lower PVar. We also show that our PVar-based selection method is robust, when using smaller reward models (1B, 3B) for selection. Notably, in a separate experiment using the original human annotations from the UltraFeedback
dataset, we found that training on only the top 10% of
prompts with the highest PVar yields better evaluation performance than training on the full
dataset, highlighting the importance of preference variance in identifying informative examples for efficient LLM
alignment.
Read the full abstract ▾
All authors Jiacheng Guo, Zihao Li, Jiahao Qiu, Yue Wu, Mengdi Wang
Computation and Language
CurLL: A Developmental Framework to Evaluate Continual Learning in Language Models
Pavan Kalyan, Shubhra Mishra, Satya Lokam et al. (+1 more)
We introduce a comprehensive continual learning
dataset and
benchmark (CurlL) grounded in human developmental trajectories from ages 5-10, enabling systematic and fine-grained assessment of models' ability to progressively acquire new skills. CurlL spans five developmental stages (0-4) covering ages 5-10, supported by a skill graph that breaks down broad skills into smaller abilities, concrete goals, and measurable indicators, while also capturing which abilities build on others. We generate a 23.4B-
token synthetic
dataset with controlled skill progression, vocabulary complexity, and format diversity, comprising paragraphs, comprehension-based QA (CQA), skill-testing QA (CSQA), and instruction-response (IR) pairs. Stage-wise
token counts range from 2.12B to 6.78B
tokens, supporting precise analysis of forgetting, forward transfer, and backward transfer. Using a 135M-parameter
transformer trained under independent, joint, and sequential (continual) setups, we show trade-offs in skill retention and transfer efficiency. By mirroring human learning patterns and providing fine-grained control over skill dependencies, this work advances continual learning evaluations for
language models.
Read the full abstract ▾
All authors Pavan Kalyan, Shubhra Mishra, Satya Lokam, Navin Goyal
Computation and Language
OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning
Yifeng Xiong and Xiaohui Xie
Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large
language models but suffers from catastrophic forgetting when learned updates interfere with the dominant singular directions that encode essential pre-trained knowledge. We propose Orthogonal Projection LoRA (OPLoRA), a theoretically grounded approach that prevents this interference through double-sided orthogonal projections. By decomposing frozen weights via SVD, OPLoRA constrains LoRA updates to lie entirely within the orthogonal complement of the top-$k$ singular subspace using projections $P_L = I - U_k U_k^\top$ and $P_R = I - V_k V_k^\top$. We prove that this construction exactly preserves the top-$k$ singular triples, providing mathematical guarantees for knowledge retention. To quantify subspace interference, we introduce $\rho_k$, a metric measuring update
alignment with dominant directions. Extensive experiments across commonsense
reasoning, mathematics, and code generation demonstrate that OPLoRA significantly reduces forgetting while maintaining competitive task-specific performance on LLaMA-2 7B and Qwen2.5 7B, establishing orthogonal projection as an effective mechanism for knowledge preservation in parameter-efficient fine-tuning.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Max It or Miss It: Benchmarking LLM On Solving Extremal Problems
Binxin Gao and Jingjun Han
Test-time scaling has enabled Large
Language Models (LLMs) with remarkable
reasoning capabilities, particularly in mathematical domains, through intermediate chain-of-thought (CoT)
reasoning before generating final answers. However, the specific sources and mechanisms underlying these
reasoning capabilities remain insufficiently understood. Optimization
reasoning, i.e. finding extrema under constraints, represents a fundamental abstraction that underpins critical applications in planning, control, resource allocation, and
prompt search. To systematically evaluate this capability, we introduce ExtremBench, a
benchmark dataset for solving mathematical extremal problems, curated from inequality exercises used for Chinese Mathematical Olympiad and transformed into $93$ standardized extrema-finding problems. We conduct extensive evaluations across various state-of-the-art open-source model families, including the Qwen3, GPT-OSS, and DeepSeek. Our results reveal that LLMs' extremal-solving
reasoning capabilities do not always align with those of current mathematical
benchmarks such as AIME25 and MATH-500, with some models showing strong general mathematical
reasoning but poor extremal-solving skills, and vice versa. This discrepancy highlights a critical gap in current evaluation practices and suggests that existing
benchmarks may not comprehensively capture the full spectrum of mathematical
reasoning abilities.
Read the full abstract ▾
Computation and Language
Tailored untruths: How personalisation challenges LLM safeguards
João A. Leite, João Luz, Silvia Gargova et al. (+5 more)
Large
Language Models (LLMs) can generate highly persuasive disinformation, yet little is known about how effectively they personalise it across
languages and demographic groups. We present the first large-scale
multilingual study of persona-targeted disinformation generation by LLMs. Using a red-teaming methodology, we prompted eight leading models with 324 false narratives and 150 demographic personas in four
languages (English, Russian, Portuguese, and Hindi), creating AI-TRAITS, a
dataset of 1.6 million personalised disinformation texts. We treat safeguards as compromised whenever a model generates the requested falsehood, whether directly or accompanied by a safety disclaimer. Across models, safeguards failed for 80% of non-personalised
prompts and 77.7% of personalised ones, with Grok producing disinformation in over 94% of cases. All models effectively tailored outputs to target personas, employing substantially more persuasive techniques than in non-personalised content. Additional analyses reveal persona-specific linguistic and psychological patterns and show that safeguard effectiveness varies markedly across
languages. Together, these findings expose significant weaknesses in current LLM safety mechanisms and highlight the need for more robust,
multilingual safeguards against personalised AI-generated disinformation.
Read the full abstract ▾
All authors João A. Leite, João Luz, Silvia Gargova, Arnav Arora, Gustavo Sampaio, Ian Roberts, Carolina Scarton, Kalina Bontcheva
Computation and Language · Cross-listed from cs.RO
UNCAP: Uncertainty-Guided Neurosymbolic Planning Using Natural Language Communication for Cooperative Autonomous Vehicles
Neel P. Bhatt, Po-han Li, Kushagra Gupta et al. (+7 more)
Safe large-scale coordination of multiple cooperative connected autonomous vehicles (CAVs) hinges on communication that is both efficient and interpretable. Existing approaches either rely on transmitting high-bandwidth raw sensor data streams or neglect perception and planning uncertainties inherent in shared data, resulting in systems that are neither scalable nor safe. To address these limitations, we propose Uncertainty-Guided Natural
Language Cooperative Autonomous Planning (UNCAP), a
vision-
language model-based planning approach that enables CAVs to communicate via lightweight natural
language messages while explicitly accounting for perception uncertainty in decision-making. UNCAP features a two-stage communication protocol: (i) an ego CAV first identifies the subset of vehicles most relevant for information exchange, and (ii) the selected CAVs then transmit messages that quantitatively express their perception uncertainty. By selectively fusing messages that maximize mutual information, this strategy allows the ego vehicle to integrate only the most relevant signals into its decision-making, improving both the scalability and reliability of cooperative planning. Experiments across diverse driving scenarios show a 63% reduction in communication bandwidth with a 31% increase in driving safety score, a 61% reduction in decision uncertainty, and a four-fold increase in collision distance margin during near-miss events. Project website: https://uncap-project.github.io/
Read the full abstract ▾
All authors Neel P. Bhatt, Po-han Li, Kushagra Gupta, Rohan Siva, Daniel Milan, Alexander T. Hogue, Sandeep P. Chinchali, David Fridovich-Keil, Zhangyang Wang, Ufuk Topcu
Computation and Language · Cross-listed from cs.AI
DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping
Wei Fan, Wenlin Yao, Zheng Li et al. (+6 more)
Large
language models (LLMs) augmented with multi-step
reasoning and action generation abilities have shown promise in leveraging external tools to tackle complex tasks that require long-horizon planning. However, existing approaches either rely on implicit planning in the
reasoning stage or introduce explicit planners without systematically addressing how to optimize the planning stage. As evidence, we observe that under vanilla reinforcement learning (RL), planning
tokens exhibit significantly higher
entropy than other action
tokens, revealing uncertain decision points that remain under-optimized. To address this, we propose DeepPlanner, an end-to-end RL framework that effectively enhances the planning capabilities of deep research
agents. Our approach shapes
token-level advantage with an
entropy-based term to allocate larger updates to high
entropy tokens, and selectively upweights sample-level advantages for planning-intensive rollouts. Extensive experiments across seven deep research
benchmarks demonstrate that DeepPlanner improves planning quality and achieves state-of-the-art results under a substantially lower training budget.
Read the full abstract ▾
All authors Wei Fan, Wenlin Yao, Zheng Li, Feng Yao, Xin Liu, Liang Qiu, Qingyu Yin, Yangqiu Song, Bing Yin
Computation and Language
3-Model Speculative Decoding
Sanghyun Byun, Mohanad Odema, Jung Ick Guack et al. (+3 more)
Speculative Decoding (SD) accelerates inference in large
language models by using a smaller draft model to propose
tokens, which are then verified by a larger target model. However, the throughput gains of SD are fundamentally limited by a trade-off between draft model size and
token acceptance: smaller draft models generate
tokens more quickly but exhibit greater divergence from the target model, resulting in lower acceptance rates and reduced speedups. We introduce Pyramid Speculative Decoding (PyramidSD), an extension of SD that inserts an intermediate qualifier model between the draft and target to bridge the distributional gap in output predictions, allowing smaller model to be used for drafting. This hierarchical decoding strategy improves
alignment across models, enabling higher acceptance rates and allowing the use of significantly smaller draft models without sacrificing overall performance. PyramidSD builds on fuzzy acceptance criteria to support relaxed divergence thresholds at each stage, improving throughput. In experiments, PyramidSD achieves up to 1.91x generation speed over standard SD, reaching 124
tokens per second on a consumer GPU (RTX 4090). In small-
memory settings with a 1B-parameter draft model and an 8B target model, PyramidSD minimally trades target model quality for improved throughput. Overall, PyramidSD offers a practical approach to enhancing speculative decoding efficiency and can be readily applied to existing inference pipelines.
Read the full abstract ▾
All authors Sanghyun Byun, Mohanad Odema, Jung Ick Guack, Baisub Lee, Jacob Song, Woo Seong Chung
Computation and Language
The Curious Case of Curiosity across Human Cultures and LLMs
Angana Borah, Zhijing Jin and Rada Mihalcea
Recent advances in Large
Language Models (LLMs) have expanded their role in human interaction, yet curiosity – a central driver of inquiry – remains underexplored in these systems, particularly across cultural contexts. In this work, we investigate cultural variation in curiosity using Yahoo! Answers, a real-world multi-country
dataset spanning diverse topics. We introduce CUEST (CUriosity Evaluation across SocieTies), an evaluation framework that measures human-model
alignment in curiosity through linguistic (style), topic preference (content) analysis and grounding insights in social science constructs. Across open- and closed-source models, we find that LLMs flatten cross-cultural diversity, aligning more closely with how curiosity is expressed in Western countries. We then explore fine-tuning strategies to induce curiosity in LLMs, narrowing the human-model
alignment gap by up to 50%. Finally, we demonstrate the practical value of curiosity for LLM adaptability across cultures, showing its importance for future NLP research.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Unifying Vision-Language Latents for Zero-label Image Caption Enhancement
Sanghyun Byun, Jung Ick Guack, Mohanad Odema et al. (+3 more)
Vision-
language models (VLMs) achieve remarkable performance through large-scale image-text pretraining. However, their reliance on labeled image
datasets limits scalability and leaves vast amounts of unlabeled image data underutilized. To address this, we propose Unified
Vision-
Language Alignment for Zero-Label Enhancement (ViZer), an enhancement training framework that enables zero-label learning in image captioning, providing a practical starting point for broader zero-label adaptation in
vision-
language tasks. Unlike prior approaches that rely on human or synthetically annotated
datasets, ViZer actively aligns
vision and
language representation features during training, enabling existing VLMs to generate improved captions without requiring text labels or full retraining. We demonstrate ViZer's advantage in qualitative evaluation, as automated caption metrics such as CIDEr and BERTScore often penalize details that are absent in reference captions. Applying ViZer on SmolVLM-Base and Qwen2-VL, we observe consistent qualitative improvements, producing captions that are more grounded and descriptive than their baseline.
Read the full abstract ▾
All authors Sanghyun Byun, Jung Ick Guack, Mohanad Odema, Baisub Lee, Jacob Song, Woo Seong Chung
Computation and Language
Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answering
Nil-Jana Akpinar, Chia-Jung Lee, Vanessa Murdock et al. (+1 more)
Large
Language Models (LLMs) should answer factual questions truthfully, grounded in objective knowledge, regardless of user context such as self-disclosed personal information, or system personalization. In this paper, we present the first systematic evaluation of LLM robustness to inquiry personas, i.e. user profiles that convey attributes like identity, expertise, or belief. While prior work has primarily focused on
adversarial inputs or distractors for robustness testing, we evaluate plausible, human-centered inquiry persona cues that users disclose in real-world interactions. We find that such cues can meaningfully alter QA accuracy and trigger failure modes such as refusals, hallucinated limitations, and role confusion. These effects highlight how model sensitivity to user framing can compromise factual reliability, and position inquiry persona testing as an effective tool for robustness evaluation.
Read the full abstract ▾
All authors Nil-Jana Akpinar, Chia-Jung Lee, Vanessa Murdock, Pietro Perona
Computation and Language · Cross-listed from cs.CY
Toward LLM-Supported Automated Assessment of Critical Thinking Subskills
Marisa C. Peczuh, Nischal Ashok Kumar, Ryan Baker et al. (+11 more)
As the world becomes increasingly saturated with AI-generated content, disinformation, and algorithmic persuasion, critical thinking - the capacity to evaluate evidence, detect unreliable claims, and exercise independent judgment - is becoming a defining human skill. Developing critical thinking skills through timely assessment and feedback is crucial; however, there has not been extensive work in educational data mining on defining, measuring, and supporting critical thinking. In this paper, we investigate the feasibility of measuring "subskills" that underlie critical thinking. We ground our work in an authentic task where students operationalize critical thinking by writing argumentative essays. We developed a coding rubric based on an established skills progression and completed human coding for a corpus of student essays. We then evaluated three distinct approaches to automated scoring: zero-shot prompting, few-shot prompting, and supervised fine-tuning, implemented across three large
language models (GPT-5, Llama 3.1 8B, and ModernBERT). Fine-tuning Llama 3.1 8B achieved the best results and demonstrated particular strength on subskills with highly separable proficiency levels with balanced labels across levels, while lower performance was observed for subskills that required detection of subtle distinctions between proficiency levels or imbalanced labels. Our exploratory work represents an initial step toward scalable assessment of critical thinking skills across authentic educational contexts. Future research should continue to combine automated critical thinking assessment with human validation to more accurately detect and measure dynamic, higher-order thinking skills.
Read the full abstract ▾
All authors Marisa C. Peczuh, Nischal Ashok Kumar, Ryan Baker, Blair Lehman, Danielle Eisenberg, Caitlin Mills, Payu Wittawatolarn, Kushaan Naskar, Keerthi Chebrolu, Sudhip Nashi, Cadence Young, Brayden Liu, Sherry Lachman, Andrew Lan
Computation and Language
EduDial: Constructing a Large-scale Multi-turn Teacher-Student Dialogue Corpus
Shouang Wei, Min Zhang, Xin Lin et al. (+3 more)
Recently, several multi-turn dialogue
benchmarks have been proposed to evaluate the conversational abilities of large
language models (LLMs). As LLMs are increasingly recognized as a key technology for advancing intelligent education, owing to their ability to deeply understand instructional contexts and provide personalized guidance, the construction of dedicated teacher-student dialogue
benchmarks has become particularly important. To this end, we present EduDial, a comprehensive multi-turn teacher-student dialogue
dataset. EduDial covers 345 core knowledge points and consists of 34,250 dialogue sessions generated through interactions between teacher and student
agents. Its design is guided by Bloom's taxonomy of educational objectives and incorporates ten questioning strategies, including situational questioning, zone of proximal development (ZPD) questioning, and metacognitive questioning-thus better capturing authentic classroom interactions. Furthermore, we design differentiated teaching strategies for students at different cognitive levels, thereby providing more targeted teaching guidance. Building on EduDial, we further develop EduDial-LLM 32B via training and propose an 11-dimensional evaluation framework that systematically measures the teaching abilities of LLMs, encompassing both overall teaching quality and content quality. Experiments on 17 mainstream LLMs reveal that most models struggle in student-centered teaching scenarios, whereas our EduDial-LLM achieves significant gains, consistently outperforming all baselines across all metrics. The code is available at https://github.com/Mind-Lab-ECNU/EduDial/tree/main.
Read the full abstract ▾
All authors Shouang Wei, Min Zhang, Xin Lin, Bo Jiang, Zhongxiang Dai, Kun Kuang
Computation and Language · Cross-listed from cs.AI
From Literal to Liberal: A Meta-Prompting Framework for Eliciting Human-Aligned Exception Handling in Large Language Models
Imran Khan
Large
Language Models (LLMs) are increasingly being deployed as the
reasoning engines for agentic AI systems, yet they exhibit a critical flaw: a rigid adherence to explicit rules that leads to decisions misaligned with human common sense and intent. This "rule-
rigidity" is a significant barrier to building trustworthy autonomous
agents. While prior work has shown that supervised fine-tuning (SFT) with human explanations can mitigate this issue, SFT is computationally expensive and inaccessible to many practitioners. To address this gap, we introduce the Rule-Intent Distinction (RID) Framework, a novel, low-compute meta-prompting technique designed to elicit human-aligned exception handling in LLMs in a zero-shot manner. The RID framework provides the model with a structured cognitive schema for deconstructing tasks, classifying rules, weighing conflicting outcomes, and justifying its final decision. We evaluated the RID framework against baseline and Chain-of-Thought (CoT) prompting on a custom
benchmark of 20 scenarios requiring nuanced judgment across diverse domains. Our human-verified results demonstrate that the RID framework significantly improves performance, achieving a 95% Human
Alignment Score (HAS), compared to 80% for the baseline and 75% for CoT. Furthermore, it consistently produces higher-quality, intent-driven
reasoning. This work presents a practical, accessible, and effective method for steering LLMs from literal instruction-following to liberal, goal-oriented
reasoning, paving the way for more reliable and pragmatic AI
agents.
Read the full abstract ▾
Computation and Language
A Critical Review of the Need for Knowledge-Centric Evaluation of Quranic Recitation
Mohammed Hilal Al-Kharusi, Khizar Hayat, Khalil Bader Al Ruqeishi et al. (+1 more)
The art and science of Quranic recitation (Tajweed), a discipline governed by meticulous phonetic, rhythmic, and theological principles, confronts substantial educational challenges in today's digital age. Although modern technology offers unparalleled opportunities for learning, existing automated systems for evaluating recitation have struggled to gain broad acceptance or demonstrate educational effectiveness. This literature review examines this crucial disparity, offering a thorough analysis of scholarly research, digital platforms, and commercial tools developed over the past twenty years. Our analysis uncovers a fundamental flaw in current approaches that adapt Automatic
Speech Recognition (ASR) systems, which emphasize word identification over qualitative acoustic evaluation. These systems suffer from limitations such as reliance on biased
datasets, demographic disparities, and an inability to deliver meaningful feedback for improvement. Challenging these data-centric methodologies, we advocate for a paradigm shift toward a knowledge-based computational framework. By leveraging the unchanging nature of the Quranic text and the well-defined rules of Tajweed, we propose that an effective evaluation system should be built upon rule-based acoustic modeling centered on canonical pronunciation principles and articulation points (Makhraj), rather than depending on statistical patterns derived from flawed or biased data. The review concludes that the future of automated Quranic recitation assessment lies in hybrid systems that combine linguistic expertise with advanced audio processing. Such an approach paves the way for developing reliable, fair, and pedagogically effective tools that can authentically assist learners across the globe.
Read the full abstract ▾
All authors Mohammed Hilal Al-Kharusi, Khizar Hayat, Khalil Bader Al Ruqeishi, Haroon Rashid Lone
Computation and Language
Efficient Adaptive Transformer: An Empirical Study and Reproducible Framework
Jan Miller
The Efficient Adaptive
Transformer (EAT) framework unifies three adaptive efficiency techniques - progressive
token pruning, sparse
attention, and dynamic early exiting - into a single, reproducible architecture for input-adaptive inference. EAT provides an open-source benchmarking pipeline that automates data processing, timing, and ablation across GLUE tasks (SST-2, QQP, MNLI). Although this empirical study finds that combining these mechanisms can increase latency in shallow six-layer models, it demonstrates that EAT achieves slightly higher accuracy than the optimized DistilBERT baseline on SST-2, illustrating the potential of dynamic computation for latency-sensitive NLP. The main contribution is the open, end-to-end reproducible framework - complete with scripts, CSV logging, and analysis utilities - intended to serve as a community tool for further research on adaptive
transformers.
Read the full abstract ▾
Computation and Language
VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
Jesse Atuhurra, Iqra Ali, Tomoya Iwakura et al. (+2 more)
Vision Language Models (VLMs) are pivotal for advancing perception in intelligent
agents. Yet, evaluation of VLMs remains limited to predominantly English-centric
benchmarks in which the image-text pairs comprise short texts. To evaluate VLM fine-grained abilities, in four
languages under long-text settings, we introduce a novel
multilingual benchmark VLURes featuring eight
vision-and-
language tasks, and a pioneering unrelatedness task, to probe the fine-grained Visual and Linguistic Understanding capabilities of VLMs across English, Japanese, and low-resource
languages, Swahili, and Urdu. Our
datasets, curated from web resources in the target
language, encompass ten diverse image categories and rich textual context, introducing valuable
vision-
language resources for Swahili and Urdu. By prompting VLMs to generate responses and rationales, evaluated automatically and by native speakers, we uncover performance disparities across
languages and tasks critical to intelligent
agents, such as object recognition, scene understanding, and relationship understanding. We conducted evaluations of ten VLMs with VLURes. The best performing model, GPT-4o, achieves an overall accuracy of 90.8% and lags human performance by 6.7%, though the gap is larger for open-source models. The gap highlights VLURes' critical role in developing intelligent
agents to tackle multi-modal visual
reasoning.
Read the full abstract ▾
All authors Jesse Atuhurra, Iqra Ali, Tomoya Iwakura, Hidetaka Kamigaito, Tatsuya Hiraoka
Computation and Language · Cross-listed from cs.CV
SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models
Weiyang Jin, Yuwei Niu, Jiaqi Liao et al. (+4 more)
Recently, remarkable progress has been made in Unified
Multimodal Models (UMMs), which integrate
vision-
language generation and understanding capabilities within a single framework. However, a model's strong visual understanding often fails to transfer to visual generation: it may correctly judge
prompt-image
alignment while failing to generate a faithful image from the same
prompt. This raises a compelling question: Can a model improve itself by using its understanding module to reward its generation module? We introduce SRUM, a self-rewarding post-training framework directly applicable to existing UMMs of various designs. SRUM creates a feedback loop where the model's own understanding module acts as an internal “evaluator”, providing corrective signals to improve generation without additional human-labeled data or external reward models. To provide comprehensive feedback, SRUM uses a global-local dual reward system: a global reward ensures overall visual semantics and layout, while a local reward refines fine-grained, object-level fidelity. SRUM shows strong generalization, boosting performance on T2I-CompBench from 82.18 to 88.37 and on T2I-ReasonBench from 43.82 to 46.75. Overall, our work establishes a powerful paradigm for enabling a UMM's understanding module to guide and enhance its own generation via self-rewarding.
Read the full abstract ▾
All authors Weiyang Jin, Yuwei Niu, Jiaqi Liao, Chengqi Duan, Aoxue Li, Shenghua Gao, Xihui Liu
Computation and Language
Cost Analysis of Human-corrected Transcription for Predominately Oral Languages
Yacouba Diarra, Nouhoum Souleymane Coulibaly and Michael Leventhal
Creating
speech datasets for low-resource
languages is a critical yet poorly understood challenge, particularly regarding the actual cost in human labor. This paper investigates the time and complexity required to produce high-quality annotated
speech data for a subset of low-resource
languages, low literacy Predominately Oral
Languages, focusing on Bambara, a Manding
language of Mali. Through a one-month field study involving ten transcribers with native proficiency, we analyze the correction of ASR-generated transcriptions of 53 hours of Bambara voice data. We report that it takes, on average, 30 hours of human labor to accurately transcribe one hour of
speech data under laboratory conditions and 36 hours under field conditions. The study provides a baseline and practical insights for a large class of
languages with comparable profiles undertaking the creation of NLP resources.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
Content Anonymization for Privacy in Long-form Audio
Cristina Aggazzotti, Ashi Garg, Zexin Cai et al. (+1 more)
Voice anonymization techniques have been found to successfully obscure a speaker's acoustic identity in short, isolated utterances in
benchmarks such as the VoicePrivacy Challenge. In practice, however, utterances seldom occur in isolation: long-form audio is commonplace in domains such as interviews, phone calls, and meetings. In these cases, many utterances from the same speaker are available, which pose a significantly greater
privacy risk: given multiple utterances from the same speaker, an attacker could exploit an individual's vocabulary, syntax, and turns of phrase to re-identify them, even when their voice is completely disguised. To address this risk, we propose a new approach that performs a contextual rewriting of the transcripts in an ASR-TTS pipeline to eliminate speaker-specific style while preserving meaning. We present results in a long-form telephone conversation setting demonstrating the effectiveness of a content-based attack on voice-anonymized
speech. Then we show how the proposed content-based anonymization methods can mitigate this risk while preserving
speech utility. Overall, we find that paraphrasing is an effective defense against content-based attacks and recommend that stakeholders adopt this step to ensure anonymity in long-form audio.
Read the full abstract ▾
All authors Cristina Aggazzotti, Ashi Garg, Zexin Cai, Nicholas Andrews
Computation and Language
Dr.LLM: Dynamic Layer Routing in LLMs
Ahmed Heakl, Martin Gubri, Salman Khan et al. (+2 more)
Large
Language Models (LLMs) process every
token through all layers of a
transformer stack, causing wasted computation on simple queries and insufficient flexibility for harder ones that need deeper
reasoning. Adaptive-depth methods can improve efficiency, but prior approaches rely on costly inference-time search, architectural changes, or large-scale retraining, and in practice often degrade accuracy despite efficiency gains. We introduce Dr. LLM, Dynamic routing of Layers for LLMs, a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block. Routers are trained with explicit supervision: using Monte Carlo Tree Search (MCTS), we derive high-quality layer configurations that preserve or improve accuracy under a compute budget. Our design, windowed pooling for stable routing, focal loss with class balancing, and bottleneck MLP routers, ensures robustness under class imbalance and long sequences. On ARC (logic) and DART (math), Dr. LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average. Routers generalize to out-of-domain tasks (MMLU, GSM8k, AIME, TruthfulQA, SQuADv2, GPQA, PIQA, AGIEval) with only 0.85% accuracy drop while retaining efficiency, and outperform prior routing methods by up to +7.7%p. Overall, Dr. LLM shows that explicitly supervised routers retrofit frozen LLMs for budget-aware, accuracy-driven inference without altering base weights. Code is available at https://github.com/parameterlab/dr-llm.
Read the full abstract ▾
All authors Ahmed Heakl, Martin Gubri, Salman Khan, Sangdoo Yun, Seong Joon Oh
Computation and Language
Łukasz Borchmann
Linguistic commentary on LLMs, heavily influenced by the theoretical frameworks of de Saussure and Chomsky, is often speculative and unproductive. Critics challenge whether LLMs can legitimately model
language, citing the need for "deep structure" or "grounding" to achieve an idealized linguistic "competence." We argue for a radical shift in perspective towards the empiricist principles of Witold Mańczak, a prominent general and historical linguist. He defines
language not as a "system of signs" or a "computational system of the
brain" but as the totality of all that is said and written. Above all, he identifies frequency of use of particular
language elements as
language's primary governing principle. Using his framework, we challenge prior critiques of LLMs and provide a constructive guide for designing, evaluating, and interpreting
language models.
Read the full abstract ▾
Computation and Language
Hey, wait a minute: on at-issue sensitivity in Language Models
Sanghee J. Kim and Kanishka Misra
Evaluating the naturalness of dialogue in
language models (LMs) is not trivial: notions of 'naturalness' vary, and scalable quantitative metrics remain limited. This study leverages the linguistic notion of 'at-issueness' to assess dialogue naturalness and introduces a new method: Divide, Generate, Recombine, and Compare (DGRC). DGRC (i) divides a dialogue as a
prompt, (ii) generates continuations for subparts using LMs, (iii) recombines the dialogue and continuations, and (iv) compares the likelihoods of the recombined sequences. This approach mitigates bias in linguistic analyses of LMs and enables systematic testing of discourse-sensitive behavior. Applying DGRC, we find that LMs prefer to continue dialogue on at-issue content, with this effect enhanced in instruct-tuned models. They also reduce their at-issue preference when relevant cues (e.g., "Hey, wait a minute") are present. Although instruct-tuning does not further amplify this modulation, the pattern reflects a hallmark of successful dialogue dynamics.
Read the full abstract ▾
Computation and Language
Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages
Nadine El-Naggar, Tatsuki Kuribayashi and Ted Briscoe
Whether
language models (LMs) have inductive biases that favor typologically frequent grammatical properties over rare, implausible ones has been investigated, typically using artificial
languages (ALs) (White and Cotterell, 2021; Kuribayashi et al., 2024). In this paper, we extend these works from two perspectives. First, we extend their context-free AL formalization by adopting Generalized Categorial Grammar (GCG) (Wood, 2014), which allows ALs to cover attested but previously overlooked constructions, such as unbounded dependency and mildly context-sensitive structures. Second, our evaluation focuses more on the generalization ability of LMs to process unseen longer test sentences. Thus, our ALs better capture features of natural
languages and our experimental paradigm leads to clearer conclusions – typologically plausible word orders tend to be easier for LMs to productively generalize.
Read the full abstract ▾
Computation and Language
Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception
Ziyang Ma, Ruiyang Xu, Zhenghao Xing et al. (+9 more)
Fine-grained perception of
multimodal information is critical for advancing human-AI interaction. With recent progress in audio-visual technologies, Omni
Language Models (OLMs), capable of processing audio and
video signals in parallel, have emerged as a promising paradigm for achieving richer understanding and
reasoning. However, their capacity to capture and describe fine-grained details remains limited explored. In this work, we present a systematic and comprehensive investigation of omni detailed perception from the perspectives of the data pipeline, models, and
benchmark. We first identify an inherent "co-growth" between detail and
hallucination in current OLMs. To address this, we propose Omni-Detective, an agentic data generation pipeline integrating tool-calling, to autonomously produce highly detailed yet minimally hallucinatory
multimodal data. Based on the data generated with Omni-Detective, we train two captioning models: Audio-Captioner for audio-only detailed perception, and Omni-Captioner for audio-visual detailed perception. Under the cascade evaluation protocol, Audio-Captioner achieves the best performance on MMAU and MMAR among all open-source models, surpassing Gemini 2.5 Flash and delivering performance comparable to Gemini 2.5 Pro. On existing detailed captioning
benchmarks, Omni-Captioner sets a new state-of-the-art on VDC and achieves the best trade-off between detail and
hallucination on the
video-SALMONN 2 testset. Given the absence of a dedicated
benchmark for omni detailed perception, we design Omni-Cloze, a novel cloze-style evaluation for detailed audio, visual, and audio-visual captioning that ensures stable, efficient, and reliable assessment. Experimental results and analysis demonstrate the effectiveness of Omni-Detective in generating high-quality detailed captions, as well as the superiority of Omni-Cloze in evaluating such detailed captions.
Read the full abstract ▾
All authors Ziyang Ma, Ruiyang Xu, Zhenghao Xing, Yunfei Chu, Yuxuan Wang, Jinzheng He, Jin Xu, Pheng-Ann Heng, Kai Yu, Junyang Lin, Eng Siong Chng, Xie Chen
Computation and Language
Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generations
Sunny Yu, Ahmad Jabbar, Robert Hawkins et al. (+2 more)
Different open-ended generation tasks require different degrees of output diversity. However, current LLMs are often miscalibrated. They collapse to overly homogeneous outputs for creative tasks and hallucinate diverse but incorrect responses for factual tasks. We argue that these two failure modes are unified by, and can both be addressed by, the notion of effective generation space size (GSS) – the set of semantically distinct outputs a model considers for a
prompt. We present GSSBench, a task suite of
prompt pairs with ground-truth GSS relationships to assess different metrics and understand where models diverge from desired behavior. We find that
hallucination detection metrics, particularly EigenScore, consistently outperform standard diversity and uncertainty quantification metrics, while using only model internals, providing interpretable insights into a model's internal task representations. We demonstrate three applications of GSS: (1) detecting
prompt ambiguity and predicting clarification questions for better grounding, (2) interpreting overthinking and underthinking in
reasoning models, and (3) steering models to expand their generation space to yield high-quality and diverse outputs.
Read the full abstract ▾
All authors Sunny Yu, Ahmad Jabbar, Robert Hawkins, Dan Jurafsky, Myra Cheng
Computation and Language · Cross-listed from cs.HC
Who is a Better Matchmaker? Human vs. Algorithmic Judge Assignment in a High-Stakes Startup Competition
Sarina Xi, Orelia Pi, Miaomiao Zhang et al. (+3 more)
There is growing interest in applying artificial intelligence (AI) to automate and support complex decision-making tasks. However, it remains unclear how
algorithms compare to human judgment in contexts requiring semantic understanding and domain expertise. We examine this in the context of the judge assignment problem, matching submissions to suitably qualified judges. Specifically, we tackled this problem at the Harvard President's Innovation Challenge, the university's premier venture competition awarding over \$500,000 to student and alumni startups. This represents a real-world environment where high-quality judge assignment is essential. We developed an AI-based judge-assignment algorithm, Hybrid Lexical-Semantic Similarity Ensemble (HLSE), and deployed it at the competition. We then evaluated its performance against human expert assignments using blinded match-quality scores from judges on $309$ judge-venture pairs. Using a Mann-Whitney U statistic based test, we found no statistically significant difference in assignment quality between the two approaches ($AUC=0.48, p=0.40$); on average, algorithmic matches are rated $3.90$ and manual matches $3.94$ on a 5-point scale, where 5 indicates an excellent match. Furthermore, manual assignments that previously required a full week could be automated in several hours by the
algorithm during deployment. These results demonstrate that HLSE achieves human-expert-level matching quality while offering greater scalability and efficiency, underscoring the potential of AI-driven solutions to support and enhance human decision-making for judge assignment in high-stakes settings.
Read the full abstract ▾
All authors Sarina Xi, Orelia Pi, Miaomiao Zhang, Becca Xiong, Jacqueline Ng Lane, Nihar B. Shah
Computation and Language · Cross-listed from cs.LG
Demystifying Hybrid Thinking: Can LLMs Truly Switch Between Think and No-Think?
Shouren Wang, Wang Yang, Xianxuan Long et al. (+4 more)
Hybrid thinking enables LLMs to switch between
reasoning and direct answering, offering a balance between efficiency and
reasoning capability. Yet our experiments reveal that current hybrid thinking LLMs only achieve partial mode separation:
reasoning behaviors often leak into the no-think mode. To understand and mitigate this, we analyze the factors influencing controllability and identify four that matter most: (1) larger data scale, (2) using think and no-think answers from different questions rather than the same question, (3) a moderate increase in no-think data number, and (4) a two-phase strategy that first trains
reasoning ability and then applies hybrid think training. Building on these findings, we propose a practical recipe that, compared to standard training, can maintain accuracy in both modes while significantly reducing no-think output length (from 1085 to 585 on MATH500) and occurrences of
reasoning-supportive
tokens such as "wait" (from 5917 to 522 on MATH500). Our findings highlight the limitations of current hybrid thinking and offer directions for strengthening its controllability. The code is available at: https://github.com/SR-A-W/demystifying-hybrid-thinking
Read the full abstract ▾
All authors Shouren Wang, Wang Yang, Xianxuan Long, Debargha Ganguly, Qifan Wang, Vipin Chaudhary, Xiaotian Han
Computation and Language · Cross-listed from cs.IR
Understanding Parametric Knowledge Injection in Retrieval-Augmented Generation
Minghao Tang, Shiyu Ni, Jingtong Wu et al. (+2 more)
Context-grounded generation underpins many LLM applications, including long-document question answering (QA), conversational personalization, and
retrieval-augmented generation (RAG). However, classic
token-based context concatenation is costly for long inputs and can be lost in the middle at extreme context lengths. Recent work explores context parameterization, which encodes context into lightweight trainable parameters (e.g., LoRA adapters) injected into a frozen LLM. Extending this idea to retrieved evidence yields parametric RAG (P-RAG), which incorporates knowledge via parameter updates rather than
token-level
attention. In this paper, we present a systematic study of this emerging RAG paradigm-parametric knowledge injection. First, we reassess P-RAG under answer-presence accuracy and show that it does not consistently outperform standard
token-based RAG (T-RAG), while combining both (PT-RAG) achieves the best overall performance. Second, we introduce a QA
benchmark with up-to-date knowledge beyond the LLM's internal
memory to enable controlled analysis. Our representational and mechanistic results indicate that parametric representations capture document-level semantics and primarily influence deeper feed-forward computations, providing high-level guidance but limited evidence consolidation. Finally, we evaluate parametric injection under key RAG challenges, demonstrating improved faithfulness under knowledge conflicts, stronger robustness to
retrieval noise, and solid generalization to tasks beyond QA. Our findings clarify the strengths and limitations of parametric RAG and provide practical guidance for future
retrieval-augmented LLM systems.
Read the full abstract ▾
All authors Minghao Tang, Shiyu Ni, Jingtong Wu, Zengxin Han, Keping Bi