Computation and Language · Cross-listed from cs.AI
How Clued up are LLMs? Evaluating Multi-Step Deductive Reasoning in a Text-Based Game Environment
Rebecca Ansell and Autumn Toney-Wails
Deducing whodunit proves challenging for LLM
agents. In this paper, we implement a text-based multi-
agent version of the classic board game Clue as a rule-based testbed for evaluating multi-step deductive
reasoning, with six
agents drawn from GPT-4o-mini and Gemini-2.5-Flash. We further investigate whether fine-tuning on structured logic puzzles transfers to improved in-game
reasoning and gameplay. Across 18 simulated games,
agents achieve only four correct wins, indicating difficulty in maintaining consistent deductive
reasoning over the course of a full game. Additionally, we find that fine-tuning does not reliably improve performance and, in some cases, appears to increase
reasoning volume without improving
reasoning precision.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
Multilingual Reference Need Assessment System for Wikipedia
Aitolkyn Baigutanova, Francisco Navas, Pablo Aragon et al. (+5 more)
Wikipedia is a critical source of information for millions of users across the Web. It serves as a key resource for large
language models, search engines, question-answering systems, and other Web-based applications. In Wikipedia, content needs to be verifiable, meaning that readers can check that claims are backed by references to reliable sources. This depends on manual verification by editors, an effective but labor-intensive process, especially given the high volume of daily edits. To address this challenge, we introduce a
multilingual machine learning system to assist editors in identifying claims requiring citations. Our approach is tested in 10
language editions of Wikipedia, outperforming existing
benchmarks for reference need assessment. We not only consider machine learning evaluation metrics but also system requirements, allowing us to explore the trade-offs between model accuracy and computational efficiency under real-world infrastructure constraints. We deploy our system in production and release data and code to support further research.
Read the full abstract ▾
All authors Aitolkyn Baigutanova, Francisco Navas, Pablo Aragon, Mykola Trokhymovych, Muniza Aslam, Ai-Jou Chou, Miriam Redi, Diego Saez-Trumper
Computation and Language
Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency
Lucas Bandarkar, Alan Ansell and Trevor Cohn
Modern LLMs continue to exhibit significant variance in behavior across
languages, such as being able to recall factual information in some
languages but not others. While typically studied as a problem to be mitigated, in this work, we propose leveraging this cross-lingual inconsistency as a tool for interpretability in mixture-of-experts (MoE) LLMs. Our knowledge localization framework contrasts routing for sets of
languages where the model correctly recalls information from
languages where it fails. This allows us to isolate model components that play a functional role in answering about a piece of knowledge. Our method proceeds in two stages: (1) querying the model with difficult factual questions across a diverse set of
languages to generate "success" and "failure" activation buckets and then (2) applying a statistical contrastive analysis to the MoE router logits to identify experts important for knowledge. To validate the necessity of this small number of experts for answering a knowledge question, we deactivate them and re-ask the question. We find that despite only deactivating about 20 out of 6000 experts, the model no longer answers correctly in over 40% of cases. Generally, this method provides a realistic and scalable knowledge localization approach to address increasingly complex LLMs.
Read the full abstract ▾
Computation and Language
Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction
Ryo Kamoi, Ameya Godbole, Binglin Zhou et al. (+5 more)
Simulating human conversations using large
language models (LLMs) has emerged as a scalable methodology for modeling human social interaction. This paper reconsiders the evaluation of simulated conversations by explicitly recognizing that human conversations inherently involve inconsistent and uncollaborative behaviors, such as misunderstandings and interruptions. Since these behaviors contribute to the complexity of human social interaction, we argue that LLM-simulated conversations should reproduce them at frequencies comparable to those observed in human conversations. To support a detailed and interpretable evaluation of these behaviors, we introduce CoCoEval, a framework consisting of an evaluation scheme based on turn-level detection of 10 types of inconsistent and uncollaborative behaviors and a
benchmark for simulating conversations in professional scenarios involving collaboration and conflict. Using CoCoEval, we compare human conversations with those simulated by GPT-4.1, GPT-5.1, and Claude Opus 4. The results show that (1) LLM-simulated conversations exhibit far fewer inconsistent and uncollaborative behaviors than human conversations under vanilla prompting, and (2)
prompt engineering and supervised fine-tuning do not provide reliable control over these behaviors, often leading to the overproduction of specific behaviors. CoCoEval identifies gaps between human and LLM-simulated conversations that are not captured by conventional evaluation based on conversation-level Likert scales, raising concerns about the use of LLMs as proxies for human social interaction.
Read the full abstract ▾
All authors Ryo Kamoi, Ameya Godbole, Binglin Zhou, Xiaoxin Lu, Longqi Yang, Rui Zhang, Mengting Wan, Pei Zhou
Computation and Language
Ensemble Self-Training for Unsupervised Machine Translation
Ido Aharon, Jonathan Shaki and Sarit Kraus
We present an ensemble-driven self-training framework for unsupervised neural machine translation (UNMT). Starting from a primary
language pair, we train multiple UNMT models that share the same translation task but differ in an auxiliary
language, inducing structured diversity across models. We then generate pseudo-translations for the primary pair using
token-level ensemble decoding, averaging model predictions in both directions. These ensemble outputs are used as synthetic parallel data to further train each model, allowing the models to improve via shared supervision. At deployment time, we select a single model by validation performance, preserving single-model inference cost. Experiments show statistically significant improvements over single-model UNMT baselines, with mean gains of 1.7 chrF when translating from English and 0.67 chrF when translating into English.
Read the full abstract ▾
Computation and Language
Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scripts
Lucas Bandarkar, Alan Ansell and Trevor Cohn
In this work, we analyze shortcomings in cross-lingual knowledge transfer in large, modern
reasoning LLMs. We demonstrate that the perceived gap in knowledge transfer is primarily a script barrier. First, we conduct an observational data analysis on the performance of thinking models on two
datasets with local knowledge from around the world, ECLeKTic and MultiLoKo. Our regression analysis shows that script match - not
language or family - is the primary predictor of knowledge transfer failure once model capability and question difficulty are accounted for. We further this finding by providing the LLMs with the key entities of the questions in their source
language and find that this disproportionately improves cross-script questions. We then posit that these LLMs could be
reasoning better at test-time. To evaluate this, we develop a synthetic generation pipeline to design SFT samples to encourage the model to better reason about transliteration ambiguities when trying to fetch parametric knowledge at inference-time. We show that teaching two models to reason better reduces the cross-script transfer gap. As a result, we conclude that there is potential to improve cross-lingual parametric knowledge transfer during post-training.
Read the full abstract ▾
Computation and Language
Evaluating Ill-Defined Tasks in Large Language Models
Yi Zhou and Basel Shbita
Many evaluations of Large
Language Models (LLMs) target tasks that are inherently ill-defined, with unclear input and output spaces and ambiguous success criteria. We analyze why existing evaluation
benchmarks and metrics fail to provide reliable or diagnostic signals of model capability for such tasks. We examine two case studies: Complex Instruction Following (CIF), where we identify recurring issues including limited coverage of real-world instruction complexity, sensitivity to instruction phrasing, inconsistent and non-comparable metrics, and instability introduced by LLM-based judges; and Natural
Language to Mermaid Sequence Diagrams (NL2Mermaid), where we show how multi-faceted evaluation criteria can yield actionable insights beyond aggregate scores. Together, these case studies show that current evaluations frequently conflate distinct failure modes, yielding scores that are unstable, non-diagnostic, and difficult to act upon. Our findings expose fundamental limitations in existing evaluation practices for ill-defined tasks and motivate more robust, interpretable evaluation designs.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning
Shenzhi Wang, Shixuan Liu, Jing Zhou et al. (+8 more)
Vision-
language models (VLMs) show strong
multimodal capabilities but still struggle with fine-grained
vision-
language reasoning. We find that long chain-of-thought (CoT)
reasoning exposes diverse failure modes, including perception,
reasoning, knowledge, and
hallucination errors, which can compound across intermediate steps. However, most existing
vision-
language data used for reinforcement learning with verifiable rewards (RLVR) does not involve complex
reasoning chains that rely on visual evidence throughout, leaving these weaknesses largely unexposed. We therefore propose HopChain, a scalable framework for synthesizing multi-hop
vision-
language reasoning data for RLVR training of VLMs. Each synthesized multi-hop query forms a logically dependent chain of instance-grounded hops, where earlier hops establish the instances, sets, or conditions needed for later hops, while the final answer remains a specific, unambiguous number suitable for verifiable rewards. We train Qwen3.5-35B-A3B and Qwen3.5-397B-A17B under two RLVR settings: the original data alone, and the original data plus HopChain's multi-hop data, and compare them across 24
benchmarks spanning STEM and Puzzle, General VQA, Text Recognition and Document Understanding, and
Video Understanding. Although this multi-hop data is not synthesized for any specific
benchmark, it improves 20 of 24
benchmarks on both models, indicating broad and generalizable gains. Consistently, replacing full chained queries with half-multi-hop or single-hop variants reduces the average score across five representative
benchmarks from 70.4 to 66.7 and 64.3, respectively. Notably, multi-hop gains peak in long-CoT
vision-
language reasoning, exceeding 50 points in the ultra-long-CoT regime. These experiments establish HopChain as an effective, scalable framework for synthesizing multi-hop data that improves generalizable
vision-
language reasoning.
Read the full abstract ▾
All authors Shenzhi Wang, Shixuan Liu, Jing Zhou, Chang Gao, Xiong-Hui Chen, Binghai Wang, An Yang, Shiji Song, Bowen Yu, Gao Huang, Junyang Lin
Computation and Language
LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings
Lifu Tu, Rongguang Wang, Tao Sheng et al. (+2 more)
Robustness evaluation for Natural
Language to SQL (NL2SQL) systems is essential because real-world database environments are dynamic, noisy, and continuously evolving, whereas conventional
benchmark evaluations typically assume static schemas and well-formed user inputs. In this work, we introduce a robustness evaluation
benchmark containing approximately ten types of perturbations and conduct evaluations under both traditional and agentic settings. We assess multiple state-of-the-art large
language models (LLMs), including Grok-4.1, Gemini-3-Pro, Claude-Opus-4.6, and GPT-5.2. Our results show that these models generally maintain strong performance under several perturbations; however, notable performance degradation is observed for surface-level noise (e.g., character-level corruption) and linguistic variation that preserves semantics while altering lexical or syntactic forms. Furthermore, we observe that surface-level noise causes larger performance drops in traditional pipelines, whereas linguistic variation presents greater challenges in agentic settings. These findings highlight the remaining challenges in achieving robust NL2SQL systems, particularly in handling linguistic variability.
Read the full abstract ▾
All authors Lifu Tu, Rongguang Wang, Tao Sheng, Sujjith Ravi, Dan Roth
Computation and Language · Cross-listed from cs.LG
Efficient Reasoning on the Edge
Yelysei Bondarenko, Thomas Hehn, Rob Hesselink et al. (+15 more)
Large
language models (LLMs) with chain-of-thought
reasoning achieve state-of-the-art performance across complex problem-solving tasks, but their verbose
reasoning traces and large context requirements make them impractical for edge deployment. These challenges include high
token generation costs, large KV-cache footprints, and inefficiencies when distilling
reasoning capabilities into smaller models for mobile devices. Existing approaches often rely on distilling
reasoning traces from larger models into smaller models, which are verbose and stylistically redundant, undesirable for on-device inference. In this work, we propose a lightweight approach to enable
reasoning in small LLMs using LoRA adapters combined with supervised fine-tuning. We further introduce budget forcing via reinforcement learning on these adapters, significantly reducing response length with minimal accuracy loss. To address
memory-bound decoding, we exploit parallel test-time scaling, improving accuracy at minor latency increase. Finally, we present a dynamic adapter-switching mechanism that activates
reasoning only when needed and a KV-cache sharing strategy during
prompt encoding, reducing time-to-first-
token for on-device inference. Experiments on Qwen2.5-7B demonstrate that our method achieves efficient, accurate
reasoning under strict resource constraints, making LLM
reasoning practical for mobile scenarios.
Videos demonstrating our solution running on mobile devices are available on our project page.
Read the full abstract ▾
All authors Yelysei Bondarenko, Thomas Hehn, Rob Hesselink, Romain Lepert, Fabio Valerio Massoli, Evgeny Mironov, Leyla Mirvakhabova, Tribhuvanesh Orekondy, Spyridon Stasis, Andrey Kuzmin, Anna Kuzina, Markus Nagel, Ankita Nayak, Corrado Rainone, Ork de Rooij, Paul N Whatmough, Arash Behboodi, Babak Ehteshami Bejnordi
Computation and Language
Chronos: Temporal-Aware Conversational Agents with Structured Event Retrieval for Long-Term Memory
Sahil Sen, Elias Lumer, Anmol Gulati et al. (+1 more)
Recent advances in Large
Language Models (LLMs) have enabled conversational AI
agents to engage in extended multi-turn interactions spanning weeks or months. However, existing
memory systems struggle to reason over temporally grounded facts and preferences that evolve across months of interaction and lack effective
retrieval strategies for multi-hop, time-sensitive queries over long dialogue histories. We introduce Chronos, a novel temporal-aware
memory framework that decomposes raw dialogue into subject-verb-object event tuples with resolved datetime ranges and entity aliases, indexing them in a structured event calendar alongside a turn calendar that preserves full conversational context. At query time, Chronos applies dynamic prompting to generate tailored
retrieval guidance for each question, directing the
agent on what to retrieve, how to filter across time ranges, and how to approach multi-hop
reasoning through an iterative tool-calling loop over both calendars. We evaluate Chronos with 8 LLMs, both open-source and closed-source, on the LongMemEvalS
benchmark comprising 500 questions spanning six categories of dialogue history tasks. Chronos Low achieves 92.60% and Chronos High scores 95.60% accuracy, setting a new state of the art with an improvement of 7.67% over the best prior system. Ablation results reveal the events calendar accounts for a 58.9% gain on the baseline while all other components yield improvements between 15.5% and 22.3%. Notably, Chronos Low alone surpasses prior approaches evaluated under their strongest model configurations.
Read the full abstract ▾
All authors Sahil Sen, Elias Lumer, Anmol Gulati, Vamse Kumar Subbiah
Computation and Language
Online Experiential Learning for Language Models
Tianzhu Ye, Li Dong, Qingxiu Dong et al. (+3 more)
The prevailing paradigm for improving large
language models relies on offline training with human annotations or simulated environments, leaving the rich experience accumulated during real-world deployment entirely unexploited. We propose Online Experiential Learning (OEL), a framework that enables
language models to continuously improve from their own deployment experience. OEL operates in two stages: first, transferable experiential knowledge is extracted and accumulated from interaction trajectories collected on the user side; second, this knowledge is consolidated into model parameters via on-policy context
distillation, requiring no access to the user-side environment. The two stages are iterated to form an online learning loop, where the improved model collects higher-quality trajectories that yield richer experiential knowledge for subsequent rounds. We evaluate OEL on text-based game environments across multiple model scales and both thinking and non-thinking variants. OEL achieves consistent improvements over successive iterations, enhancing both task accuracy and
token efficiency while preserving out-of-distribution performance. Our analysis further shows that extracted experiential knowledge is significantly more effective than raw trajectories, and that on-policy consistency between the knowledge source and the policy model is critical for effective learning.
Read the full abstract ▾
All authors Tianzhu Ye, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, Furu Wei
Computation and Language
Mediocrity is the key for LLM as a Judge Anchor Selection
Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen et al. (+1 more)
The “LLM-as-a-judge” paradigm has become a standard method for evaluating open-ended generation. To address the quadratic scalability costs of pairwise comparisons, popular
benchmarks like Arena-Hard and AlpacaEval compare all models against a single anchor. However, despite its widespread use, the impact of anchor selection on the reliability of the results remains largely unexplored. In this work, we systematically investigate the effect of anchor selection by evaluating 22 different anchors on the Arena-Hard-v2.0
dataset. We find that the choice of anchor is critical: a poor anchor can dramatically reduce correlation with human rankings. We identify that common anchor choices (best-performing and worst-performing models) make poor anchors. Because these extreme anchors are consistently better or worse than all other models, they are seldom indicative of the relative ranking of the models. We further quantify the effect size of anchor selection, showing it is comparable to the selection of a judge model. We conclude with actionable recommendations. First, we conduct a power analysis, and compute sufficient
benchmark sizes for anchor-based evaluation, finding that standard
benchmark sizes are insufficient for pairwise evaluation and fail to distinguish between competitive models reliably. Second, we provide guidelines for selecting informative anchors to ensure reliable and efficient evaluation practices.
Read the full abstract ▾
All authors Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen, Omri Abend
Computation and Language · Cross-listed from cs.AI
Prompt Programming for Cultural Bias and Alignment of Large Language Models
Maksim Eren, Eric Michalak, Brian Cook et al. (+1 more)
Culture shapes
reasoning, values, prioritization, and strategic decision-making, yet large
language models (LLMs) often exhibit cultural biases that misalign with target populations. As LLMs are increasingly used for strategic decision-making, policy support, and document engineering tasks such as summarization, categorization, and compliance-oriented
auditing, improving cultural
alignment is important for ensuring that downstream analyses and recommendations reflect target-population value profiles rather than default model priors. Previous work introduced a survey-grounded cultural
alignment framework and showed that culture-specific prompting can reduce misalignment, but it primarily evaluated proprietary models and relied on manual
prompt engineering. In this paper, we validate and extend that framework by reproducing its social sciences survey based projection and distance metrics on open-weight LLMs, testing whether the same cultural skew and benefits of culture conditioning persist outside closed LLM systems. Building on this foundation, we introduce use of
prompt programming with DSPy for this problem-treating
prompts as modular, optimizable programs-to systematically tune cultural conditioning by optimizing against cultural-distance objectives. In our experiments, we show that
prompt optimization often improves upon cultural
prompt engineering, suggesting
prompt compilation with DSPy can provide a more stable and transferable route to culturally aligned LLM responses.
Read the full abstract ▾
All authors Maksim Eren, Eric Michalak, Brian Cook, Johnny Seales Jr
Computation and Language · Cross-listed from cs.AI
Is Conformal Factuality for RAG-based LLMs Robust? Novel Metrics and Systematic Insights
Yi Chen, Daiwei Chen, Sukrut Madhav Chikodikar et al. (+2 more)
Large
language models (LLMs) frequently hallucinate, limiting their reliability in knowledge-intensive applications.
Retrieval-augmented generation (RAG) and conformal factuality have emerged as potential ways to address this limitation. While RAG aims to ground responses in retrieved evidence, it provides no statistical guarantee that the final output is correct. Conformal factuality filtering offers distribution-free statistical reliability by scoring and filtering atomic claims using a threshold calibrated on held-out data, however, the informativeness of the final output is not guaranteed. We systematically analyze the reliability and usefulness of conformal factuality for RAG-based LLMs across generation, scoring, calibration, robustness, and efficiency. We propose novel informativeness-aware metrics that better reflect task utility under conformal filtering. Across three
benchmarks and multiple model families, we find that (i) conformal filtering suffers from low usefulness at high factuality levels due to vacuous outputs, (ii) conformal factuality guarantee is not robust to distribution shifts and distractors, highlighting the limitation that requires calibration data to closely match deployment conditions, and (iii) lightweight entailment-based verifiers match or outperform LLM-based model confidence scorers while requiring over $100\times$ fewer FLOPs. Overall, our results expose factuality-informativeness trade-offs and fragility of conformal filtering framework under distribution shifts and distractors, highlighting the need for new approaches for reliability with robustness and usefulness as key metrics, and provide actionable guidance for building RAG pipelines that are both reliable and computationally efficient.
Read the full abstract ▾
All authors Yi Chen, Daiwei Chen, Sukrut Madhav Chikodikar, Caitlyn Heqi Yin, Ramya Korlakai Vinayak
Computation and Language
SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue
Jonggeun Lee, Junseong Pyo, Jeongmin Park et al. (+1 more)
Robust voice
agents require exposure to the full diversity of how people interact through
speech. However, obtaining enough spoken interactions is prohibitively expensive. Building spoken user simulators that address this requires large-scale spoken task-oriented dialogue (TOD) data encompassing spoken user behaviors, yet existing
datasets are limited in scale and domain coverage, with no systematic pipeline for augmenting them. To address this, we introduce SpokenTOD, a spoken TOD
dataset of 52,390 dialogues and 1,034 hours of
speech augmented with four spoken user behaviors—cross-turn slots, barge-in, disfluency, and emotional prosody—across diverse speakers and domains. Building on SpokenTOD, we present SpokenUS, a spoken user simulator grounded in TOD that decides when to speak through a dedicated turn-taking head. SpokenUS achieves comparable goal coverage to much larger models while substantially outperforming all baselines in human MOS, disclosing slot values gradually across the dialogue as humans do rather than front-loading them. Further analysis confirms that SpokenUS's spoken behaviors pose meaningful challenges to voice
agents, making it a practical tool for evaluating more robust spoken dialogue systems. Our code is available at https://github.com/holi-lab/SpokenUS.
Read the full abstract ▾
All authors Jonggeun Lee, Junseong Pyo, Jeongmin Park, Yohan Jo
Computation and Language · Cross-listed from cs.LG
SOMP: Scalable Gradient Inversion for Large Language Models via Subspace-Guided Orthogonal Matching Pursuit
Yibo Li and Qiongxiu Li
Gradient inversion attacks reveal that private training text can be reconstructed from shared gradients, posing a
privacy risk to large
language models (LLMs). While prior methods perform well in small-batch settings, scaling to larger batch sizes and longer sequences remains challenging due to severe signal mixing, high computational cost, and degraded fidelity. We present SOMP (Subspace-Guided Orthogonal Matching Pursuit), a scalable gradient inversion framework that casts text recovery from aggregated gradients as a sparse signal recovery problem. Our key insight is that aggregated
transformer gradients retain exploitable head-wise
geometric structure together with sample-level sparsity. SOMP leverages these properties to progressively narrow the search space and disentangle mixed signals without exhaustive search. Experiments across multiple LLM families, model scales, and five
languages show that SOMP consistently outperforms prior methods in the aggregated-gradient regime.For long sequences at batch size B=16, SOMP achieves substantially higher reconstruction fidelity than strong baselines, while remaining computationally competitive. Even under extreme aggregation (up to B=128), SOMP still recovers meaningful text, suggesting that
privacy leakage can persist in regimes where prior attacks become much less effective.
Read the full abstract ▾
Computation and Language
TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities
Victoria Graf, Valentina Pyatkin, Nouha Dziri et al. (+2 more)
Multi-turn conversations are a common and critical mode of
language model interaction. However, current open training and evaluation data focus on single-turn settings, failing to capture the additional dimension of these longer interactions. To understand this multi-/single-turn gap, we first introduce a new
benchmark, TurnWiseEval, for multi-turn capabilities that is directly comparable to single-turn chat evaluation. Our evaluation isolates multi-turn specific conversational ability through pairwise comparison to equivalent single-turn settings. We additionally introduce our synthetic multi-turn data pipeline TurnWiseData which allows the scalable generation of multi-turn training data. Our experiments with Olmo 3 show that training with multi-turn data is vital to achieving strong multi-turn chat performance, and that including as little as 10k multi-turn conversations during post-training can lead to a 12% improvement on TurnWiseEval.
Read the full abstract ▾
All authors Victoria Graf, Valentina Pyatkin, Nouha Dziri, Nathan Lambert, Hannaneh Hajishirzi
Computation and Language
Probing Cultural Signals in Large Language Models through Author Profiling
Valentin Lafargue, Ariel Guerra-Adames, Emmanuelle Claeys et al. (+2 more)
Large
language models (LLMs) are increasingly deployed in applications with societal impact, raising concerns about the cultural biases they encode. We probe these representations by evaluating whether LLMs can perform author profiling from song lyrics in a zero-shot setting, inferring singers' gender and ethnicity without task-specific fine-tuning. Across several open-source models evaluated on more than 10,000 lyrics, we find that LLMs achieve non-trivial profiling performance but demonstrate systematic cultural
alignment: most models default toward North American ethnicity, while DeepSeek-1.5B aligns more strongly with Asian ethnicity. This finding emerges from both the models' prediction distributions and an analysis of their generated rationales. To quantify these disparities, we introduce two
fairness metrics, Modality Accuracy Divergence (MAD) and Recall Divergence (RD), and show that Ministral-8B displays the strongest ethnicity bias among the evaluated models, whereas Gemma-12B shows the most balanced behavior. Our code is available on [GitHub](https://github.com/ValentinLafargue/CulturalProbingLLM) and results on [HuggingFace](https://huggingface.co/
datasets/ValentinLAFARGUE/AuthorProfilingResults).
Read the full abstract ▾
All authors Valentin Lafargue, Ariel Guerra-Adames, Emmanuelle Claeys, Elouan Vuichard, Jean-Michel Loubes
Computation and Language · Cross-listed from cs.CV
Retrieving Counterfactuals Improves Visual In-Context Learning
Guangzhi Xiong, Sanchit Sinha, Zhenghao He et al. (+1 more)
Vision-
language models (VLMs) have achieved impressive performance across a wide range of
multimodal reasoning tasks, but they often struggle to disentangle fine-grained visual attributes and reason about underlying causal relationships. In-context learning (ICL) offers a promising avenue for VLMs to adapt to new tasks, but its effectiveness critically depends on the selection of demonstration examples. Existing
retrieval-augmented approaches typically rely on passive similarity-based
retrieval, which tends to select correlated but non-causal examples, amplifying spurious associations and limiting model robustness. We introduce CIRCLES (Composed Image
Retrieval for Causal Learning Example Selection), a novel framework that actively constructs demonstration sets by retrieving counterfactual-style examples through targeted, attribute-guided composed image
retrieval. By incorporating counterfactual-style examples, CIRCLES enables VLMs to implicitly reason about the causal relations between attributes and outcomes, moving beyond superficial correlations and fostering more robust and grounded
reasoning. Comprehensive experiments on four diverse
datasets demonstrate that CIRCLES consistently outperforms existing methods across multiple architectures, especially on small-scale models, with pronounced gains under information scarcity. Furthermore, CIRCLES retrieves more diverse and causally informative examples, providing qualitative insights into how models leverage in-context demonstrations for improved
reasoning. Our code is available at https://github.com/gzxiong/CIRCLES.
Read the full abstract ▾
All authors Guangzhi Xiong, Sanchit Sinha, Zhenghao He, Aidong Zhang
Computation and Language · Cross-listed from cs.AI
IQuest-Coder-V1 Technical Report
Jian Yang, Wei Zhang, Shawn Guo et al. (+35 more)
In this report, we introduce the IQuest-Coder-V1 series-(7B/14B/40B/40B-Loop), a new family of code large
language models (LLMs). Moving beyond static code representations, we propose the code-flow multi-stage training paradigm, which captures the dynamic
evolution of software logic through different phases of the pipeline. Our models are developed through the evolutionary pipeline, starting with the initial pre-training consisting of code facts, repository, and completion data. Following that, we implement a specialized mid-training stage that integrates
reasoning and agentic trajectories in 32k-context and repository-scale in 128k-context to forge deep logical foundations. The models are then finalized with post-training of specialized coding capabilities, which is bifurcated into two specialized paths: the thinking path (utilizing
reasoning-driven RL) and the instruct path (optimized for general assistance). IQuest-Coder-V1 achieves state-of-the-art performance among competitive models across critical dimensions of code intelligence: agentic software engineering, competitive programming, and complex tool use. To address deployment constraints, the IQuest-Coder-V1-Loop variant introduces a recurrent mechanism designed to optimize the trade-off between model capacity and deployment footprint, offering an architecturally enhanced path for efficacy-efficiency trade-off. We believe the release of the IQuest-Coder-V1 series, including the complete white-box chain of checkpoints from pre-training bases to the final thinking and instruction models, will advance research in autonomous code intelligence and real-world agentic systems.
Read the full abstract ▾
All authors Jian Yang, Wei Zhang, Shawn Guo, Zhengmao Ye, Lin Jing, Shark Liu, Yizhi Li, Jiajun Wu, Cening Liu, X. Ma, Yuyang Song, Siwei Wu, Yuwen Li, L. Liao, T. Zheng, Ziling Huang, Zelong Huang, Che Liu, Yan Xing, Renyuan Li, Qingsong Cai, Hanxu Yan, Siyue Wang, Shikai Li, Jason Klein Liu, An Huang, Yongsheng Kang, Jinxing Zhang, Chuan Hao, Haowen Wang, Weicheng Gu, Ran Tao, Mingjie Tang, Peihao Wu, Jianzhou Wang, Xianglong Liu, Weifeng Lv, Bryan Dai
Computation and Language
Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models
Mohamed Adel, Bashar Alhafni and Nizar Habash
LLMs perform strongly across NLP, but their ability to produce explicit grammatical analyses remains unclear. Arabic provides a challenging testbed due to its rich morphology and orthographic ambiguity, which create strong morphology-syntax interactions. We present a unified evaluation of LLMs on Arabic morphosyntactic tagging and dependency parsing, covering pre-tokenized, raw-text, and cascaded settings. We compare zero-shot prompting with
retrieval-based in-context learning. Relevant demonstrations substantially improve performance. The strongest LLMs approach supervised tagging and parsing systems; however, they require substantial annotated data for demonstration
retrieval and considerable computational resources. We make all code and data used in this paper publicly available.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
CritiSense: Critical Digital Literacy and Resilience Against Misinformation
Firoj Alam, Fatema Ahmad, Ali Ezzat Shahroor et al. (+5 more)
Misinformation on social media undermines informed decision-making and public trust. Prebunking offers a proactive complement by helping users recognize manipulation tactics before they encounter them in the wild. We present CritiSense, a mobile media-literacy app that builds these skills through short, interactive challenges with instant feedback. It is the first
multilingual (supporting nine
languages) and modular platform, designed for rapid updates across topics and domains. We report a usability study with 93 users: 83.9% expressed overall satisfaction and 90.1% rated the app as easy to use. Qualitative feedback indicates that CritiSense helps improve digital literacy skills. Overall, it provides a
multilingual prebunking platform and a testbed for measuring the impact of microlearning on misinformation resilience. Over 6 months, we have reached 500+ active users. It is freely available to all users on the Apple App Store (https://apps.apple.com/us/app/critisense/id6749675792) and Google Play Store (https://play.google.com/store/apps/details?id=com.critisense&hl=en).
Read the full abstract ▾
All authors Firoj Alam, Fatema Ahmad, Ali Ezzat Shahroor, Mohamed Bayan Kmainasi, Elisa Sartori, Giovanni Da San Martino, Abul Hasnat, Raian Ali
Computation and Language
Can Linguistically Related Languages Guide LLM Translation in Low-Resource Settings?
Aishwarya Ramasethu, Niyathi Allu, Rohin Garg et al. (+2 more)
Large
Language Models (LLMs) have achieved strong performance across many downstream tasks, yet their effectiveness in extremely low-resource machine translation remains limited. Standard adaptation techniques typically rely on large-scale parallel data or extensive fine-tuning, which are infeasible for the long tail of underrepresented
languages. In this work, we investigate a more constrained question: in data-scarce settings, to what extent can linguistically similar pivot
languages and few-shot demonstrations provide useful guidance for on-the-fly adaptation in LLMs? We study a data-efficient experimental setup that combines linguistically related pivot
languages with few-shot in-context examples, without any parameter updates, and evaluate translation behavior under controlled conditions. Our analysis shows that while pivot-based prompting can yield improvements in certain configurations, particularly in settings where the target
language is less well represented in the model's vocabulary, the gains are often modest and sensitive to few shot example construction. For closely related or better represented varieties, we observe diminishing or inconsistent gains. Our findings provide empirical guidance on how and when inference-time prompting and pivot-based examples can be used as a lightweight alternative to fine-tuning in low-resource translation settings.
Read the full abstract ▾
All authors Aishwarya Ramasethu, Niyathi Allu, Rohin Garg, Harshwardhan Fartale, Dun Li Chan
Computation and Language
Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models
Xiaojie Gu, Sherry T. Tong, Aosong Feng et al. (+8 more)
Evaluating the
reasoning abilities of large
language models (LLMs) solely from final answers can obscure failures in intermediate steps, especially in multi-hop QA
benchmarks without step-level annotations. To address this gap, we introduce Omanic, an open-domain 4-hop QA
benchmark designed not only to measure final-answer accuracy but also to diagnose where
reasoning breaks down. Omanic contains 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench), with each evaluation question decomposed into single-hop sub-questions, intermediate answers, and structured graph
topologies. Experiments with proprietary and open-source LLMs show that Omanic is challenging, while step-wise analysis reveals a later-hop bottleneck, factual knowledge floor, and error propagation along
reasoning chains. Fine-tuning on OmanicSynth transfers to six
reasoning and mathematics
benchmarks, yielding a 7.41-point average gain and validating its effectiveness as supervision for
reasoning-capability transfer. We release the data at https://huggingface.co/
datasets/li-lab/Omanic and the code at https://github.com/XiaojieGu/Omanic.
Read the full abstract ▾
All authors Xiaojie Gu, Sherry T. Tong, Aosong Feng, Sophia Simeng Han, Jinghui Lu, Yingjian Chen, Yusuke Iwasawa, Yutaka Matsuo, Chanjun Park, Rex Ying, Irene Li
Computation and Language
Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy
Zhaoxin Feng, Zheng Chen, Jianfei Ma et al. (+3 more)
Alignment techniques often inadvertently induce sycophancy in LLMs. While prior studies studied this behaviour in direct-answer settings, the role of Chain-of-Thought (CoT)
reasoning remains under-explored: does it serve as a logical constraint that mitigates sycophancy, or a tool for post-hoc rationalization that masks it? We evaluate a range of models across objective and subjective tasks to investigate the issue. Results show that
reasoning generally reduces sycophancy in final decisions but also masks sycophancy in some samples, where models construct deceptive justifications through logical inconsistencies, calculation errors, and one-sided arguments etc. Furthermore, LLMs are more prone to sycophancy in subjective tasks and under authority-bias. Our mechanistic analysis on three open-source models reveals that the tendency of sycophancy is dynamic during the
reasoning process rather than being pre-determined at the input stage.
Read the full abstract ▾
All authors Zhaoxin Feng, Zheng Chen, Jianfei Ma, Yip Tin Po, Emmanuele Chersoni, Bo Li
Computation and Language · Cross-listed from cs.AI
When AI Navigates the Fog of War
Ming Li, Xirui Li and Tianyi Zhou
Can AI reason about a war before its trajectory becomes historically obvious? Analyzing this capability is difficult because retrospective geopolitical prediction is heavily confounded by training-data leakage. We address this challenge through a temporally grounded case study of the early stages of the 2026 Middle East conflict, which unfolded after the training cutoff of current frontier models. We construct 11 critical temporal nodes, 42 node-specific verifiable questions, and 5 general exploratory questions, requiring models to reason only from information that would have been publicly available at each moment. This design substantially mitigates training-data leakage concerns, creating a setting well-suited for studying how models analyze an unfolding crisis under the fog of war, and provides, to our knowledge, the first temporally grounded analysis of LLM
reasoning in an ongoing geopolitical conflict. Our analysis reveals three main findings. First, current state-of-the-art large
language models often display a striking degree of strategic realism,
reasoning beyond surface rhetoric toward deeper structural incentives. Second, this capability is uneven across domains: models are more reliable in economically and logistically structured settings than in politically ambiguous multi-actor environments. Finally, model narratives evolve over time, shifting from early expectations of rapid containment toward more systemic accounts of regional entrenchment and attritional de-escalation. Since the conflict remains ongoing at the time of writing, this work can serve as an archival snapshot of model
reasoning during an unfolding geopolitical crisis, enabling future studies without the hindsight bias of retrospective analysis.
Read the full abstract ▾
Computation and Language
Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model
Ryo Kishino, Riku Shiomi, Hiroaki Yamagiwa et al. (+2 more)
Instead of directly distilling a
language model, this study addresses the problem of aligning a base model with a target model in distribution by designing the domain mixture of training data for pretraining or continued pretraining as a fixed training recipe. We propose a method for determining domain weights by viewing models as points in log-likelihood space and aligning the training update direction with the direction toward the target model. Experiments with NanoGPT show that the proposed method consistently reduces the KL divergence between the trained base model and the target model relative to training with Pile-original weighting. Although knowledge
distillation remains more effective when available, the proposed method achieves meaningful
alignment, and downstream task performance also tends to become closer to that of the target model.
Read the full abstract ▾
All authors Ryo Kishino, Riku Shiomi, Hiroaki Yamagiwa, Momose Oyama, Hidetoshi Shimodaira
Computation and Language
Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech
Omnilingual SONAR Team: João Maria Janeiro, Pere-Lluís Huguet Cabot, Ioannis Tsiamas et al. (+16 more)
Cross-lingual sentence encoders typically cover only a few hundred
languages and often trade downstream quality for stronger
alignment, limiting their adoption. We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-modal sentence
embedding models that natively embed text,
speech, code, and mathematical expressions in a single semantic space, while delivering state-of-the-art downstream performance at the scale of thousands of
languages, from high-resource to extremely low-resource varieties. To reach this scale without representation collapse, we use progressive training. We first learn a strong foundational space for 200
languages with an LLM-initialized encoder-decoder, combining
token-level decoding with a novel split-softmax contrastive loss and synthetic hard negatives. Building on this foundation, we expand to several thousands
language varieties via a two-stage teacher-student encoder
distillation framework. Finally, we demonstrate the cross-modal extensibility of this space by seamlessly mapping 177 spoken
languages into it. OmniSONAR halves cross-lingual similarity search error on the 200-
language FLORES
dataset and reduces error by a factor of 15 on the 1,560-
language BIBLE
benchmark. It also enables strong translation, outperforming NLLB-3B on
multilingual benchmarks and exceeding prior models (including much larger LLMs) by 15 chrF++ points on 1,560
languages into English BIBLE translation. OmniSONAR also performs strongly on MTEB and XLCoST. For
speech, OmniSONAR achieves a 43% lower similarity-search error and reaches 97% of SeamlessM4T
speech-to-text quality, despite being zero-shot for translation (trained only on ASR data). Finally, by training an encoder-decoder LM, Spectrum, exclusively on English text processing OmniSONAR
embedding sequences, we unlock high-performance transfer to thousands of
languages and
speech for complex downstream tasks.
Read the full abstract ▾
All authors Omnilingual SONAR Team: João Maria Janeiro, Pere-Lluís Huguet Cabot, Ioannis Tsiamas, Yen Meng, Vivek Iyer, Guillem Ramírez, Loic Barrault, Belen Alastruey, Xiang "Tony" Cao, Yu-An Chung, Marta R. Costa-Jussa, David Dale, Kevin Heffernan, Jaehyeong Jo, Artyom Kozhevnikov, Alexandre Mourachko, Christophe Ropers, Holger Schwenk, Paul-Ambroise Duquenne
Computation and Language
Tarab: A Multi-Dialect Corpus of Arabic Lyrics and Poetry
Mo El-Haj
We introduce the Tarab Corpus, a large-scale cultural and linguistic resource that brings together Arabic song lyrics and poetry within a unified analytical framework. The corpus comprises 2.56 million verses and more than 13.5 million
tokens, making it, to our knowledge, the largest open Arabic corpus of creative text spanning both classical and contemporary production. Tarab is broadly balanced between songs and poems and covers Classical Arabic, Modern Standard Arabic (MSA), and six major regional varieties: Egyptian, Gulf, Levantine, Iraqi, Sudanese, and Maghrebi Arabic. The artists and poets represented in the corpus are associated with 28 modern nation states and multiple historical eras, covering over fourteen centuries of Arabic creative expression from the Pre-Islamic period to the twenty-first century. Each verse is accompanied by structured metadata describing linguistic variety, geographic origin, and historical or cultural context, enabling comparative linguistic, stylistic, and diachronic analysis across genres and time. We describe the data collection, normalisation, and validation pipeline and present baseline analyses for variety identification and genre differentiation. The
dataset is publicly available on HuggingFace at https://huggingface.co/
datasets/drelhaj/Tarab.
Read the full abstract ▾