Computation and Language
ARC-AGI-2 Technical Report
Wallyson Lemes de Oliveira, Mekhron Bobokhonov, Matteo Caorsi et al. (+10 more)
The Abstraction and
Reasoning Corpus (ARC) is designed to assess generalization beyond pattern matching, requiring models to infer symbolic rules from very few examples. In this work, we present a
transformer-based system that advances ARC performance by combining neural inference with structure-aware priors and online task adaptation. Our approach is built on four key ideas. First, we reformulate ARC
reasoning as a sequence modeling problem using a compact task encoding with only 125
tokens, enabling efficient long-context processing with a modified LongT5 architecture. Second, we introduce a principled augmentation framework based on group
symmetries, grid traversals, and automata perturbations, enforcing invariance to representation changes. Third, we apply test-time training (TTT) with lightweight LoRA adaptation, allowing the model to specialize to each unseen task by learning its transformation logic from demonstrations. Fourth, we design a
symmetry-aware decoding and scoring pipeline that aggregates likelihoods across augmented task views, effectively performing “multi-perspective
reasoning” over candidate solutions. We demonstrate that these components work synergistically: augmentations expand hypothesis space, TTT sharpens local
reasoning, and
symmetry-based scoring improves solution consistency. Our final system achieves a significant improvement over
transformer baselines and surpasses prior neural ARC solvers, closing the gap toward human-level generalization.
Read the full abstract ▾
All authors Wallyson Lemes de Oliveira, Mekhron Bobokhonov, Matteo Caorsi, Aldo Podestà, Gabriele Beltramo, Luca Crosato, Matteo Bonotto, Federica Cecchetto, Hadrien Espic, Dan Titus Salajan, Stefan Taga, Luca Pana, Joe Carthy
Computation and Language
Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge
Fiona Lau
Large
language models are increasingly used as automated evaluators in research and enterprise settings, a practice known as LLM-as-a-judge. While prior work has examined accuracy, bias, and
alignment with human preferences, far less
attention has been given to how consistently LLMs assign numerical scores, an important concern for many production workflows. This study systematically evaluates scoring stability across five commonly used models, GPT-4o, GPT-4o-mini, Gemini-2.5-Flash, Claude-Haiku-4.5, and Claude-Sonnet-4.5, two temperature settings, and real enterprise question-answer pairs drawn from a
retrieval-augmented generation (RAG) system. We address three questions: how stable a model's scores are across repeated runs, how differently models score identical inputs, and how temperature affects scoring consistency. Temperature controls the determinism of an LLM's output. Despite expectations of stability at temperature=0, we observe substantial variability across models, with completeness scoring showing the largest fluctuations. Cross-model comparisons reveal systematic differences in strictness and interpretive style, leading to divergent ratings for the same answers. Lower temperatures improve stability for some models, notably GPT-4o and Gemini, but have limited or inconsistent effects for Anthropic models. These findings have important implications for enterprise pipelines that rely on LLM-generated scores for routing, triage, gating, or quality control. Identical inputs can receive different scores depending on model, family, or temperature, raising concerns around
fairness, reproducibility, and operational reliability. Our results highlight the need for monitoring, robust parsing, and hybrid human-LLM evaluation strategies to ensure dependable use of LLM-as-a-judge in production environments.
Read the full abstract ▾
Computation and Language
Optimizing What We Trust: Reliability-Guided QUBO Selection of Multi-Agent Weak Framing Signals for Arabic Sentiment Prediction
Rabab Alkhalifa
Framing detection in Arabic social media is difficult due to interpretive ambiguity, cultural grounding, and limited reliable supervision. Existing LLM-based weak supervision methods typically rely on label aggregation, which is brittle when annotations are few and socially dependent. We propose a reliability-aware weak supervision framework that shifts the focus from label fusion to data curation. A small multi-
agent LLM pipeline, two framers, a critic, and a discriminator, treats disagreement and
reasoning quality as epistemic signals and produces instance-level reliability estimates. These estimates guide a QUBO-based subset selection procedure that enforces frame balance while reducing redundancy. Intrinsic diagnostics and an out-of-domain Arabic sentiment transfer test show that the selected subsets are more reliable and encode non-random, transferable structure, without degrading strong text-only baselines.
Read the full abstract ▾
Computation and Language
Dual Tuning for Reasoning Efficacy-Driven Data Curation in Multimodal LLM Training
Ruobing Zheng, Tianqi Li, Jianing Li et al. (+3 more)
Reasoning post-training improves Large
Language Models (LLMs) on complex tasks such as mathematics and coding, but its benefits across diverse
multimodal tasks remains uncertain. The trend of releasing parallel "Instruct" and "Thinking" models by leading teams is both resource-intensive and user-unfriendly. Prior work finds that the gains from
reasoning training are influenced by multiple factors, such as base model capabilities, task characteristics, and Chain-of-Thought (CoT) data quality. However, principled criteria for determining when
reasoning post-training is beneficial and which data should support it are still lacking. In this paper, we propose Dual Tuning, a
reasoning efficacy-driven data curation framework for
multimodal LLMs training. Given a target task and a base model, Dual Tuning jointly evaluates whether the training data is beneficial and whether
reasoning training with current CoT content yields positive gains over non-
reasoning alternatives. We apply Dual Tuning across spatial, mathematical, and multi-disciplinary tasks, and further analyze how reinforcement learning and thinking patterns affect
reasoning efficacy. The Dual Tuning results guide data curation by identifying data that benefit
reasoning training, data better suited to direct-answer training, and data that are detrimental under both training modes. Our work provides quantitative criteria for selecting appropriate training data and matching post-training strategies.
Read the full abstract ▾
All authors Ruobing Zheng, Tianqi Li, Jianing Li, Qingpei Guo, Yi Yuan, Jingdong Chen
Computation and Language
GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency
Changhao Wang, Jiaolong Yang, Xinhao Yao et al. (+7 more)
The performance of Large
Language Models (LLMs) is increasingly governed by data efficiency rather than raw scaling volume. However, existing selection methods often decouple global distribution balancing from local instance selection, compromising the hierarchical integrity of the training set. We introduce GRIP (
Geometric Refinement and Adaptive Information Potential), a framework that unifies these dimensions by modeling the corpus as an information-dense
geometric space. GRIP employs a Rapid Adaptation Probe (RAP) to quantify the information potential of semantic clusters, dynamically re-allocating the sampling budget to regions with the highest representation deficits. Subsequently, we perform Intra-Cluster Selection using a length-rectified
geometric prior to counteract
embedding density artifacts and preserve long-tail logical sequences. Extensive evaluations on Mixture-of-Experts (MoE) models up to 300B
tokens demonstrate that GRIP consistently outperforms state-of-the-art baselines, \textbf{surpassing the performance of models trained on $3\times$ larger uncurated
datasets}. Our work establishes a robust
geometric foundation for adaptive data curation in large-scale pre-training.
Read the full abstract ▾
All authors Changhao Wang, Jiaolong Yang, Xinhao Yao, Yunfei Yu, Peng Jiao, Lu Yu, Junpeng Fang, Riccardo Cantoro, Qing Cui, Jun Zhou
Computation and Language
SimpleTool: Parallel Decoding for Real-Time LLM Function Calling
Xiaoxin Shi, Jiaxin Wan, Linkang Dong et al. (+3 more)
LLM-based function calling enables intelligent
agents to interact with external tools and environments, yet autoregressive decoding imposes a fundamental latency bottleneck that limits real-time applications such as
embodied intelligence, game AI, and interactive avatars (e.g., 10 Hz control frequency). We observe that function calling differs fundamentally from free-form text generation: structured outputs exhibit substantial
token redundancy (delimiters, parameter names), and arguments exhibit weak causal dependencies. Crucially, these two properties must be exploited jointly to achieve real-time performance. We present SimpleTool, which introduces special
tokens that serve a dual role: compressing low-
entropy tokens (4-6x reduction) while acting as mode selectors that enable independent parallel generation of function name and arguments. This synergistic design achieves 3-6x end-to-end speedup (up to 9.6x) with only +8.2% parallelization overhead. Experiments on five
benchmarks across Qwen-series models (0.5B-14B) demonstrate substantial speedup while maintaining competitive or improved accuracy. On Mobile Actions, ST-Qwen-0.5B outperforms Google's FunctionGemma in both accuracy and latency consistency. With quantization on consumer-grade GPU, SimpleTool achieves 61.2ms P50 latency, enabling 16 Hz real-time control at 4B model scale, bridging the gap between LLM function calling and latency-critical real-world deployment.
Read the full abstract ▾
All authors Xiaoxin Shi, Jiaxin Wan, Linkang Dong, Wei Jiang, Yue Liu, Zengfeng Huang
Computation and Language
Embracing Anisotropy: Turning Massive Activations into Interpretable Control Knobs for Large Language Models
Youngji Roh, Hyunjin Cho and Jaehyung Kim
Large
Language Models (LLMs) exhibit highly anisotropic internal representations, often characterized by massive activations, a phenomenon where a small subset of feature dimensions possesses magnitudes significantly larger than the rest. While prior works view these extreme dimensions primarily as artifacts to be managed, we propose a distinct perspective: these dimensions serve as intrinsic interpretable functional units arising from domain specialization. Specifically, we propose a simple magnitude-based criterion to identify Domain-Critical Dimensions in a training-free manner. Our analyses reveal that such dimensions behave as interpretable semantic detectors for symbolic/quantitative patterns or domain-specific terms. In addition, we introduce Critical Dimension Steering, which applies activation steering exclusively to the identified dimensions. Empirical results show that this approach outperforms conventional whole-dimension steering in domain adaptation and jailbreaking scenarios.
Read the full abstract ▾
Computation and Language
EPPCMinerBen: A Novel Benchmark for Evaluating Large Language Models on Electronic Patient-Provider Communication via the Patient Portal
Samah Fodeh, Yan Wang, Linhai Ma et al. (+3 more)
Effective communication in health care is critical for treatment outcomes and adherence. With patient-provider exchanges shifting to secure messaging, analyzing electronic patient-communication (EPPC) data is both essential and challenging. We introduce EPPCMinerBen, a
benchmark for evaluating LLMs in detecting communication patterns and extracting insights from electronic patient-provider messages. EPPCMinerBen includes three sub-tasks: Code Classification, Subcode Classification, and Evidence Extraction. Using 1,933 expert annotated sentences from 752 secure messages of the patient portal at Yale New Haven Hospital, it evaluates LLMs on identifying communicative intent and supportive text.
Benchmarks span various LLMs under zero-shot and few-shot settings, with data to be released via the NCI Cancer Data Service. Model performance varied across tasks and settings. Llama-3.1-70B led in evidence extraction (F1: 82.84%) and performed well in classification. Llama-3.3-70b-Instruct outperformed all models in code classification (F1: 67.03%). DeepSeek-R1-Distill-Qwen-32B excelled in subcode classification (F1: 48.25%), while sdoh-llama-3-70B showed consistent performance. Smaller models underperformed, especially in subcode classification (>30% F1). Few-shot prompting improved most tasks. Our results show that large, instruction-tuned models generally perform better in EPPCMinerBen tasks, particularly evidence extraction while smaller models struggle with fine-grained
reasoning. EPPCMinerBen provides a
benchmark for discourse-level understanding, supporting future work on model generalization and patient-provider communication analysis. Keywords: Electronic Patient-Provider Communication, Large
language models, Data collection,
Prompt engineering
Read the full abstract ▾
All authors Samah Fodeh, Yan Wang, Linhai Ma, Srivani Talakokkul, Jordan M. Alpert, Sarah Schellhorn
Computation and Language
Xiaohui Zhang, Zequn Sun, Chengyuan Yang et al. (+3 more)
Memory management is essential for LLM
agents in long-term interactions. Current
memory frameworks typically treat
agents as passive “recorders” and retrieve information without understanding its deeper implications. They may fail in scenarios requiring
reasoning and complex decision-making. To bridge this critical gap, we propose a novel actionable
memory framework called ActMem that integrates
memory retrieval with active causal
reasoning. ActMem transforms unstructured dialogue history into a structured causal and semantic graph. By leveraging counterfactual
reasoning and commonsense completion, it enables
agents to deduce implicit constraints and resolve potential conflicts between past states and current intentions. Furthermore, we introduce a comprehensive
dataset ActMemEval to evaluate
agent reasoning capabilities in logic-driven scenarios, moving beyond the fact-
retrieval focus of existing
memory benchmarks. Experiments demonstrate that ActMem significantly outperforms baselines in handling complex,
memory-dependent tasks, paving the way for more consistent and reliable intelligent assistants.
Read the full abstract ▾
All authors Xiaohui Zhang, Zequn Sun, Chengyuan Yang, Yaqin Jin, Yazhong Zhang, Wei Hu
Computation and Language · Cross-listed from q-fin.GN
Sumin Kim, Jihoon Kwon, Yoon Kim et al. (+9 more)
Mention
markets, a type of prediction
market in which contracts resolve based on whether a specified keyword is mentioned during a future public event, require accurate probabilistic forecasts of keyword-mention outcomes. While recent work shows that large
language models (LLMs) can generate forecasts competitive with human forecasters, it remains unclear how input context should be designed to support accurate prediction. In this paper, we study this question through experiments on earnings-call mention
markets, which require
forecasting whether a company will mention a specified keyword during its upcoming call. We run controlled comparisons varying (i) which contextual information is provided (news and/or prior earnings-call transcripts) and (ii) how
market probability, (i.e., prediction
market contract price) is used. We introduce
Market-Conditioned Prompting (MCP), which explicitly treats the
market-implied probability as a prior and instructs the LLM to update this prior using textual evidence, rather than re-predicting the base rate from scratch. In our experiments, we find three insights: (1) richer context consistently improves
forecasting performance; (2)
market-conditioned prompting (MCP), which treats the
market probability as a prior and updates it using textual evidence, yields better-calibrated forecasts; and (3) a mixture of the
market probability and MCP (MixMCP) outperforms the
market baseline. By dampening the LLM's posterior update with the
market prior, MixMCP yields more robust predictions than either the
market or the LLM alone.
Read the full abstract ▾
All authors Sumin Kim, Jihoon Kwon, Yoon Kim, Nicole Kagan, Raffi Khatchadourian, Wonbin Ahn, Alejandro Lopez-Lira, Jaewon Lee, Yoontae Hwang, Oscar Levy, Yongjae Lee, Chanyeol Choi
Computation and Language
ImpRIF: Stronger Implicit Reasoning Leads to Better Complex Instruction Following
Yuancheng Yang, Lin Yang, Xu Wang et al. (+2 more)
As applications of large
language models (LLMs) become increasingly complex, the demand for robust complex instruction following capabilities is growing accordingly. We argue that a thorough understanding of the instruction itself, especially the latent
reasoning structure embedded between the lines, is crucial for improving instruction following. Therefore we target complex instructions that involve implicit
reasoning, intricate logical relations, and multi-constraint dependencies. We propose ImpRIF, a method to enhance LLMs' understanding of implicit
reasoning instructions, thereby improving its ability to follow complex instructions. We formalize such instructions as verifiable
reasoning graphs, enabling programmatic verification and graph-driven chain-of-thought
reasoning. Based on this formulation, we synthesize large-scale single- and multi-turn data, propose fine-tuning with graph
reasoning, and apply reinforcement learning to explicitly train models to reason along the graph. On five complex instruction following
benchmarks, our models substantially outperform their base models. These results demonstrate that enhancing implicit
reasoning capabilities can significantly improve complex instruction following.
Read the full abstract ▾
All authors Yuancheng Yang, Lin Yang, Xu Wang, Chao Tong, Haihua Yang
Computation and Language
Budget-Aware Agentic Routing via Boundary-Guided Training
Caiqi Zhang, Menglin Xia, Xuchao Zhang et al. (+5 more)
As large
language models (LLMs) evolve into autonomous
agents that execute long-horizon workflows, invoking a high-capability model at every step becomes economically unsustainable. While model routing is effective for single-turn queries, agentic routing is a sequential, path-dependent problem: early mistakes compound, feedback is often at the end of the episode, and deployments often demand strict per-task spending limits. We propose Budget-Aware Agentic Routing, which selects between a cheap and an expensive model at each step to optimize the cost–success frontier and to operate under strict per-task budgets. We propose Boundary-Guided Training, which leverages two boundary policies (always-small vs.\ always-large) to build a difficulty taxonomy and to anchor learning under sparse rewards. Our approach warms start with boundary-guided SFT data synthesis via stratified sampling of cost-efficient trajectories, then applies Boundary-Guided Policy Optimization (BoPO), combining boundary-relative rewards with a reference-guided advantage to avoid degenerate cheap-failure solutions. Experiment results show that our method improves the efficiency frontier, matching strong routing baselines at substantially lower cost while demonstrating generalization to strict inference-time budget constraints. Overall, our work establishes a foundational framework for agentic routing, shifting the paradigm from static model selection to dynamic, budget-aware sequential decision-making.
Read the full abstract ▾
All authors Caiqi Zhang, Menglin Xia, Xuchao Zhang, Daniel Madrigal, Ankur Mallick, Samuel Kessler, Victor Ruehle, Saravan Rajmohan
Computation and Language · Cross-listed from cs.CY
Exploring the Ethical Concerns in User Reviews of Mental Health Apps using Topic Modeling and Sentiment Analysis
Mohammad Masudur Rahman and Beenish Moalla Chaudhry
The rapid growth of AI-driven mental health mobile apps has raised concerns about their ethical considerations and user trust. This study proposed a natural
language processing (NLP)-based framework to evaluate ethical aspects from user-generated reviews from the Google Play Store and Apple App Store. After gathering and cleaning the data, topic modeling was applied to identify latent themes in the context of ethics using topic words and then map them to well-recognized existing ethical principles described in different ethical frameworks; in addition to that, a bottom-up approach is applied to find any new and emergent ethics from the reviews using a
transformer-based zero-shot classification model. Sentiment analysis was then used to capture how users feel about each ethical aspect. The obtained results reveal that well-known ethical considerations are not enough for the modern AI-based technologies and are missing emerging ethical challenges, showing how these apps either uphold or overlook key moral values. This work contributes to developing an ongoing evaluation system that can enhance the
fairness, transparency, and trustworthiness of AI-powered mental health chatbots.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
LATMiX: Learnable Affine Transformations for Microscaling Quantization of LLMs
Ofir Gordon, Lior Dikstein, Arnon Netzer et al. (+2 more)
Post-training quantization (PTQ) is a widely used approach for reducing the
memory and compute costs of large
language models (LLMs). Recent studies have shown that applying invertible transformations to activations can significantly improve quantization robustness by reducing activation outliers; however, existing approaches are largely restricted to rotation or Hadamard-based transformations. Moreover, most studies focused primarily on traditional quantization schemes, whereas modern hardware increasingly supports the microscaling (MX) data format. Attempts to combine both showed severe performance degradation, leading prior work to introduce assumptions on the transformations. In this work, we take a complementary perspective. First, we provide a theoretical analysis of transformations under MX quantization by deriving a bound on the quantization error. Our analysis emphasizes the importance of accounting for both the activation distribution and the underlying quantization structure. Building on this analysis, we propose LATMiX, a method that generalizes outlier reduction to learnable invertible affine transformations optimized using standard deep learning tools. Experiments show consistent improvements in average accuracy for MX low-bit quantization over strong baselines on a wide range of zero-shot
benchmarks, across multiple model sizes.
Read the full abstract ▾
All authors Ofir Gordon, Lior Dikstein, Arnon Netzer, Idan Achituve, Hai Victor Habi
Computation and Language · Cross-listed from cs.LG
Directional Concentration Uncertainty: A representational approach to uncertainty quantification for generative models
Souradeep Chattopadhyay, Brendan Kennedy, Sai Munikoti et al. (+2 more)
In the critical task of making generative models trustworthy and robust, methods for Uncertainty Quantification (UQ) have begun to show encouraging potential. However, many of these methods rely on rigid heuristics that fail to generalize across tasks and modalities. Here, we propose a novel framework for UQ that is highly flexible and approaches or surpasses the performance of prior heuristic methods. We introduce Directional Concentration Uncertainty (DCU), a novel statistical procedure for quantifying the concentration of
embeddings based on the von Mises-Fisher (vMF) distribution. Our method captures uncertainty by measuring the
geometric dispersion of multiple generated outputs from a
language model using continuous
embeddings of the generated outputs without any task specific heuristics. In our experiments, we show that DCU matches or exceeds calibration levels of prior works like semantic
entropy (Kuhn et al., 2023) and also generalizes well to more complex tasks in multi-modal domains. We present a framework for the wider potential of DCU and its implications for integration into UQ for multi-modal and agentic frameworks.
Read the full abstract ▾
All authors Souradeep Chattopadhyay, Brendan Kennedy, Sai Munikoti, Soumik Sarkar, Karl Pazdernik
Computation and Language
Mechanistic Evidence for Faithfulness Decay in Chain-of-Thought Reasoning
Donald Ye, Max Loffgren, Om Kotadia et al. (+2 more)
Chain-of-Thought (CoT) explanations are widely used to interpret how
language models solve complex problems, yet it remains unclear whether these step-by-step explanations reflect how the model actually reaches its answer, or merely post-hoc justifications. We propose Normalized Logit Difference Decay (NLDD), a metric that measures whether individual
reasoning steps are faithful to the model's decision-making process. Our approach corrupts individual
reasoning steps from the explanation and measures how much the model's confidence in its answer drops, to determine if a step is truly important. By standardizing these measurements, NLDD enables rigorous cross-model comparison across different architectures. Testing three model families across syntactic, logical, and arithmetic tasks, we discover a consistent
Reasoning Horizon (k*) at 70–85% of chain length, beyond which
reasoning tokens have little or negative effect on the final answer. We also find that models can encode correct internal representations while completely failing the task. These results show that accuracy alone does not reveal whether a model actually reasons through its chain. NLDD offers a way to measure when CoT matters.
Read the full abstract ▾
All authors Donald Ye, Max Loffgren, Om Kotadia, Linus Wong, Jonas Rohweder
Computation and Language
When and What to Ask: AskBench and Rubric-Guided RLVR for LLM Clarification
Jiale Zhao, Ke Fang and Lu Cheng
Large
language models (LLMs) often respond even when
prompts omit critical details or include misleading information, leading to
hallucinations or reinforced misconceptions. We study how to evaluate and improve LLMs' ability to decide when and what to ask for clarification without sacrificing task performance. We introduce AskBench, an interactive
benchmark that converts standard QA pairs into multi-turn interactions with explicit checkpoints. A unified judge loop evaluates final answers and simulates user responses as needed. AskBench covers two settings: AskMind, with intent-deficient queries requiring clarification, and AskOverconfidence, with queries containing false premises that must be identified and corrected. We further propose rubric-guided reinforcement learning with verifier-based rewards (RLVR), which uses structured rubrics to encourage targeted clarification. Experiments show consistent improvements in accuracy, rubric adherence, and interaction efficiency, with strong generalization to unseen domains.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Reliable and Responsible Foundation Models: A Comprehensive Survey
Xinyu Yang, Junlin Han, Rishi Bommasani et al. (+49 more)
Foundation models, including Large
Language Models (LLMs),
Multimodal Large
Language Models (MLLMs), Image Generative Models (i.e, Text-to-Image Models and Image-Editing Models), and
Video Generative Models, have become essential tools with broad applications across various domains such as law, medicine, education, finance, science, and beyond. As these models see increasing real-world deployment, ensuring their reliability and responsibility has become critical for academia, industry, and government. This survey addresses the reliable and responsible development of foundation models. We explore critical issues, including bias and
fairness, security and
privacy, uncertainty, explainability, and distribution shift. Our research also covers model limitations, such as
hallucinations, as well as methods like
alignment and Artificial Intelligence-Generated Content (AIGC) detection. For each area, we review the current state of the field and outline concrete future research directions. Additionally, we discuss the intersections between these areas, highlighting their connections and shared challenges. We hope our survey fosters the development of foundation models that are not only powerful but also ethical, trustworthy, reliable, and socially responsible.
Read the full abstract ▾
All authors Xinyu Yang, Junlin Han, Rishi Bommasani, Jinqi Luo, Wenjie Qu, Wangchunshu Zhou, Adel Bibi, Xiyao Wang, Jaehong Yoon, Elias Stengel-Eskin, Shengbang Tong, Lingfeng Shen, Rafael Rafailov, Runjia Li, Zhaoyang Wang, Yiyang Zhou, Chenhang Cui, Yu Wang, Wenhao Zheng, Huichi Zhou, Jindong Gu, Zhaorun Chen, Peng Xia, Tony Lee, Thomas Zollo, Vikash Sehwag, Jixuan Leng, Jiuhai Chen, Yuxin Wen, Huan Zhang, Zhun Deng, Linjun Zhang, Pavel Izmailov, Pang Wei Koh, Yulia Tsvetkov, Andrew Wilson, Jiaheng Zhang, James Zou, Cihang Xie, Hao Wang, Philip Torr, Julian McAuley, David Alvarez-Melis, Florian Tramèr, Kaidi Xu, Suman Jana, Chris Callison-Burch, Rene Vidal, Filippos Kokkinos, Mohit Bansal, Beidi Chen, Huaxiu Yao
Computation and Language · Cross-listed from cs.AI
Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?
Pingyue Zhang, Zihan Huang, Yue Wang et al. (+11 more)
Spatial
embodied intelligence requires
agents to act to acquire information under partial observability. While
multimodal foundation models excel at passive perception, their capacity for active, self-directed exploration remains understudied. We propose Theory of Space, defined as an
agent's ability to actively acquire information through self-directed, active exploration and to construct, revise, and exploit a spatial belief from sequential, partial observations. We evaluate this through a
benchmark where the goal is curiosity-driven exploration to build an accurate cognitive map. A key innovation is spatial belief probing, which
prompts models to reveal their internal spatial representations at each step. Our evaluation of state-of-the-art models reveals several critical bottlenecks. First, we identify an Active-Passive Gap, where performance drops significantly when
agents must autonomously gather information. Second, we find high inefficiency, as models explore unsystematically compared to program-based proxies. Through belief probing, we diagnose that while perception is an initial bottleneck, global beliefs suffer from instability that causes spatial knowledge to degrade over time. Finally, using a false belief paradigm, we uncover Belief Inertia, where
agents fail to update obsolete priors with new evidence. This issue is present in text-based
agents but is particularly severe in
vision-based models. Our findings suggest that current foundation models struggle to maintain coherent, revisable spatial beliefs during active exploration.
Read the full abstract ▾
All authors Pingyue Zhang, Zihan Huang, Yue Wang, Jieyu Zhang, Letian Xue, Zihan Wang, Qineng Wang, Keshigeyan Chandrasegaran, Ruohan Zhang, Yejin Choi, Ranjay Krishna, Jiajun Wu, Li Fei-Fei, Manling Li
Computation and Language · Cross-listed from cs.AI
SocialVeil: Probing Social Intelligence of Language Agents under Communication Barriers
Keyang Xuan, Pengda Wang, Chongrui Ye et al. (+3 more)
Large
language models (LLMs) are increasingly evaluated in interactive environments to test their social intelligence. However, existing
benchmarks often assume idealized communication between
agents, limiting our ability to diagnose whether LLMs can maintain and repair interactions in more realistic, imperfect settings. To close this gap, we present SocialVeil, a social learning environment that can simulate social interaction under cognitive-difference-induced communication barriers. Grounded in a systematic literature review of communication challenges in human interaction, SocialVeil introduces three representative types of such disruption, semantic vagueness, sociocultural mismatch, and emotional interference. We also introduce two barrier-aware evaluation metrics, unresolved confusion and mutual understanding, to evaluate interaction quality under impaired communication. Experiments across 720 scenarios and four frontier LLMs show that barriers consistently impair performance, with mutual understanding reduced by over 45% on average, and confusion elevated by nearly 50%. Human evaluations validate the fidelity of these simulated barriers (ICC$\approx$0.78, Pearson r$\approx$0.80). We further demonstrate that adaptation strategies (Repair Instruction and Interactive learning) only have a modest effect far from barrier-free performance. This work takes a step toward bringing social interaction environments closer to real-world communication, opening opportunities for exploring the social intelligence of LLM
agents.
Read the full abstract ▾
All authors Keyang Xuan, Pengda Wang, Chongrui Ye, Haofei Yu, Tal August, Jiaxuan You
Computation and Language
Multilingual Extraction and Recognition of Implicit Discourse Relations in Speech and Text
Ahmed Ruby, Christian Hardmeier and Sara Stymne
Implicit discourse relation classification is a challenging task, as it requires inferring meaning from context. While contextual cues can be distributed across modalities and vary across
languages, they are not always captured by text alone. To address this, we introduce an automatic method for distantly related and unrelated
language pairs to construct a
multilingual and
multimodal dataset for implicit discourse relations in English, French, and Spanish. For classification, we propose a
multimodal approach that integrates textual and acoustic information through Qwen2-Audio, allowing joint modeling of text and audio for implicit discourse relation classification across
languages. We find that while text-based models outperform audio-based models, integrating both modalities can enhance performance, and cross-lingual transfer can provide substantial improvements for low-resource
languages.
Read the full abstract ▾
Computation and Language
Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Models
Michael Browder, Kevin Duh, J. David Harris et al. (+5 more)
Scarcity of labeled training data remains the long pole in the tent for building performant
language technology and generative AI models.
Transformer models – particularly LLMs – are increasingly being used to mitigate the data scarcity problem via synthetic data generation. However, because the models are black boxes, the properties of the synthetic data are difficult to predict. In practice it is common for
language technology engineers to 'fiddle' with the LLM temperature setting and hope that what comes out the other end improves the downstream model. Faced with this uncertainty, here we propose Data Kernel Perspective Space (DKPS) to provide the foundation for mathematical analysis yielding concrete statistical guarantees for the quality of the outputs of
transformer models. We first show the mathematical derivation of DKPS and how it provides performance guarantees. Next we show how DKPS performance guarantees can elucidate performance of a downstream task, such as neural machine translation models or LLMs trained using Contrastive Preference Optimization (CPO). Limitations of the current work and future research are also discussed.
Read the full abstract ▾
All authors Michael Browder, Kevin Duh, J. David Harris, Vince Lyzinski, Paul McNamee, Youngser Park, Carey E. Priebe, Peter Viechnicki
Computation and Language
Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories
Sidi Lu, Zhenwen Liang, Dongyang Ma et al. (+3 more)
In this paper, we aim to bridge test-time-training with a new type of parametric
memory that can be flexibly offloaded from or merged into model parameters. We present Locas, a Locally-Supported parametric
memory that shares the design of FFN blocks in modern
transformers, allowing it to be flexibly permanentized into the model parameters while supporting efficient continual learning. We discuss two major variants of Locas: one with a conventional two-layer MLP design that has a clearer theoretical guarantee; the other one shares the same GLU-FFN structure with SOTA LLMs, and can be easily attached to existing models for both parameter-efficient and computation-efficient continual learning. Crucially, we show that proper initialization of such low-rank sideway-FFN-style
memories – performed in a principled way by reusing model parameters, activations and/or gradients – is essential for fast convergence, improved generalization, and catastrophic forgetting prevention. We validate the proposed
memory mechanism on the PG-19 whole-book
language modeling and LoCoMo long-context dialogue question answering tasks. With only 0.02% additional parameters in the lowest case, Locas-GLU is capable of storing the information from past context while maintaining a much smaller context window. In addition, we also test the model's general capability loss after memorizing the whole book with Locas, through comparative MMLU evaluation. Results show the promising ability of Locas to permanentize past context into parametric knowledge with minimized catastrophic forgetting of the model's existing internal knowledge.
Read the full abstract ▾
All authors Sidi Lu, Zhenwen Liang, Dongyang Ma, Yan Wang, Haitao Mi, Dong Yu
Computation and Language · Cross-listed from cs.LG
StagePilot: Stage-Level Planning for Long-Horizon Dialogue Simulation in Cybergrooming
Heajun An, Qi Zhang, Minqian Liu et al. (+5 more)
Cybergrooming is an evolving threat to youth, requiring proactive educational interventions. We address this by modeling dialogue progression as a structured planning problem over stage-wise interactions. We propose StagePilot, a dialogue framework that separates stage-level planning from response generation, in which the model selects the next stage under constrained transitions and generates responses conditioned on it, enabling coherent and realistic progression. Reinforcement learning is used to learn stage-level policies from offline data, optimizing for both emotional
alignment and goal-consistent progression. Our empirical experiments show that StagePilot generates more structured, coherent dialogue trajectories and reduces conversational stagnation compared to baselines; notably, the IQL+AWAC variant reaches the final stage more often while maintaining over 70% positive or neutral responses, yielding a 43% relative improvement.
Read the full abstract ▾
All authors Heajun An, Qi Zhang, Minqian Liu, Xinyi Zhang, Sang Won Lee, Lifu Huang, Pamela J. Wisniewski, Jin-Hee Cho
Computation and Language · Cross-listed from cs.CR
Grounded but Misleading: Evaluating Semantic Alignment in AI-Generated Security Explanations
Heajun An, Connor Ng, Sandesh Sharma Dulal et al. (+2 more)
Online scams increasingly leverage fluent and context-aware social engineering strategies, creating growing demand for AI systems that explain why a message may be risky. However, explanations that cite detector-derived evidence may still semantically weaken or redirect the intended risk interpretation. We introduce VEXA: Verifying Semantic Explanation
Alignment, a controlled testbed for studying the gap between lexical grounding and semantic risk
alignment in AI-generated scam-risk explanations. VEXA generates ungrounded, risk-aligned, and risk-diluting explanations by independently controlling evidence grounding and semantic framing. Through LLM-as-a-judge and human evaluations, we show that explanations may continue to appear comparatively grounded even when their semantic interpretation weakens the detector's intended risk assessment. In human evaluation, risk-diluting XAI-grounded explanations retained comparatively elevated Perceived Evidence Grounding scores (3.66) despite lower Helpfulness (3.00) and
Reasoning Support (3.14) scores. These findings provide controlled evidence of grounding illusion effects in AI-generated security explanations and suggest that trustworthy explanation evaluation must verify not only whether evidence is cited, but also how that evidence is interpreted.
Read the full abstract ▾
All authors Heajun An, Connor Ng, Sandesh Sharma Dulal, Junghwan Kim, Jin-Hee Cho
Computation and Language
Capacity Constraints and the Multilingual Penalty for Lexical Disambiguation
Sean Trott and Pamela D. Rivière
Multilingual language models (LMs) sometimes under-perform their monolingual counterparts, possibly due to capacity limitations. We quantify this “
multilingual penalty” for lexical disambiguation–a task requiring precise semantic representations and contextualization mechanisms–using controlled
datasets of human relatedness judgments for ambiguous words in both English and Spanish. Comparing monolingual and
multilingual LMs from the same families, we find consistently reduced performance in
multilingual LMs. We then explore three potential capacity constraints: representational (reduced
embedding isotropy), attentional (reduced
attention to disambiguating cues), and vocabulary-related (increased multi-
token segmentation).
Multilingual LMs show some evidence of all three limitations; moreover, these factors statistically account for the variance formerly attributed to a model's
multilingual status. These findings suggest both that
multilingual LMs do suffer from multiple capacity constraints, and that these constraints correlate with reduced disambiguation performance.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
DeepRead: Document Structure-Aware Reasoning to Enhance Agentic Search
Zhanli Li, Huiwen Tian, Lvzhou Luo et al. (+2 more)
With the rapid advancement of tool-use capabilities in Large
Language Models (LLMs),
Retrieval-Augmented Generation (RAG) is shifting from static, one-shot
retrieval toward autonomous, multi-turn evidence acquisition. However, existing agentic search frameworks typically treat long documents as flat collections of unstructured chunks, disregarding the native hierarchical organization and sequential logic essential for human comprehension. To bridge this gap, we introduce DeepRead, a structure-aware document
reasoning agent designed to operationalize document-native structural priors into actionable
reasoning capabilities. Leveraging the structural fidelity of modern OCR, DeepRead constructs a paragraph-level, coordinate-based
navigation system and equips the LLM with two synergistic tools: Retrieve for scanning-aware localization, and ReadSection for contiguous, order-preserving reading within specific hierarchical scopes. This design elicits a human-like “locate-then-read”
reasoning paradigm, effectively mitigating the context fragmentation inherent in traditional
retrieval methods. Extensive evaluations across four
benchmarks spanning diverse document types demonstrate that DeepRead outperforms Search-o1-style agentic search baselines by an average of 10.3%. Fine-grained behavioral analysis further confirms that DeepRead autonomously adopts human-aligned reading strategies, validating the critical role of structural awareness in achieving precise document
reasoning. Our code is available at https://github.com/Zhanli-Li/DeepRead.
Read the full abstract ▾
All authors Zhanli Li, Huiwen Tian, Lvzhou Luo, Yixuan Cao, Ping Luo
Computation and Language · Cross-listed from cs.LG
Enhanced QKNorm normalization for neural transformers with the Lp norm
Ezequiel Lopez-Rubio, Javier Montes-Perez and Esteban Jose Palomo
The normalization of query and key vectors is an essential part of the
Transformer architecture. It ensures that learning is stable regardless of the scale of these vectors. Some normalization approaches are available. In this preliminary work, a generalization of the QKNorm normalization scheme is proposed. The approach is based on the Lp norm, allowing non-Euclidean norms to be employed. Experimental results demonstrate the suitability of the method for a simple problem.
Read the full abstract ▾
Computation and Language
CoWork-X: Experience-Optimized Co-Evolution for Multi-Agent Collaboration System
Zexin Lin, Jiachen Yu, Haoyang Zhang et al. (+5 more)
Large
language models are enabling
language-conditioned
agents in interactive environments, but highly cooperative tasks often impose two simultaneous constraints: sub-second real-time coordination and sustained multi-episode adaptation under a strict online
token budget. Existing approaches either rely on frequent in-episode
reasoning that induces latency and timing jitter, or deliver post-episode improvements through unstructured text that is difficult to compile into reliable low-cost execution. We propose CoWork-X, an active co-
evolution framework that casts peer collaboration as a closed-loop optimization problem across episodes, inspired by fast–slow
memory separation. CoWork-X instantiates a Skill-
Agent that executes via HTN (hierarchical task network)-based skill
retrieval from a structured, interpretable, and compositional skill library, and a post-episode Co-Optimizer that performs patch-style skill consolidation with explicit budget constraints and drift regularization. Experiments in challenging Overcooked-AI-like realtime collaboration
benchmarks demonstrate that CoWork-X achieves stable, cumulative performance gains while steadily reducing online latency and
token usage.
Read the full abstract ▾
All authors Zexin Lin, Jiachen Yu, Haoyang Zhang, Yuzhao Li, Zhonghang Li, Yujiu Yang, Junjie Wang, Xiaoqiang Ji
Computation and Language · Cross-listed from cs.LG
Atula Tejaswi, Litu Rout, Constantine Caramanis et al. (+2 more)
Reward guidance, also known as posterior sampling, is a popular method for test-time adaptation and post-training in continuous
diffusion models. In this paper, we study reward guidance for discrete
diffusion language models; now, one cannot differentiate through the natural outputs of the model because they are discrete
tokens. We introduce a novel mechanism called EntRGi (
Entropy aware Reward Guidance) to address this issue. EntRGi dynamically interpolates between continuous
token relaxations and sampled hard
tokens, on a
token-by-
token basis, using the
diffusion model's predictive
entropy. We demonstrate that EntRGi maintains both reward model reliability and optimization accuracy, while existing approaches sacrifice one for the other. We empirically validate our approach on 7B-parameter
diffusion language models across two settings: (1) test-time adaptation, and (2) RGRL (Reward Guided Reinforcement Learning), our recipe for post-training on reward-guided data, showing consistent improvements over state-of-the-art methods. Our code is available at https://atutej.github.io/entrgi-rgrl
Read the full abstract ▾
All authors Atula Tejaswi, Litu Rout, Constantine Caramanis, Sanjay Shakkottai, Sujay Sanghavi