Computation and Language
Generative Models Enhanced by Sequence Labelling and Aspect-Code Switching Improve Cross-lingual Aspect-Based Sentiment Analysis
Jakub Šmíd, Pavel Přibáň and Pavel Král
Cross-lingual aspect-based sentiment analysis (ABSA) transfers knowledge from a source
language with annotated data to a target
language, enabling fine-grained sentiment analysis without annotated target-
language data. While monolingual ABSA has seen significant progress, cross-lingual ABSA remains underexplored, especially for complex tasks involving multiple sentiment elements like target-aspect-sentiment detection (TASD). In this paper, we propose a novel SeqLab framework that enhances cross-lingual ABSA using a sequence-to-sequence model with an auxiliary sequence-labelling task performed by the encoder, enhancing aspect term recognition and sentiment predictions. Additionally, we incorporate aspect-code switching (ACS), a translation-based technique that swaps aspect terms between source and translated sentences, generating additional training data to enhance the model's cross-lingual understanding. We evaluate our approach across eleven
languages, three domains, and two backbone models, surpassing previous state-of-the-art results for the commonly studied E2E-ABSA task. Unlike most prior work that relies solely on English as the source
language, we systematically assess different source-target
language pairs and extend our evaluation to the more challenging, yet underexplored TASD task in cross-lingual settings. Finally, we provide a detailed error analysis highlighting key challenges and limitations.
Read the full abstract ▾
Computation and Language
SemPOI-RL: Aligning LLM Semantic Reasoning for Interpretable Out-of-Town POI Sequential Generation
Yunqi Liu, Yang Zhang, Ruixing Zhang et al. (+4 more)
Large
language models (LLMs) exhibit strong semantic
reasoning and open-ended generation abilities, but aligning these abilities with structured sequential generation remains challenging. This challenge is particularly evident in out-of-town (OOT) POI sequence generation, where a model must infer transferable travel intent from a user's hometown behaviors, adapt to cross-city interest drift, and generate a coherent destination trajectory under structural constraints. Existing approaches either rely on latent ID-based transfer with limited interpretability or directly use LLMs for sequence generation without explicitly grounding inferred semantics into position-aware predictions. To address this gap, we propose SemPOI-RL, a framework that aligns LLM semantic
reasoning with structured sequence generation for interpretable OOT recommendation. Specifically, we first fine-tune an LLM to infer destination-oriented travel styles from users' hometown trajectories, using natural
language as an interpretable semantic intermediate. We then introduce a Semantic POI
Alignment Module (SPAM) to ground these inferred styles into a style-conditioned masked autoencoder for position-aware trajectory generation. Finally, we apply reinforcement learning with recommendation-oriented rewards to align LLM-generated styles with downstream sequence quality. Experiments on two real-world
datasets show that SemPOI-RL consistently outperforms both traditional recommenders and direct LLM baselines, while providing interpretable style attribution across different phases of a trip. The code is available at https://github.com/Wind-Flipped/SemPOI-RL .
Read the full abstract ▾
All authors Yunqi Liu, Yang Zhang, Ruixing Zhang, Liangzhe Han, Yi Qiao, Tongyu Zhu, Leilei Sun
Computation and Language
Beyond Polarization: The Generative Constraint of Chain-of-Thought in Pointwise Reranking
Xiaoyang Chen, Jie Liu, Haijin Liang et al. (+5 more)
In pointwise document reranking, Chain-of-Thought models typically underperform direct scoring models. While existing diagnostics attribute this to inferior classification, score polarization, or calibration breakdown, whether targeted training can bridge this gap remains unclear. Our empirical study first confirms that this gap is stable across scales up to 32B parameters, ruling out model and data capacity confounders. We then apply stress tests utilizing reinforcement learning, fine-grained supervision, and architectural decoupling to explicitly repair these deviations. Although these interventions improve classification accuracy and absolute scores, the relative ranking gap persists. These findings suggest that, within the pointwise scoring paradigm, routing continuous relevance semantics through discrete text constrains ranking signal resolution, revealing a bottleneck that is stable and difficult to overcome under current standard methods, rather than an easily resolvable training bias.
Read the full abstract ▾
All authors Xiaoyang Chen, Jie Liu, Haijin Liang, Haibo Shi, Jin Ma, Ben He, Yingfei Sun, Dezhi Ye
Computation and Language
Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generation
Jinyoung Kim, Muhammad Khalifa, Lajanugen Logeswaran et al. (+4 more)
Natural-
language critiques provide supervision beyond scalar rewards for non-verifiable generation, which lacks deterministic verifiers. In critique-guided refinement, a critic gives feedback on an initial response and an actor revises it. However, final revision quality does not reveal whether the critique was actually useful: a capable actor may improve without following the feedback, while valid feedback may fail if the actor cannot execute it. We frame critique as actor-conditioned revision guidance, where usefulness depends on whether the feedback helps the target actor address the intended weakness. We introduce TAIScore (Targeted Actionable Improvement Score), a reward that evaluates the instruction, initial response, critique, and revision together, assessing whether the critique targets a real weakness, whether the actor follows it, and whether the intended aspect improves. We use this reward to train an actor-tailored critic with GRPO, and use critique-guided refinements to construct DPO preference pairs for the actor, forming a co-evolving critic-actor loop where the critic adapts to the actor's changing capability. Experiments show that an 8B critic trained with TAIScore outperforms both a zero-shot 120B critic and critics trained with outcome-only or critique-only reward signals. Co-evolving the critic and actor further improves performance, suggesting that effective critique supervision should adapt as the actor changes.
Read the full abstract ▾
All authors Jinyoung Kim, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, Lu Wang
Computation and Language
When LLM Meets Tree Search: A Systematic View of Inference as Search in Large Language Models
Jiaqi Wei, Xiang Zhang, Yuejin Yang et al. (+10 more)
As pretraining scaling laws approach saturation, Test-Time Scaling (TTS) has emerged as an important direction for improving
reasoning by allocating inference-time compute to a fixed model prior. Viewed at a high level, TTS reframes inference as search over a space of partial
reasoning states. While Chain-of-Thought (CoT) exposes intermediate steps, common instantiations rely on single-trajectory decoding, limiting recovery from early errors and exploration. This survey systematizes recent progress in tree-search-based
reasoning, viewing inference as instance-specific optimization rather than decoding. We trace the
evolution from uninformed search to Monte Carlo Tree Search (MCTS), highlighting how sampling-based control supports principled exploration-exploitation trade-offs. To unify a fragmented literature, we introduce a Unified Design Space spanning search
topology, evaluation signals, and control dynamics, and advocate a standardized compute-reporting abstraction to make compute-accuracy trade-offs explicit and comparable.
Read the full abstract ▾
All authors Jiaqi Wei, Xiang Zhang, Yuejin Yang, Wenxuan Huang, Juntai Cao, Sheng Xu, Xiang Zhuang, Zhangyang Gao, Muhammad Abdul-Mageed, Laks VS Lakshmanan, Chenyu You, Wanli Ouyang, Siqi Sun
Computation and Language
Quantitative Evidence Mining for Plausibility-Aware Biomedical AI: A Narrative Review and Conceptual Framework
Negin Sadat Babaiha, Stefan Geissler, Marie-Christine Simon et al. (+2 more)
Biomedical artificial intelligence is moving from literature
retrieval toward evidence synthesis for knowledge graphs, clinical decision support, and computational models. Yet most information-extraction systems still represent findings as simple relations, discarding the quantitative and contextual detail needed for interpretation and reuse. A claim that one entity affects another is insufficient when the magnitude, unit, population, comparator, experimental conditions, uncertainty, and provenance are missing. We define quantitative evidence mining as a framework for transforming biomedical findings into structured, context-rich, and auditable evidence units. We define the core elements of an evidence unit: the claim; measured entity and property; value, unit, or scale; comparator; population; biological or clinical conditions; temporal context; uncertainty; provenance; validation results; and expert-review status. We propose an eight-stage reference architecture spanning corpus selection, entity recognition, quantity extraction, context linking, normalization, evidence-unit assembly, multidimensional plausibility assessment, and export and governance. A central principle is that plausibility should not be collapsed into a single truth label; statistical, biological, methodological, contextual, and provenance-based support should remain explicit. The framework links information extraction to evidence synthesis and computational reuse, with applications in clinical-trial analysis, biomarker research, pharmacovigilance, knowledge-graph construction, and mechanistic modelling. It is a research agenda rather than a validated end-to-end system. Progress will require annotated
multimodal benchmarks, rigorous component- and workflow-level evaluation, prospective testing, transparent provenance, and sustained expert oversight.
Read the full abstract ▾
All authors Negin Sadat Babaiha, Stefan Geissler, Marie-Christine Simon, Martin Hofmann-Apitius, Marc Jacobs
Computation and Language
Using Grounded Theory for Agent Behavior Analysis at Scale
Zhuoran Lu, Yangyang Yu, Zhuoyan Li et al. (+5 more)
Understanding
agent behavior requires methods that scale to thousands of trajectories and surface new patterns in long, often unfamiliar tasks where pre-built classifiers fall short. We propose to bring grounded theory into
agent trajectory analysis: a six-decade-old qualitative method from the social sciences, with a principled saturation criterion and an auditable trail from data to theory. We propose AutoTraceGT (Automated Trace analysis through Grounded Theory), the first multi-
agent pipeline that automates grounded theory on
agent trajectories. It iteratively performs open, axial, and theoretical coding until saturation, producing a behavioral taxonomy tailored to each task. Across six trajectory corpora, AutoTraceGT produces codebooks that recover 73-91 percent of the failure modes in human-annotated taxonomies and surface additional patterns that those taxonomies miss. The emergent theoretical narrative aligns with prior expert accounts. Used as a deductive feature space, the codebook outperforms zero-shot and few-shot LLM baselines on downstream failure prediction. These results suggest Grounded Theory offers a scalable analytic tool for ML researchers and
agent developers studying what
agents actually do.
Read the full abstract ▾
All authors Zhuoran Lu, Yangyang Yu, Zhuoyan Li, Yibo Meng, Nan Jiang, Chengxi Zang, Jie Gao, Ziang Xiao
Computation and Language
Kathleen Remembers: Length-Invariant One-Shot Recall Without Attention
George Fountzoulas
Recurrent,
attention-free sequence models share a structural weakness: a fading state cannot perform exact recall of something seen once, far in the past. We add to the Kathleen trunk a second
memory layer – a "notebook": a fixed-key
holographic (HRR) associative store with a learned local write gate, a self-gating raw read, and write-triggered forgetting – 25K parameters that attach to the logits of any trunk. (1) Mechanism: on a controlled needle-in-haystack task the notebook reaches 80-82% one-shot recall at 4x the training length, where the bare trunk scores ~4% and a parameter-matched
attention head scores 100% inside its training length and 0% beyond it. Addressing is length-invariant by construction; the untrained
memory alone recalls at 90% accuracy identically at 512, 2048 and 4096 bytes. Because the store is a linear superposition, two capabilities follow from arithmetic alone: selective unlearning (one subtraction erases one fact to chance, retained facts unharmed) and per-
token attribution (counterfactual erasure names the source fact of every correct byte, 100% provenance). (2) Real text: on WikiText-2 bytes the notebook improves prediction of repeated rare words by +0.15-0.27 bits/byte, the gain growing with the distance between mentions and holding zero-shot at 4x training length; write-triggered forgetting eliminates
memory pollution at 8x length (first-mention cost +0.33 -> -0.004). (3) Scope and scale: a parameter-matched
attention head does generalize on natural-text repetition, so the notebook's claim is exact recall at O(L); on a WikiText-103 ladder (8 to 512 MB) the zero-shot repeat gain rises monotonically. All experiments are pre-registered, seeds reported, and reproducible on a single free-tier GPU.
Read the full abstract ▾
Computation and Language
Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation
Minsoo Song, Chanwoo Kim, Sugyeong Eo et al. (+1 more)
Multi-
Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple
agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-
agent agreement does not guarantee
alignment with human judgments. In this paper, we compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks and design three ablations to isolate the impact of role prompting, multi-round interaction, and explicit score sharing. Evaluations across six LLMs show that the single-judge baseline achieves the strongest human
alignment on average across six judge models, whereas MAD shows degradation in human
alignment on both tasks. Our ablations demonstrate that this performance drop stems primarily from asymmetric role prompting rather than the interaction itself. Specifically, assigning a strict judge role introduces a systematic downward bias that the consensus process fails to correct. The central finding is that this bias reflects strict-stance dominance beyond averaging: the consensus score falls well beyond the arithmetic midpoint of the standalone strict and lenient conditions, rather than averaging them out. Removing role asymmetry (Symmetric MAD) largely recovers baseline performance, while masking peer scores widens inter-
agent disagreement on average and worsens average human
alignment. These findings demonstrate that multi-
agent consensus can enforce artificial agreement at the expense of true human
alignment, revealing a structural limitation in consensus-style, role-specialized MAD protocols for subjective scoring.
Read the full abstract ▾
All authors Minsoo Song, Chanwoo Kim, Sugyeong Eo, Chanjun Park
Computation and Language
Auditing MCQA Benchmarks through Probability Landscapes
Minsoo Song and Chanjun Park
As Large
Language Models rapidly advance, performance on standard multiple-choice question answering (MCQA)
benchmarks is reaching saturation. While the community has responded by developing increasingly difficult
datasets, validating question quality and filtering flawed items remains a labor-intensive process. To provide a scalable diagnostic approach, we propose a two-component probabilistic framework for
auditing MCQA
benchmarks using model output distributions. First, for
benchmark-level analysis, we characterize the probability landscape using the top prediction probability ($P_{top1}$) and normalized residual
entropy ($H_{norm}$), summarized globally by Mean Pairwise Distance (MPD). Second, for item-level diagnostics, we introduce noise injection to reduce meaningful distractor competition, enabling us to flag candidate items for targeted human review and categorize residual failure patterns. Across four MCQA
benchmarks, our landscape analysis reveals
benchmark-level differences in model confidence and residual option competition. Concurrently, our noise-injection method flags potentially actionable item-level issues, showing
alignment with expert error annotations from MMLU-Redux. These results suggest that our probability-based framework provides a lightweight audit lens for comparing macro-level
benchmark structure and prioritizing individual items for targeted human review.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Will the User Ever Know? Covert Indirect Prompt Injection Attacks on Tool-Using LLM Agents
Yunseok Lee, Yunji Kim and Woojin Lee
As LLM
agents take real-world actions through tools, indirect
prompt injection (IPI) has emerged as a serious threat. The standard metric, Attack Success Rate (ASR), counts whether an injection succeeds but ignores what the user notices in the
agent's final response. Looking at successful injection traces, we find two distinct outcomes: the
agent executes the injection while returning an otherwise normal response, or reports the injected action in its final response, giving the user a chance to notice. We call these covert and overt successes. From the user's perspective, we decompose ASR into the Covert Success Rate (CSR), counting successes leaving no trace in the final response, and the Overt Success Rate (OSR), counting successes the user can detect. To understand what drives the gap, we analyze successful trajectories and find that the
agent's behavior after the injection separates covert from overt: covert traces hand control back to the user task before ending, while overt traces end at the attack itself. This split follows from the ReAct format, where the final response summarizes the most recent action. Building on this observation, we propose ICoA (Induced Covert Attack), an IPI attack designed to induce covert outcomes by steering the
agent back to the user task after executing the injection. Across four target models on AgentDojo, ICoA achieves the highest CSR, with gains of 3.79-12.01 percentage points over the strongest baseline.
Read the full abstract ▾
Computation and Language
Do Small Models Use the Law You Give Them? Measuring Context Use on a Bilingual Bangladesh Legal Benchmark
Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa et al. (+3 more)
Fine-tuning can improve legal question-answering accuracy without improving how models use law supplied in context. We study this distinction in bilingual Bangladeshi legal QA, where observed errors can arise from answer scoring,
retrieval, or failure to use relevant law. We construct a hierarchy-preserving statutory corpus, 2,165 reviewed bilingual fine-tuning examples, and a 150-item supplied-law control. We evaluate six instruction-tuned models: Llama-3.2-1B, Llama-3.2-3B, Qwen3.5-0.8B, Qwen3.5-2B, Qwen3.5-4B, and Gemma-4-E2B, with three LoRA seeds per model. To separate effects, we combine constrained option-letter scoring, cyclic option rotation, and controlled removal of the governing provision. On 398 Bar Council outputs, an exact-line parser attributes an accuracy gain of 50.0% to the Qwen3.5-2B seed-42 adapter, whereas option scoring yields only $3.0\%$. For Gemma-4-E2B, the two scoring methods favor different systems. When the governing provision is guaranteed to be present, five of six reference models improve by $14.7\%-19.3\%$ under the four-order criterion. Removing that provision reduces accuracy by $8.0\%-15.3\%$ for models and by $13.8\%-14.9\%$ points for their adapters. However, difference-in differences estimates show no increase in reliance on the governing provision after fine-tuning. Results show that legal adaptation claims require separating scorer, retriever, and model effects. Our Code and data are available at https://anonymous.4open.science/r/bangladesh-legal-qa-11E3
Read the full abstract ▾
All authors Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa, Mir Mohammad Asif Abdullah, Swakkhar Shatabda, Md Adnan Arefeen
Computation and Language
Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS
Yan Zhou, Yun Hong and Yang Feng
Natural-
language instructions enable flexible control of synthesized
speech, yet emotional TTS systems primarily model a single utterance-level affect, leaving multi-
emotion control underexplored. We study two complementary multi-
emotion TTS tasks:
emotion trajectory, which spans several ordered affective stages, and
emotion blending, in which multiple
emotions coexist throughout an utterance. These tasks expose a supervision mismatch: supervised fine-tuning (SFT) does not explicitly evaluate
emotion features, while single-
emotion rewards provide neither structure-aware feedback for trajectory completion nor pair-aware feedback for blending. We introduce HybridEmo, a post-training framework that initializes both tasks with SFT and then aligns the
speech-
token policy through Group Relative Policy Optimization using a sample-aware hybrid reward. For trajectory samples, segment-aligned consistency combines average and weakest-stage evidence to preserve the correctness and completeness of prescribed stages. For blending samples, a GMM-based reward combines frame-level support from the union of target-
emotion anchors in an offline
emotion space with an utterance-level weaker-target margin. Both branches share an ASR reward and are routed within a unified policy. On MultiEmo-Test, HybridEmo significantly improves trajectory correctness and blending intensity, without a noticeable degradation in speaker similarity. Human evaluation prefers HybridEmo to CosyVoice 3 and EmoVoice-0.5B, with nearly balanced preferences against Qwen3-TTS.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents
Hanlin Tian, Minhao Li, Yu Mi et al. (+5 more)
Professional
agent tasks often depend on conventions that are absent from public corpora, yet
benchmarks rarely control whether an
agent has access to those conventions. We introduce a knowledge-gated task-construction protocol that separates a task instruction from a compact artefact containing private conventions, reference tables, and utility operators. Construction-time provenance, byte-identical task instructions across the provided- and withheld-artefact conditions, leak audits, and executable witnesses make dependence on the artefact explicit and testable. Across fifteen calibration tasks, one frontier
agent configuration achieves a 68.0% pass rate with the artefact and 0% without it; on one task, a plausible but incorrect artefact also yields 0% across five trials. Deterministic solvers and rule corpora provide exact ground truth for structured tasks, while named criterion-level rubrics support outputs that cannot be checked by a single executable oracle. A configuration-relative calibration screen retains seven tasks satisfying our five-trial empirical knowledge-gating screen. These experiments validate the behavior of the construction protocol; they do not establish that the retained tasks improve post-training. We publicly release part of the task suite and supporting tooling at https://github.com/DatagridsAI/Knowledge-Gated-Task-Construction.
Read the full abstract ▾
All authors Hanlin Tian, Minhao Li, Yu Mi, Sihan Zhu, Zhao Yang, Yuxiang Wang, Hongquan Zhu, Qiufei Hu
Computation and Language
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Zihan Qiu, Zekun Wang, Xiao Li et al. (+33 more)
We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per
token, and additional 51B parameters of n-gram
embedding tables held off the accelerator. On fourteen pre-training
benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training
tokens, and roughly 1/9 the training FLOPs.
Token mixing uses a layer-wise hybrid of Gated DeltaNet (GDN) and global
attention, with one full-
attention layer in every four; at continued-pretraining time those full-
attention layers are replaced by Qwen Sparse
Attention (QSA), which scores context at micro-block granularity with a compressed lightweight indexer. The residual stream is widened to four branches and read through an elementwise gate, a design we call the Gated Residual (GR). Capacity is added outside the backbone by a single n-gram
embedding layer whose tables are prefetched from host
memory. We evaluate every candidate change along three axes: loss together with downstream
benchmarks; the cost of the change in training, prefill and decode; and its effect on the optimal hyperparameters and training stability. Loss and downstream accuracy do not always move together: enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates. The architecture and the
Muon optimizer together shift the optimal learning rate and batch size upwards, render batch-size warmup unnecessary, and substantially improve stability under stress tests. Loss,
benchmarks, efficiency and stability form one design problem. Solved jointly, they yield a recipe that is simultaneously more efficient, more capable and more stable.
Read the full abstract ▾
All authors Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, Dayiheng Liu
Computation and Language
Beyond Token-Level Guidance: Inference-Time Alignment of Specialized LLMs via Cross-Family Representation Steering
Jin Gan, Xin Li and Jun Luo
Large
language models (LLMs) finetuned for specialized domains represent crucial high-impact applications. Inference-time
alignment improves safety degraded from specialization finetuning without requiring substantial computational resources, complementing finetuning-based methods with an easy-to-use, plug-and-play solution. However, existing inference-time methods fail to reliably improve safety without disrupting domain capability. We identify the root cause as complementary expertise orthogonality: specialized base models and general-domain guidance models have orthogonal competencies, making the guidance signal unreliable for specialized generation. This primarily manifests as stop
token interference, where the guidance model's tendency toward continuation overrides the base model's decision to stop, burying correct answers under guidance-induced continuation. To address this problem, we propose CREST, an inference-time
alignment method that steers base model hidden representations using safety directions extracted from a guidance model of any family, avoiding
token-level structural limitations entirely. CREST improves safety where specialization has weakened it while preserving both domain-specific capability and the safety of already well-aligned models, outperforming baselines by up to 22.2% on safety
benchmarks. Our code is available at: https://github.com/DecayingSeart/CREST.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization
Junjie Yao, Liangkai Hang and Zhi-Qin John Xu
Token embeddings are the basic representational units that connect discrete
tokens with continuous computation in
language models. Although modern
language models learn
embeddings from random initialization through gradient-based training, the dynamical mechanism by which meaningful
embedding structures emerge remains unclear. In this work, we identify that the evolving
embedding structures are closely related to
token-conditioned label and contextual distributions, which we formalize as probability signatures. We observe a progressive learning process, which we term Context Staircase:
embeddings learn the low-order statistic signatures of the data before the high-order ones. More specifically, we observe that early in training they align with the simplest, context-free signature linking a
token to its label, and as training proceeds, they progressively reflect signatures involving more and more context
tokens. We then analyze the gradient flow of
embeddings under small initialization to explain this phenomenon, deriving
embedding evolution equations for feed-forward and self-
attention architectures. We further extend these observations to real
language-model training. Finally, we show that these
embedding structures play an important role in both task learning and the incorporation of semantic structure into the
embedding space. Overall, our results provide a dynamic explanation of how data statistics and architecture jointly shape
token embeddings in
language models, and reveal an implicit bias in the space of data statistics: training proceeds from simpler, low-order statistical relations toward increasingly complex, context-dependent ones.
Read the full abstract ▾
Computation and Language
Lazy Grounding: Attacking Search Agents with Factual Evidence
Yulin Zhang, Yukun Huang, Sanxing Chen et al. (+4 more)
Search
agents mitigate
hallucination by grounding their answers in retrieved web results. However,
retrieval-based approaches also introduce an attack surface:
agents may cite misinformation from poisoned search corpora containing false or malicious documents. We demonstrate that, in some cases, search
agents'
reasoning and responses may be steered by completely factual but distracting information. We refer to this failure as lazy grounding. We expose lazy grounding by injecting nearby evidence from answer-changing rewrites of
benchmark questions into the search corpora. Each document contains factual evidence that supports a neighboring rewritten question but is retrieved for the original question. Across 12 model-
benchmark pairs, the attack causes the accuracy of search
agents' responses to drop by 5.9 points on average and by up to 17.3 points, while inducing nearby-answer adoption in every setting. The effect is even stronger when nearby evidence appears later or is more answer-shaped. Our results show that robust search
agents must defend against not only misinformation but also the misapplication of factual evidence. The code is publicly available at https://github.com/frankyzha/lazy-grounding.
Read the full abstract ▾
All authors Yulin Zhang, Yukun Huang, Sanxing Chen, Tianyi Lin, Ziang Yang, Xunjian Yin, Bhuwan Dhingra
Computation and Language
AIA$^{2}$: Attribute-Agnostic Imbalance Augmentation for Subgroup Robustness
Hanshu Rao, Guangzeng Han and Xiaolei Huang
Attributes describing data content and context can induce diverse imbalance patterns that go beyond label imbalance alone. However, existing studies primarily address label imbalance while overlooking data attributes, such as topics and demographics, which can induce meaningful subgroup structure while causing model degradation on underrepresented subgroups. We propose Attribute-Agnostic Imbalance Augmentation (AIA$^{2}$), a framework for improving model robustness under varying subgroup imbalances without explicit subgroup annotations. AIA$^{2}$ automatically discovers varying imbalances via latent semantic distributions, obtains slices with both learning difficulty and subgroup imbalance deficits, and deploys a large
language model (LLM) for subgroup-aware imbalance augmentation. We have evaluated AIA$^{2}$ on 5 popular corpora with rich domains and their attribute values, covering social issues and diverse topics. Results show improved performance on the lowest-performing subgroups and consistent gains over competitive baselines. Ablation studies confirm complementary contributions from each component, and additional analyses show that AIA$^{2}$ provides a practical and consistent way to improve worst-group robustness under data subgroup imbalance. Code is available at https://github.com/trust-nlp/AIA2-Subgroup-Robustness.
Read the full abstract ▾
Computation and Language
Read the Room, Read the Image: Understanding Indirect Speech Acts in Multimodal Visual Contexts
Jaehee Kim, Ji Hoon Chung, Seoyoon Park et al. (+5 more)
Indirect
speech acts (ISAs) require pragmatic
reasoning over context, as directive intent can- not be inferred from surface form alone. Prior text-based studies and existing
multimodal benchmarks largely overlook this requirement, focusing instead on explicitly encoded context or perceptual recognition, and thus underex- plore context-dependent pragmatic understand- ing, particularly in high-context
languages such as Korean. We introduce READI, a
multimodal benchmark for evaluating ISA understanding through integrated
reasoning over visual con- text and dialogue. READI models graded in- directness grounded in pragmatic theory and formulates the task as
vision-based pragmatic question answering (V-PQA), supporting cross- lingual evaluation in English and Korean. Ex- periments show that even state-of-the-art multi- modal models struggle with visually grounded indirect
speech acts, with performance declin- ing as indirectness increases, underscoring the need for
benchmarks that explicitly target con- textual pragmatic
reasoning.
Read the full abstract ▾
All authors Jaehee Kim, Ji Hoon Chung, Seoyoon Park, Unsol Kim, Kyungwon Park, Ji Hak Kim, Yi-Jun Chen, Hansaem Kim
Computation and Language
Using Prosody to Predict Syntactic Structure
Junghyun Min, Alex Warstadt, Tamar I. Regev et al. (+2 more)
While it is well-established that prosody carries crucial cues for syntactic structure, the degree and nature of correspondence between these two domains remains contested. We investigate the syntax-prosody interface through an information-theoretic lens, quantifying the interaction between prosodic features and syntactic representations as their mutual information. We provide a general-purpose framework for estimating this quantity over large
speech-text corpora using
multimodal language models. Our framework is structure-agnostic and modular, insofar as it can be used to measure the contributions of individual prosodic features or components of structure. We evaluate the syntax-prosody relationship for two features (word duration and inter-word pauses) across two domains–read audiobooks and spontaneous conversations–both in English. Our results demonstrate that prosody contains measurable syntactic information, with prosodic features reducing syntactic uncertainty in spontaneous conversations by up to 10.2%. Our findings offer new empirical support for several theoretical accounts of the syntax-prosody interface.
Read the full abstract ▾
All authors Junghyun Min, Alex Warstadt, Tamar I. Regev, Tiago Pimentel, Ethan Gotlieb Wilcox
Computation and Language
Stratified Consistency Distillation for Natural Language Formalization
Zhichao Hou, Ferhat Erata, Joe Lilien et al. (+1 more)
Neurosymbolic
reasoning has shown promising success in addressing complex
reasoning tasks by combining large
language models (LLMs) and symbolic solvers. While this approach shows promise, a fundamental challenge remains: improving the accuracy of translations from natural
language to logical formulas. Current methods predominantly rely on
prompt engineering, which is difficult to scale across different domains and input formats. Drawing inspiration from the success of fine-tuning in other model adaptation and
alignment applications, we propose a fine-tuning-based Stratified Consistency
Distillation approach: (1) We generate K logical translations per input using a frontier LLM and cluster them by semantic equivalence (2) Based on the
entropy level, we apply majority voting (low
entropy), LLM-as-a-Judge (medium
entropy), or unification/abstention (high
entropy), and (3) fine-tune a smaller model using the selected pseudo-labels. Our experiments show significant and consistent improvements in both Pass@K and our novel Equivalent Logical Similarity metrics, demonstrating the potential of advancing logical translation through consistency
distillation.
Read the full abstract ▾
All authors Zhichao Hou, Ferhat Erata, Joe Lilien, MohamadAli Torkamani
Computation and Language
Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs
Ramya Keerthy Thatikonda, Wray Buntine and Ehsan Shareghi
Logical
reasoning with large
language models (LLMs) is a critical capability, as it reflects a system's ability to correctly deduce hypotheses from a given context using faithful deductive processes. However, LLM
reasoning has often been shown to be sensitive to small surface-level variations in problem formulation, raising questions about whether models truly follow the underlying logical structure. Studying this behavior is challenging because the symbolic components of logical problems, such as operators and predicates, are difficult to systematically manipulate in natural
language. We introduce a tool-driven framework for generating controlled, label-preserving edits to logical
reasoning problems. Our method operates on symbolic representations of first-order logic and constraint satisfaction problem tasks, enabling targeted modifications to logical operators and other structural components before translating them back into natural
language. Using this framework, we evaluate various LLMs under cumulative and individual operator edits and analyze their behavior in response to these changes. Our quantitative and qualitative analyses show that LLM
reasoning behavior under controlled operator edits is inconsistent, regardless of model size or family: models sometimes adapt correctly to structural changes but often fail to track their logical consequences. The results from this automated stress test enable an evaluation of
language models across different dimensions and help measure the reliability of their
reasoning.
Read the full abstract ▾
Computation and Language
PaperBanana-Interact: Scientific Diagram Refinement with Multi-Turn Human Feedback
Xueqing Wu, Ashwin Balasubramanian, Bingxuan Li et al. (+7 more)
Recent efforts have aimed to automate scientific diagram generation from paper content (Lin et al., 2026; Zhu et al., 2026a). However, fully satisfying an author's visual and communicative preferences in a single turn is challenging: in our formative user study (N = 14), all participants requested further revisions after viewing an initial draft, and 86% of them rated the refined diagrams as more satisfactory. Despite the clear demand, the multi-turn workflow remains largely underexplored. To bridge this gap, we present MTPaperBananaBench, a
benchmark for multi-turn diagram generation containing 292 images annotated with 3,518 user requirements. To reduce expensive human studies and enable scalable benchmarking, we construct a user simulator that, at each turn, identifies unsatisfied requirements and converts k of them into natural
language feedback. Evaluating both requirement satisfaction and overall diagram quality reveals two key failure modes shared across baseline multiturn systems: (1) quality drift, where diagram quality progressively declines over turns, and (2) forgetting, where previously implemented features are lost in subsequent turns. To address these issues, we introduce PaperBanana-Interact, a multi-
agent system that refines diagrams via an internal critique-and-refine loop. PaperBanana-Interact consistently improves rather than degrades diagram quality across turns, outperforming baselines by 11.9-18.6 points in quality score and reducing forgetting by 3.7-6.2 points.
Read the full abstract ▾
All authors Xueqing Wu, Ashwin Balasubramanian, Bingxuan Li, Dawei Zhu, Kai-Wei Chang, Yale Song, Yiwen Song, Rui Meng, Tomas Pfister, Nanyun Peng
Computation and Language
Quantifying and Mitigating Korean Jamo-Level Typographical Vulnerabilities in Large Language Models
Seojin Lee and Hwanhee Lee
Korean introduces an additional typographical perturbation level not captured by ordinary character-level edit models: because syllable blocks are internally composed of sub-character units called jamo, keyboard-level errors can occur within a syllable, either producing a valid but semantically altered character or exposing raw jamo on the surface. Both outcomes disrupt sub-word tokenization and are not reliably corrected by existing grammatical error correction pipelines, leaving LLMs directly exposed to corrupted inputs. To quantify this vulnerability, we apply five jamo-level perturbation types to the KMMLU
benchmark and evaluate four
language models, finding that accuracy declines monotonically with perturbation intensity and that parameter scaling does not confer robustness against intra-syllabic noise. We further show that typo-corrupted inputs induce a distinct shift in internal representations that is not reducible to ordinary answer incorrectness, and that a simple linear probe trained on these representations detects unseen perturbation types with high AUROC. Motivated by this signal, we propose Typo-Aware Chain-of-Thought (TACoT), which routes inputs to chain-of-thought inference only when the probe detects a likely typo, recovering a substantial portion of the CoT accuracy gain at a fraction of the inference cost.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
LaMoC: Loss-Aware Modular Compression for LLMs
Mohanad Odema and Jacob Song
Modular compression has enabled considerable parameter reduction in LLMs while preserving strong
language understanding and downstream task accuracy. However, existing joint modular compression methods primarily rely on activation statistics, leaving loss-sensitivity information and its module-level characterization underexplored. We investigate addressing this gap with LaMoC, a loss-aware modular compression methodology that blends activation and Empirical Fisher statistics through gradient-error
alignment. LaMoC improves joint compression by selecting compression statistics that better align local module reconstruction error with the downstream loss. Our contributions are three-fold: (1) We characterize the Empirical Fisher as a module-level loss-aware proxy that can be blended with the activation statistics required for compression. (2) We reformulate joint modular compression as a two-tiered optimization problem that minimizes module reconstruction error while tuning the activation and gradient information blending rate. (3) We implement an empirically driven methodology with statistical validation to solve the resulting compression problem. We evaluate LaMoC across four model families spanning eight models. On the 4-8B models, LaMoC achieves an average 2.5% reduction in perplexity and a 1% relative improvement in task accuracy over state-of-the-art modular compression methods.
Read the full abstract ▾
Computation and Language
The Differential Reasoning Router: Operationalizing Cost-Aware LLM Annotation in E-commerce
Cheng Lyu, Jingyue Zhang, Vinny DeGenova et al. (+2 more)
Large
Language Models (LLMs) are increasingly used to annotate structured product data in e-commerce, but early deployment often begins as a cold-start problem: only limited pre-launch labels are available, the value of expensive
reasoning is unknown, and human review is needed before the system can be trusted at scale. This challenge is especially common in rule-based annotation workflows, where each item must satisfy multiple business rules and both model errors and ambiguous rule boundaries affect final decisions. We introduce the Differential
Reasoning Router (DRR), a cost-aware framework for cold-start LLM annotation that jointly optimizes model selection and human escalation. Rather than treating a
reasoning model as a default fallback, DRR estimates separate success probabilities for a direct model and a
reasoning model at both the sample and business-rule levels, enabling adaptive routing: easy cases are handled directly,
reasoning is reserved for cases where it is expected to improve the decision, and likely double-failure or rule-disagreement cases are escalated to human annotators. The resulting labels provide targeted ground truth for
prompt engineering, supervised fine-tuning, calibration, and rule refinement, enabling a gradual shift from human-heavy cold-start annotation toward high-confidence automated routing. In a production e-commerce workflow, DRR reaches accuracy parity with the strongest confidence-based router while achieving more than 60%
reasoning-
token cost savings.
Read the full abstract ▾
All authors Cheng Lyu, Jingyue Zhang, Vinny DeGenova, Mengwei Li, Yuanli Pei
Computation and Language
Label Semantic Expansion via Label Guided Neural Topic Modeling
Haojia Zheng, Yuyin Lu, Juntian Huang et al. (+4 more)
Topic models are widely used for content analysis, where users often analyze corpora around predefined labels rather than unordered latent topics. Existing label-aware topic models mainly follow a labels-for-topics perspective, using labels to guide topic learning, while the learned topics are not directly usable for label-centered analysis. We explore the reverse topics-for-labels perspective and instantiate it as Label Semantic Expansion (LSE), which enriches sparse label representations with corpus-grounded descriptive topic words. To exploit topics in LSE effectively, we propose a Label-Guided Neural Topic Model (LGNTM), which learns dedicated label-aligned topics, grounds them in lexical and document semantic spaces, and preserves consistency between topic structures and label structures. Experiments on label-topic
alignment, label expansion, topic quality, and downstream classification demonstrate strong overall performance across complementary evaluation dimensions.
Read the full abstract ▾
All authors Haojia Zheng, Yuyin Lu, Juntian Huang, Fan Ou, Yanghui Rao, Haoran Xie, Fu Lee Wang
Computation and Language · Cross-listed from cs.CV
Towards a Joint Khmer Text Recognition and Word Segmentation
Marry Kong, Rina Buoy, Sovisal Chenda et al. (+3 more)
Text recognition, or extracting electronic text from document images, has been indispensable for knowledge
retrieval tasks, such as
retrieval-augmented generation (RAG). For Khmer, extracted text is subject to an extra word
segmentation step, as Khmer does not use any visible word delimiters to denote word boundaries. Thus, a recognition-then-
segmentation pipeline for Khmer requires two separate sequential models; this is not only error-prone but also adds significant latency for large-scale document processing. This paper proposes a novel joint Khmer text recognition and word
segmentation framework in a unified model. The proposed model, using a connectionist-temporal-classification (CTC) decoder for fast, parallel decoding, can be instructed to recognize Khmer text with ($b=1$) and without ($b=0$) word
segmentation. Experimental results on different
benchmark datasets of different document modalities (document, scene, and handwritten images) show that the proposed model can not only recognize characters in document images but also locate word boundaries, removing the need for an extra word
segmentation step in a conventional sequential pipeline.
Read the full abstract ▾
All authors Marry Kong, Rina Buoy, Sovisal Chenda, Nguonly Taing, Masakazu Iwamura, Koichi Kise
Computation and Language
When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detection
Yongjian Chen, Pengfei Wei, Yiqun Sun et al. (+2 more)
Multimodal Large
Language Models (MLLMs) process
speech and text jointly, yet whether they exploit prosodic cues for pragmatic inference or rely on surface acoustic patterns has received little systematic investigation. We address this through sarcasm detection, evaluating Qwen2.5-Omni and Qwen3-Omni on Mandarin Chinese and English under five modality conditions that decompose the contributions of lexical content, vocal semantics, and prosodic structure. Adding audio systematically inflates false positives without improving true positive detection. Acoustic error diagnosis reveals that model errors cluster on a shared stereotype of expressive prosody, namely elevated pitch and irregular pausing, that diverges from the actual cues marking sarcasm in both
languages. Targeted manipulation of only these two dimensions causally confirms the heuristic, inducing false positive rates of up to 60%. Applying the same manipulation template to Gemini~3 Flash Preview without modification replicates the effect, suggesting that the stereotype extends beyond the Qwen Omni family rather than arising from a single model architecture.
Read the full abstract ▾
All authors Yongjian Chen, Pengfei Wei, Yiqun Sun, Zhu Li, Lawrence B. Hsieh