Computation and Language
FreeAct: Freeing Activations for LLM Quantization
Xiaohao Liu, Xiaobo Xia, Manyi Zhang et al. (+6 more)
Quantization is pivotal for mitigating the significant
memory and computational overhead of Large
Language Models (LLMs). While emerging transformation-based methods have successfully enhanced quantization by projecting feature spaces onto smoother
manifolds using orthogonal
matrices, they typically enforce a rigid one-to-one transformation constraint. This static approach fails to account for the dynamic patterns inherent in input activations, particularly within
diffusion LLMs (dLLMs) and
Multimodal LLMs (MLLMs), where varying
token types exhibit distinct distributions. To advance this, we propose FreeAct, a novel quantization framework that relaxes the static one-to-one constraint to accommodate dynamic activation disparities. Theoretically, we leverage the rank-deficient nature of activations to derive a solution space that extends beyond simple inverse
matrices, enabling the decoupling of activation transformations from weights. Methodologically, FreeAct identifies
token-specific dynamics (i.e.,
vision v.s. text, or masked
tokens) and allocates distinct transformation
matrices to the activation side, while maintaining a unified, static transformation for the weights. Extensive experiments across dLLMs and MLLMs demonstrate that FreeAct significantly outperforms baselines, up to 5.3% performance improvement, with in-depth analyses. Our code will be publicly released.
Read the full abstract ▾
All authors Xiaohao Liu, Xiaobo Xia, Manyi Zhang, Ji-Fu Li, Xianzhi Yu, Fei Shen, Xiu Su, See-Kiong Ng, Tat-Seng Chua
Computation and Language
Beyond the Resumé: A Rubric-Aware Automatic Interview System for Information Elicitation
Harry Stuart, Masahiro Kaneko and Timothy Baldwin
Effective hiring is integral to the success of an organisation, but it is very challenging to find the most suitable candidates because expert evaluation (e.g.\ interviews conducted by a technical manager) are expensive to deploy at scale. Therefore, automated resume scoring and other applicant-screening methods are increasingly used to coarsely filter candidates, making decisions on limited information. We propose that large
language models (LLMs) can play the role of subject matter experts to cost-effectively elicit information from each candidate that is nuanced and role-specific, thereby improving the quality of early-stage hiring decisions. We present a system that leverages an LLM interviewer to update belief over an applicant's rubric-oriented latent traits in a calibrated way. We evaluate our system on simulated interviews and show that belief converges towards the simulated applicants' artificially-constructed latent ability levels. We release code, a modest
dataset of public-domain/anonymised resumes, belief calibration tests, and simulated interviews, at \href{https://github.com/mbzuai-nlp/beyond-the-resume}{https://github.com/mbzuai-nlp/beyond-the-resume}. Our demo is available at \href{https://btr.hstu.net}{https://btr.hstu.net}.
Read the full abstract ▾
Computation and Language
AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestions
Nils Constantin Hellwig, Jakob Fehle, Udo Kruschwitz et al. (+1 more)
We introduce AnnoABSA, the first web-based annotation tool to support the full spectrum of Aspect-Based Sentiment Analysis (ABSA) tasks. The tool is highly customizable, enabling flexible configuration of sentiment elements and task-specific requirements. Alongside manual annotation, AnnoABSA provides optional Large
Language Model (LLM)-based
retrieval-augmented generation (RAG) suggestions that offer context-aware assistance in a human-in-the-loop approach, keeping the human annotator in control. To improve prediction quality over time, the system retrieves the ten most similar examples that are already annotated and adds them as few-shot examples in the
prompt, ensuring that suggestions become increasingly accurate as the annotation process progresses. Released as open-source software under the MIT License, AnnoABSA is freely accessible and easily extendable for research and practical applications.
Read the full abstract ▾
All authors Nils Constantin Hellwig, Jakob Fehle, Udo Kruschwitz, Christian Wolff
Computation and Language
Merve Basoz, Andrew Horne and Mattia Opper
Embedding models are crucial to modern NLP. However, the creation of the most effective models relies on carefully constructed supervised finetuning data. For high resource
languages, such as English, such
datasets are readily available. However, for hundreds of other
languages, they are simply non-existent. We investigate whether the advent of large
language models can help to bridge this gap. We test three different strategies for generating synthetic triplet data used to optimise
embedding models. These include in-context learning as well as two novel approaches, leveraging adapter composition and cross lingual finetuning of the LLM generator (XL-LoRA) respectively. We find that while in-context learning still falls short of strong non-synthetic baselines, adapter composition and XL-LoRA yield strong performance gains across a wide array of tasks and
languages, offering a clear, scalable pathway to producing performant
embedding models for a wide variety of
languages.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training
Jinluan Yang, Yuxin Liu, Zhengyu Chen et al. (+7 more)
Training tool-use
agents typically relies on outcome-based filtering: Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks. However, this paradigm ignores interaction dynamics: successful trajectories may lack error recovery or exhibit redundancy, while pass rates fail to distinguish structurally informative tasks from trivial ones. We propose TopoCurate, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient
topology. By merging equivalent action-observation states, this projection transforms scattered linear trajectories into a structured
manifold that explicitly captures how tool invocations and environmental responses drive the divergence between effective strategies and failure modes. Leveraging this representation, we introduce a dual-selection mechanism: for SFT, we prioritize trajectories demonstrating reflective recovery, semantic efficiency, and strategic diversity to mitigate
covariate shift and mode collapse; for RL, we select tasks with high error branch ratios and strategic heterogeneity, maximizing gradient Signal-to-Noise Ratio to address vanishing signals in sparse-reward settings. Evaluations on BFCLv3 and Tau2 Bench show that TopoCurate achieves consistent gains of 4.2% (SFT) and 6.9% (RL) over state-of-the-art baselines. We will release the code and data soon for further investigations.
Read the full abstract ▾
All authors Jinluan Yang, Yuxin Liu, Zhengyu Chen, Chengcheng Han, Yueqing Sun, Qi Gu, Hui Su, Xunliang Cai, Fei Wu, Kun Kuang
Computation and Language
Legal RAG Bench: an end-to-end benchmark for legal RAG
Abdur-Rahman Butler and Umar Butler
We introduce Legal RAG Bench, a
benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a
benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of
retrieval and
reasoning models in RAG. We evaluate three state-of-the-art
embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini
Embedding 001, and OpenAI's Text
Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information
retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and
retrieval accuracy by 34 points. We observe that many errors attributed to
hallucinations in legal RAG systems are in fact triggered by
retrieval failures, concluding that
retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.
Read the full abstract ▾
Computation and Language
Building a Strong Instruction Language Model for a Less-Resourced Language
Domen Vreš, Tjaša Arčon, Timotej Petrič et al. (+3 more)
Large
language models (LLMs) have become an essential tool for natural
language processing and artificial intelligence in general. Current open-source models are primarily trained on English texts, resulting in poorer performance on less-resourced
languages and cultures. We present a set of methodological approaches necessary for the successful adaptation of an LLM to a less-resourced
language, and demonstrate them using the Slovene
language. We present GaMS3-12B, a generative model for Slovene with 12 billion parameters, and demonstrate that it is the best-performing open-source model for Slovene within its parameter range. We adapted the model to the Slovene
language using three-stage continual pre-training of the Gemma 3 model, followed by two-stage supervised fine-tuning (SFT). We trained the model on a combination of 140B Slovene, English, Bosnian, Serbian, and Croatian pretraining
tokens, and over 200 thousand English and Slovene SFT examples. We evaluate GaMS3-12B on the Slovenian-LLM-Eval
datasets, English-to-Slovene translation, and the Slovene LLM arena. We show that the described model outperforms 12B Gemma 3 across all three scenarios and performs comparably to much larger commercial GPT-4o in the Slovene LLM arena, achieving a win rate of over 60 %.
Read the full abstract ▾
All authors Domen Vreš, Tjaša Arčon, Timotej Petrič, Dario Vajda, Marko Robnik-Šikonja, Iztok Lebar Bajec
Computation and Language
Asking the Right Questions: Ontology-Grounded Interpretable Embeddings for Biomedical Text
Yixuan Tang, Zhenghong Lin, Yandong Sun et al. (+3 more)
While dense biomedical
embeddings achieve strong performance, their opaque dimensions limit transparency in biomedical NLP. Recent question-based interpretable
embeddings represent text through binary answers to natural-
language questions, but existing approaches rely primarily on corpus-driven signals, often capturing topical or stylistic differences rather than fine-grained biomedical distinctions. We propose QIME, an ontology-grounded framework for interpretable biomedical text
embeddings in which each dimension corresponds to an explicit biomedical-domain yes/no question. QIME leverages a biomedical ontology to guide contrastive question generation from semantic clusters, producing atomic, domain-grounded questions. It constructs
embeddings via similarity-based semantic activation with MMR-based diversity-aware dimension selection, yielding sparse representations efficiently. Experiments on biomedical clustering, STS and
retrieval benchmarks show that QIME consistently outperforms prior interpretable
embedding methods and substantially narrows the gap to strong black-box biomedical encoders. It also imposes substantially lower cognitive burden than existing methods, providing concise and domain-specific interpretations. The code is available at https://github.com/L1nzh/QIME.
Read the full abstract ▾
All authors Yixuan Tang, Zhenghong Lin, Yandong Sun, Wynne Hsu, Mong Li Lee, Anthony K.H. Tung
Computation and Language
Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention
Wenye Lin and Kai Han
Injecting new
reasoning knowledge into Large
Language Models (LLMs) via post-training often induces catastrophic forgetting. Recent studies emphasize the importance of on-policy data but suggest that KL-divergence fails to mitigate forgetting. In contrast, we show, both analytically and empirically, that the KL-constrained reward formulation actually plays a critical role in retaining knowledge during post-training. This motivates our
Surgical Post-Training (SPOT), a proximal on-policy
distillation framework designed to optimize
reasoning efficiently while preserving prior knowledge. SPOT consists of (1) a data rectification pipeline employing an Oracle to surgically correct erroneous steps via minimal edits, generating proximal on-policy data; and (2) a reward-based binary cross-
entropy objective essential for enhancing
reasoning and mitigating forgetting. Empirically, with only 4k rectified math pairs, SPOT improves Qwen3-8B's accuracy by 6.2% on average across in-domain and out-of-domain tasks, requiring merely 16-minute model training on 8x H800 GPUs. Moreover, SPOT provides a superior initialization for subsequent reinforcement learning, significantly elevating the performance ceiling. Code: https://github.com/Visual-AI/SPoT
Read the full abstract ▾
Computation and Language
Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representations
Yibo Yan, Mingdong Ou, Yi Cao et al. (+6 more)
Harnessing the full potential of visually-rich documents requires
retrieval systems that understand not just text, but intricate layouts, a core challenge in Visual Document
Retrieval (VDR). The prevailing multi-vector architectures, while powerful, face a crucial storage bottleneck that current optimization strategies, such as
embedding merging,
pruning, or using abstract
tokens, fail to resolve without compromising performance or ignoring vital layout cues. To address this, we introduce ColParse, a novel paradigm that leverages a document parsing model to generate a small set of layout-informed sub-image
embeddings, which are then fused with a global page-level vector to create a compact and structurally-aware multi-vector representation. Extensive experiments demonstrate that our method reduces storage requirements by over 95% while simultaneously yielding significant performance gains across numerous
benchmarks and base models. ColParse thus bridges the critical gap between the fine-grained accuracy of multi-vector
retrieval and the practical demands of large-scale deployment, offering a new path towards efficient and interpretable
multimodal information systems.
Read the full abstract ▾
All authors Yibo Yan, Mingdong Ou, Yi Cao, Xin Zou, Shuliang Liu, Jiahao Huo, Yu Huang, James Kwok, Xuming Hu
Computation and Language
LexChronos: An Agentic Framework for Structured Event Timeline Extraction in Indian Jurisprudence
Anka Chandrahas Tummepalli and Preethu Rose Anish
Understanding and predicting judicial outcomes demands nuanced analysis of legal documents. Traditional approaches treat judgments and proceedings as unstructured text, limiting the effectiveness of large
language models (LLMs) in tasks such as summarization, argument generation, and judgment prediction. We propose LexChronos, an agentic framework that iteratively extracts structured event timelines from Supreme Court of India judgments. LexChronos employs a dual-
agent architecture: a LoRA-instruct-tuned extraction
agent identifies candidate events, while a pre-trained feedback
agent scores and refines them through a confidence-driven loop. To address the scarcity of Indian legal event
datasets, we construct a synthetic corpus of 2000 samples using reverse-engineering techniques with DeepSeek-R1 and GPT-4, generating gold-standard event annotations. Our pipeline achieves a BERT-based F1 score of 0.8751 against this synthetic ground truth. In downstream evaluations on legal text summarization, GPT-4 preferred structured timelines over unstructured baselines in 75% of cases, demonstrating improved comprehension and
reasoning in Indian jurisprudence. This work lays a foundation for future legal AI applications in the Indian context, such as precedent mapping, argument synthesis, and predictive judgment modelling, by harnessing structured representations of legal events.
Read the full abstract ▾
Computation and Language
Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning
Jiebin Zhang, Zhenghan Yu, Liang Wang et al. (+8 more)
Speculative decoding accelerates large
language model (LLM) inference by using a small draft model to generate candidate
tokens for a larger target model to verify. The efficacy of this technique hinges on the trade-off between the time spent on drafting candidates and verifying them. However, current state-of-the-art methods rely on a static time allocation, while recent dynamic approaches optimize for proxy metrics like acceptance length, often neglecting the true time cost and treating the drafting and verification phases in isolation. To address these limitations, we introduce Learning to Draft (LTD), a novel method that directly optimizes for throughput of each draft-and-verify cycle. We formulate the problem as a reinforcement learning environment and train two co-adaptive policies to dynamically coordinate the draft and verification phases. This encourages the policies to adapt to each other and explicitly maximize decoding efficiency. We conducted extensive evaluations on five diverse LLMs and four distinct tasks. Our results show that LTD achieves speedup ratios ranging from 2.24x to 4.32x, outperforming the state-of-the-art method Eagle3 up to 36.4%.
Read the full abstract ▾
All authors Jiebin Zhang, Zhenghan Yu, Liang Wang, Nan Yang, Eugene J. Yu, Zheng Li, Yifan Song, Dawei Zhu, Xingxing Zhang, Furu Wei, Sujian Li
Computation and Language
Measuring What VLMs Don't Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation
Aditya Parikh, Aasa Feragen, Sneha Das et al. (+1 more)
Reliable deployment of
Vision-
Language Models (VLMs) in radiology requires validation metrics that go beyond surface-level text similarity to ensure clinical fidelity and demographic
fairness. This paper investigates a critical blind spot in current model evaluation: the use of decoding strategies that lead to high aggregate
token-overlap scores despite succumbing to template collapse, in which models generate only repetitive, safe generic text and omit clinical terminology. Unaddressed, this blind spot can lead to metric gaming, where models that perform well on
benchmarks prove clinically uninformative. Instead, we advocate for lexical diversity measures to check model generations for clinical specificity. We introduce Clinical Association Displacement (CAD), a vocabulary-level framework that quantifies shifts in demographic-based word associations in generated reports. Weighted Association Erasure (WAE) aggregates these shifts to measure the clinical signal loss across demographic groups. We show that deterministic decoding produces high levels of semantic erasure, while
stochastic sampling generates diverse outputs but risks introducing new bias, motivating a fundamental rethink of how "optimal" reporting is defined.
Read the full abstract ▾
All authors Aditya Parikh, Aasa Feragen, Sneha Das, Stella Frank
Computation and Language
More Data, Fewer Diacritics: Scaling Arabic TTS
Ahmed Musleh, Yifan Zhang and Kareem Darwish
Arabic Text-to-
Speech (TTS) research has been hindered by the availability of both publicly available training data and accurate Arabic diacritization models. In this paper, we address the limitation by exploring Arabic TTS training on large automatically annotated data. Namely, we built a robust pipeline for collecting Arabic recordings and processing them automatically using voice activity detection,
speech recognition, automatic diacritization, and noise filtering, resulting in around 4,000 hours of Arabic TTS training data. We then trained several robust TTS models with voice cloning using varying amounts of data, namely 100, 1,000, and 4,000 hours with and without diacritization. We show that though models trained on diacritized data are generally better, larger amounts of training data compensate for the lack of diacritics to a significant degree. We plan to release a public Arabic TTS model that works without the need for diacritization.
Read the full abstract ▾
Computation and Language
Markovian ODE-guided scoring can assess the quality of offline reasoning traces in language models
Arghodeep Nandi, Ojasva Saxena and Tanmoy Chakraborty
Reasoning traces produced by generative
language models are increasingly used for tasks ranging from mathematical problem solving to automated fact checking. However, existing evaluation methods remain largely mechanical and fail to capture human-centric notions of
reasoning quality in a way that generalizes across varied and progressively degraded
reasoning. We introduce MarODE, an offline evaluation framework that assigns quality scores to
reasoning traces. Its effectiveness is assessed using human-centric perturbations and human judgments, which jointly evaluate the fundamental dimensions of an evaluation metric - goodness and soundness. The approach is grounded in a Markovian formulation of
reasoning progression and an ordinary differential equation based characterization of trace dynamics, enabling efficient evaluation of
reasoning quality. In a large-scale evaluation, MarODE outperforms existing baselines by over 250% under Somers' D correlation. Our results emphasize the value of theory-driven evaluation frameworks as
reasoning traces become central to
language model-based systems.
Read the full abstract ▾
Computation and Language
Extracting Training Dialogue Data from Large Language Model based Task Bots
Shuo Zhang, Junzhou Zhao, Junji Hou et al. (+3 more)
Large
Language Models (LLMs) have been widely adopted to enhance Task-Oriented Dialogue Systems (TODS) by modeling complex
language patterns and delivering contextually appropriate responses. However, this integration introduces significant
privacy risks, as LLMs, functioning as soft knowledge bases that compress extensive training data into rich knowledge representations, can inadvertently memorize training dialogue data containing not only identifiable information such as phone numbers but also entire dialogue-level events like complete travel schedules. Despite the critical nature of this
privacy concern, how LLM memorization is inherited in developing task bots remains unexplored. In this work, we address this gap through a systematic quantitative study that involves evaluating existing training data extraction attacks, analyzing key characteristics of task-oriented dialogue modeling that render existing methods ineffective, and proposing novel attack techniques tailored for LLM-based TODS that enhance both response sampling and membership inference. Experimental results demonstrate the effectiveness of our proposed data extraction attack. Our method can extract thousands of training labels of dialogue states with best-case precision exceeding 70%. Furthermore, we provide an in-depth analysis of training data memorization in LLM-based TODS by identifying and quantifying key influencing factors and discussing targeted mitigation strategies.
Read the full abstract ▾
All authors Shuo Zhang, Junzhou Zhao, Junji Hou, Pinghui Wang, Chenxu Wang, Jing Tao
Computation and Language
Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap
Ming-Hao Hsu, Xiaohai Tian, Jun Zhang et al. (+3 more)
End-to-end
speech large
language models (LLMs) often answer the same question less accurately from
speech than from text, even when the spoken content matches the text
prompt. Existing analyses study this modality gap through the geometry of representations and measure how closely
speech representations align with text.
Geometric alignment gives the gap a measurable form, but it does not show whether the model uses aligned representations in the same way. This work connects representation geometry with answer formation and asks whether
geometric alignment implies functional equivalence. We compare cross-layer Centered Kernel
Alignment (CKA) with layer-wise answer margins for four open-weight
speech LLMs on SpeechMMLU and VoiceBench BBH. In Qwen2.5-Omni and MiniCPM-o,
speech and text representations align in middle layers while their late-layer answer margins still differ, so mid-layer
alignment does not imply functional equivalence. Qwen2-Audio instead shows weak
alignment, which indicates a different failure mode. An
attention intervention at fixed input and
token count identifies a causal effect of late
attention allocation on answer-code separation, a quantity distinct from answer correctness. Compressing the
speech signal or its
token sequence repairs a subset of the errors that occur only with
speech.
Read the full abstract ▾
All authors Ming-Hao Hsu, Xiaohai Tian, Jun Zhang, Lu Lu, Yuxuan Wang, Zhizheng Wu
Computation and Language · Cross-listed from cs.AI
ProtRLSearch: A Multi-Round Multimodal Protein Search Agent with Large Language Models Trained via Reinforcement Learning
Congying Liu, Taihao Li, Ming Huang et al. (+5 more)
Protein analysis tasks arising in healthcare settings often require accurate
reasoning under
protein sequence constraints, involving tasks such as functional interpretation of disease-related variants,
protein-level analysis for clinical research, and similar scenarios. To address such tasks, search
agents are introduced to search
protein-related information, providing support for disease-related variant analysis and
protein function
reasoning in
protein-centric inference. However, such search
agents are mostly limited to single-round, text-only modality search, which prevents the
protein sequence modality from being incorporated as a
multimodal input into the search decision-making process. Meanwhile, their reliance on reinforcement learning (RL) supervision that focuses solely on the final answer results in a lack of search process constraints, making deviations in keyword selection and
reasoning directions difficult to identify and correct in a timely manner. To address these limitations, we propose ProtRLSearch, a multi-round
protein search
agent trained with multi-dimensional reward based RL, which jointly leverages
protein sequence and text as
multimodal inputs during real-time search to produce high quality reports. To evaluate the ability of models to integrate
protein sequence information and text-based
multimodal inputs in realistic
protein query settings, we construct ProtMCQs, a
benchmark of 3,000 multiple choice questions (MCQs) organized into three difficulty levels. The
benchmark evaluates
protein query tasks that range from sequence constrained
reasoning about
protein function and phenotype changes to comprehensive
protein reasoning that integrates multi-dimensional sequence features with signal pathways and regulatory networks.
Read the full abstract ▾
All authors Congying Liu, Taihao Li, Ming Huang, Xingyuan Wei, Peipei Liu, Yiqing Shen, Yanxu Mao, Tiehan Cui
Computation and Language · Cross-listed from cs.HC
Power Echoes: Investigating Moderation Biases in Online Power-Asymmetric Conflicts
Yaqiong Li, Peng Zhang, Peixu Hou et al. (+7 more)
Online power-asymmetric conflicts are prevalent, and most platforms rely on human moderators to conduct moderation currently. Previous studies have been continuously focusing on investigating human moderation biases in different scenarios, while moderation biases under power-asymmetric conflicts remain unexplored. Therefore, we aim to investigate the types of power-related biases human moderators exhibit in power-asymmetric conflict moderation (RQ1) and further explore the influence of AI's suggestions on these biases (RQ2). For this goal, we conducted a mixed design experiment with 50 participants by leveraging the real conflicts between consumers and merchants as a scenario. Results suggest several biases towards supporting the powerful party within these two moderation modes. AI assistance alleviates most biases of human moderation, but also amplifies a few. Based on these results, we propose several insights into future research on human moderation and human-AI collaborative moderation systems for power-asymmetric conflicts.
Read the full abstract ▾
All authors Yaqiong Li, Peng Zhang, Peixu Hou, Kainan Tu, Guangping Zhang, Shan Qu, Wenshi Chen, Yan Chen, Ning Gu, Tun Lu
Computation and Language · Cross-listed from cs.CV
From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents
Niu Lian, Yuting Wang, Hanshu Yao et al. (+5 more)
While
multimodal large
language models have demonstrated impressive short-term
reasoning, they struggle with long-horizon
video understanding due to limited context windows and static
memory mechanisms that fail to mirror human cognitive efficiency. Existing paradigms typically fall into two extremes:
vision-centric methods that incur high latency and redundancy through dense visual accumulation, or text-centric approaches that suffer from detail loss and
hallucination via aggressive captioning. To bridge this gap, we propose MM-Mem, a pyramidal
multimodal memory architecture grounded in Fuzzy-Trace Theory. MM-Mem structures
memory hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic Schema, enabling the progressive
distillation of fine-grained perceptual traces (verbatim) into high-level semantic schemas (gist). Furthermore, to govern the dynamic construction of
memory, we derive a Semantic Information Bottleneck objective and introduce SIB-GRPO to optimize the trade-off between
memory compression and task-relevant information retention. In inference, we design an
entropy-driven top-down
memory retrieval strategy. Extensive experiments across 4
benchmarks confirm that MM-Mem achieves state-of-the-art performance on both offline and streaming tasks, demonstrating robust generalization and validating the effectiveness of cognition-inspired
memory organization. Code and associated configurations are publicly available at https://github.com/EliSpectre/MM-Mem.
Read the full abstract ▾
All authors Niu Lian, Yuting Wang, Hanshu Yao, Jinpeng Wang, Bin Chen, Yaowei Wang, Min Zhang, Shu-Tao Xia
Computation and Language
Enhancing Persona Following at Decoding Time via Dynamic Importance Estimation for Role-Playing Agents
Yuxin Liu, Mingye Zhu, Siyuan Liu et al. (+2 more)
The utility of Role-Playing
Language Agents in sociological research is growing alongside the adoption of Large
Language Models. For realism in social simulation, these
agents must adhere to their personas defined by character profiles, yet existing strategies-static
prompt engineering or costly fine-tuning-fail to adapt personas to dynamic scenarios. Psychological theories, such as the Cognitive-Affective Personality Systems, provide a crucial explanation for this failure: a persona's influence on behavior is not static but varies with the scenarios. This context-dependence highlights the critical need for adaptive persona management. To address this gap, we propose a novel, theory-driven method that dynamically estimates context-dependent persona importance and integrates it into weighted reward-guided decoding, enabling inference-time persona following. Specifically, we introduce the Persona Dynamic Decoding (PDD) framework, which consists of two key components: (1) Persona Importance Estimation (PIE) module, which dynamically quantifies the contextual importance of persona attributes without requiring ground-truth supervision; and (2) Persona-Guided Inference-Time
Alignment (PIA) paradigm, which leverages these importance scores to construct weighted multi-objective rewards and modulate generation probabilities during inference. Extensive experiments show the effectiveness of our method in utterance consistency and behavioral fidelity.
Read the full abstract ▾
All authors Yuxin Liu, Mingye Zhu, Siyuan Liu, Bo Hu, Lei Zhang
Computation and Language
Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics
Samhruth Ananthanarayanan, Ayan Sengupta and Tanmoy Chakraborty
As context windows in LLMs scale to 100K+
tokens, the key-value (KV) cache becomes the dominant
memory bottleneck, with recent methods claiming 80-90% savings and minimal
benchmark degradation. We argue these evaluations miss a structural issue:
attention is not just storage but routing, and retaining KV pairs does not guarantee semantic accessibility. We propose a physics-inspired view of KV compression as a controlled perturbation of
token-level routing, distinguishing retention, accessibility, and utilization. Using synthetic tasks probing multi-entity tracking, disambiguation, coreference, and multi-hop
reasoning, we find that moderate compression degrades internal representations with little accuracy loss, revealing redundancy; all models exhibit a sharp
hallucination safety cliff near 90% compression, correlated with spikes in Global Eviction Ratio (GER), suggesting a phase transition in semantic reachability; and architectures differ in routing dynamics, with LLaMA showing early consensus and late diversification, and Qwen showing funnel-like late convergence, leading to distinct resilience profiles. Beyond erasure, we identify representational
rigidity, where excessive head-level consensus collapses routing flexibility despite
token survival. These results suggest sparse
token-route structures govern compression tolerance, reframing KV compression as a structural probe of
attention geometry and linking long-context scalability to sparsity and the lottery ticket hypothesis in self-
attention.
Read the full abstract ▾
Computation and Language
LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrieval
Jiajie Jin, Yanzhao Zhang, Mingxin Li et al. (+4 more)
LLMs have fundamentally transformed dense
retrieval, upgrading backbones from discriminative encoders to generative architectures. However, a critical disconnect remains: while LLMs possess strong
reasoning capabilities, current retrievers predominantly utilize them as static encoders, leaving their potential for complex
reasoning unexplored. To address this, existing approaches typically adopt rewrite-then-retrieve pipelines to generate explicit CoT rationales before
retrieval. However, this incurs prohibitive latency. In this paper, we propose
LaSER, a novel self-
distillation framework that internalizes explicit
reasoning into the latent space of dense retrievers. Operating on a shared LLM backbone,
LaSER introduces a dual-view training mechanism: an Explicit view that explicitly encodes ground-truth
reasoning paths, and a Latent view that performs implicit latent thinking. To bridge the gap between these views, we design a multi-grained
alignment strategy. Beyond standard output
alignment, we introduce a trajectory
alignment mechanism that synchronizes the intermediate latent states of the latent path with the semantic progression of the explicit
reasoning segments. This allows the retriever to think silently and effectively without autoregressive text generation. Extensive experiments on both in-domain and out-of-domain
reasoning-intensive
benchmarks demonstrate that
LaSER significantly outperforms state-of-the-art baselines. Furthermore, analyses across diverse backbones and model scales validate the robustness of our approach, confirming that our unified learning framework is essential for eliciting effective latent thinking. Our method successfully combines the
reasoning depth of explicit CoT pipelines with the inference efficiency of standard dense retrievers.
Read the full abstract ▾
All authors Jiajie Jin, Yanzhao Zhang, Mingxin Li, Dingkun Long, Pengjun Xie, Yutao Zhu, Zhicheng Dou
Computation and Language
Quantifying Conversational Reliability of Large Language Models under Multi-Turn Interaction
Jiyoon Myung
Large
Language Models (LLMs) are increasingly deployed in real-world applications where users engage in extended, mixed-topic conversations that depend on prior context. Yet, their reliability under realistic multi-turn interactions remains poorly understood. We conduct a systematic evaluation of conversational reliability through three representative tasks that reflect practical interaction challenges: (1) maintaining global constraints across topic shifts, (2) selecting the correct tool or
agent amid interleaved intents, and (3) tracking structured entities under revisions and distractions. Each task pairs single-turn and multi-turn settings, allowing us to quantify reliability degradation under extended dialogue. Across both commercial and open-source models, we observe substantial declines in reliability, particularly for smaller models. Error analyses reveal recurring failure modes such as instruction drift, intent confusion, and contextual overwriting, which compromise dependable behavior in operational systems. Our findings highlight the need for stress-testing LLMs for conversational reliability and developing more robust evaluation methods for trustworthy deployment.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
SciDER: Scientific Data-centric End-to-end Researcher
Ke Lin, Owais Aijaz, Yilin Lu et al. (+3 more)
While large
language models accelerate scientific discovery, existing
agents face severe limitations in adaptability, domain generalization, and
multimodal scalability, often struggling to autonomously process raw, domain-specific experimental data. To overcome these barriers, we introduce SciDER, a multi-
agent system designed to flexibly automate the entire research lifecycle. This framework employs a novel data-centric approach and integrates a dynamic
multimodal skill system across four specialized sub-
agents. Specifically, an ideation
agent generates novel hypotheses via Evolutionary Idea Search, a data analysis
agent systematically structures raw data, an experimentation
agent synthesizes executable code grounded in
dataset characteristics, and a critic
agent drives iterative self-refinement. To democratize open-source scientific discovery, we release OpenSciDER-SFT-8K, a high-quality execution trajectory
dataset, alongside the OpenSciDER-27B fine-tuned model. Across six
benchmarks, SciDER and OpenSciDER obtain competitive or leading results, with especially strong gains on data-centric analysis, end-to-end research execution, and
multimodal scientific visualization. By integrating data analysis with experimental execution, SciDER bridges the gap between abstract scientific
reasoning and reproducible experimentation synthesis.
Read the full abstract ▾
All authors Ke Lin, Owais Aijaz, Yilin Lu, Yiyang Luo, Xuehang Guo, Preslav Nakov
Computation and Language
Toward Graph-Tokenizing Large Language Models with Reconstructive Graph Instruction Tuning
Zhongjian Zhang, Xiao Wang, Mengmei Zhang et al. (+2 more)
The remarkable success of large
language models (LLMs) has motivated researchers to adapt them as universal predictors for various graph-related tasks, with the ultimate goal of developing a graph foundation model that generalizes diverse scenarios. The key challenge is to align graph data with
language spaces so that LLMs can better comprehend graphs. As a popular paradigm, Graph-Tokenizing LLMs (GTokenLLMs) encode complex structures and lengthy texts into a graph
token sequence, and then align them with text
tokens via
language instructions tuning. Despite their initial success, our information-theoretic analysis reveals that existing GTokenLLMs rely solely on text supervision from
language instructions, which achieve only implicit graph-text
alignment, resulting in a text-dominant bias that underutilizes graph context. To overcome this limitation, we first prove that the
alignment objective is upper-bounded by the mutual information between the input graphs and their hidden representations in the LLM, which motivates us to improve this upper bound to achieve better
alignment. To this end, we further propose a reconstructive graph instruction tuning pipeline, RGLM. Our key idea is to reconstruct the graph information from the LLM's graph
token outputs, explicitly incorporating graph supervision to constrain the
alignment process. Technically, we embody RGLM by exploring three distinct variants from two complementary perspectives: RGLM-Decoder from the input space; RGLM-Similarizer and RGLM-Denoiser from the latent space. Additionally, we theoretically analyze the
alignment effectiveness of each variant. Extensive experiments on various
benchmarks and task scenarios validate the effectiveness of the proposed RGLM, paving the way for new directions in GTokenLLMs'
alignment research.
Read the full abstract ▾
All authors Zhongjian Zhang, Xiao Wang, Mengmei Zhang, Jiarui Tan, Chuan Shi
Computation and Language · Cross-listed from cs.SD
End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation
Minghui Wu, Haitao Tang, Jiahuan Fan et al. (+2 more)
Dysarthric
speech reconstruction (DSR) typically employs a cascaded system that combines automatic
speech recognition (ASR) and sentence-level text-to-
speech (TTS) to convert dysarthric
speech into normally-prosodied
speech. However, dysarthric individuals often speak more slowly, leading to excessively long response times in such systems, rendering them impractical in long-
speech scenarios. Cascaded DSR systems based on streaming ASR and incremental TTS can help reduce latency. However, patients with differing dysarthria severity exhibit substantial pronunciation variability for the same text, resulting in poor robustness of ASR and limiting the intelligibility of reconstructed
speech. In addition, incremental TTS suffers from poor prosodic feature prediction due to a limited receptive field. In this study, we propose an end-to-end simultaneous DSR system with two key innovations: 1) A frame-level adaptor module is introduced to bridge ASR and TTS. By employing explicit-implicit semantic information fusion and joint module training, it enhances the error tolerance of TTS to ASR outputs. 2) A multiple wait-k autoregressive TTS module is designed to mitigate prosodic degradation via multi-view knowledge
distillation. Our system has an average response time of 1.03 seconds on Tesla A100, with an average real-time factor (RTF) of 0.71. On the UASpeech
dataset, it attains a mean opinion score (MOS) of 4.67 and demonstrates a 54.25% relative reduction in word error rate (WER) compared to the state-of-the-art. Our demo is available at: https://wflrz123.github.io/
Read the full abstract ▾
All authors Minghui Wu, Haitao Tang, Jiahuan Fan, Ruizhi Liao, Yanyong Zhang
Computation and Language · Cross-listed from cs.SD
DARS: Dysarthria-Aware Rhythm-Style Synthesis for ASR Enhancement
Minghui Wu, Xueling Liu, Jiahuan Fan et al. (+3 more)
Dysarthric
speech exhibits abnormal prosody and significant speaker variability, presenting persistent challenges for automatic
speech recognition (ASR). While text-to-
speech (TTS)-based data augmentation has shown potential, existing methods often fail to accurately model the pathological rhythm and acoustic style of dysarthric
speech. To address this, we propose DARS, a dysarthria-aware rhythm-style synthesis framework based on the Matcha-TTS architecture. DARS incorporates a multi-stage rhythm predictor optimized by contrastive preferences between normal and dysarthric
speech, along with a dysarthric-style conditional flow matching mechanism, jointly enhancing temporal rhythm reconstruction and pathological acoustic style simulation. Experiments on the TORGO
dataset demonstrate that DARS achieves a Mean Cepstral Distortion (MCD) of 4.29, closely approximating real dysarthric
speech. Adapting a Whisper-based ASR system with synthetic dysarthric
speech from DARS achieves a 54.22% relative reduction in word error rate (WER) compared to state-of-the-art methods, demonstrating the framework's effectiveness in enhancing recognition performance.
Read the full abstract ▾
All authors Minghui Wu, Xueling Liu, Jiahuan Fan, Haitao Tang, Yanyong Zhang, Yue Zhang
Computation and Language · Cross-listed from cs.LO
NM-DEKL$^3_\infty$: A Three-Layer Non-Monotone Evolving Dependent Type Logic
Peng Chen
We present a new dependent type system, NM-DEKL$^3_\infty$ (Non-Monotone Dependent Knowledge-Enhanced Logic), for formalising evolving knowledge in dynamic environments. The system uses a three-layer architecture separating a computational layer, a constructive knowledge layer, and a propositional knowledge layer. We define its syntax and semantics and establish Soundness and Equational Completeness; we construct a syntactic model and prove that it is initial in the category of models, from which equational completeness follows. We also give an
embedding into the $\mu$-calculus and a strict expressiveness inclusion (including the expressibility of non-bisimulation-invariant properties).
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Constructing Synthetic Instruction Datasets for Improving Reasoning in Domain-Specific LLMs: A Case Study in the Japanese Financial Domain
Yuma Okochi, Fabio Milentiansen Sim and Tomoyasu Okada
In adapting LLMs to specific domains, achieving both domain expertise and
reasoning ability remains an urgent challenge. This study proposes a general method for constructing high-quality synthetic instruction data for any domain, starting from domain-specific vocabulary. As a demonstration, we applied this method to the financial domain and constructed a large-scale instruction
dataset totaling approximately 9.5 billion
tokens with Chain-of-Thought
reasoning traces. Evaluation results confirmed performance improvements over baseline models on financial
benchmarks, demonstrating the effectiveness of our approach. We also report findings on the impact of
reasoning trace length on performance and its limitations. Lastly, we open-source our models and
datasets on https://huggingface.co/nri-ai .
Read the full abstract ▾