Computation and Language
Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts
Wenhao Li, Daohai Yu, Gen Luo et al. (+7 more)
Training Large
Language Models (LLMs) on long contexts is severely constrained by prohibitive GPU
memory overhead, not training time. The primary culprits are the activations, whose
memory footprints scale linearly with sequence length. We introduce OOMB, a highly
memory-efficient training system that directly confronts this barrier. Our approach employs a chunk-recurrent training framework with on-the-fly activation recomputation, which maintains a constant activation
memory footprint (O(1)) and shifts the primary bottleneck to the growing KV cache. To manage the KV cache, OOMB integrates a suite of synergistic optimizations: a paged
memory manager for both the KV cache and its gradients to eliminate fragmentation, asynchronous CPU offloading to hide data transfer latency, and page-level sparse
attention to reduce both computational complexity and communication overhead. The synergy of these techniques yields exceptional efficiency. Our empirical results show that for every additional 10K
tokens of context, the end-to-end training
memory overhead increases by a mere 10MB for Qwen2.5-7B. This allows training Qwen2.5-7B with a 4M-
token context on a single H200 GPU, a feat that would otherwise require a large cluster using context parallelism. This work represents a substantial advance in resource efficiency for long-context LLM training. The source code is available at https://github.com/wenhaoli-xmu/OOMB.
Read the full abstract ▾
All authors Wenhao Li, Daohai Yu, Gen Luo, Yuxin Zhang, Fei Chao, Rongrong Ji, Yifan Wu, Jiaxin Liu, Ziyang Gong, Zimu Liao
Computation and Language
Dicta-LM 3.0: Advancing The Frontier of Hebrew Sovereign LLMs
Shaltiel Shmidman, Avi Shmidman, Amir DN Cohen et al. (+1 more)
Open-weight LLMs have been released by frontier labs; however, sovereign Large
Language Models (for
languages other than English) remain low in supply yet high in demand. Training large
language models (LLMs) for low-resource
languages such as Hebrew poses unique challenges. In this paper, we introduce Dicta-LM 3.0: an open-weight collection of LLMs trained on substantially-sized corpora of Hebrew and English texts. The model is released in three sizes: 24B - adapted from the Mistral-Small-3.1 base model, 12B - adapted from the NVIDIA Nemotron Nano V2 model, and 1.7B - adapted from the Qwen3-1.7B base model. We are releasing multiple variants of each model, each with a native context length of 65k
tokens; base model and chat model with tool-calling support. To rigorously evaluate our models, we introduce a new
benchmark suite for evaluation of Hebrew chat-LLMs, covering a diverse set of tasks including Translation, Summarization, Winograd, Israeli Trivia, and Diacritization (nikud). Our work not only addresses the intricacies of training LLMs in low-resource
languages but also proposes a framework that can be leveraged for adapting other LLMs to various non-English
languages, contributing to the broader field of
multilingual NLP.
Read the full abstract ▾
All authors Shaltiel Shmidman, Avi Shmidman, Amir DN Cohen, Moshe Koppel
Computation and Language · Cross-listed from cs.LG
How Far Ahead Do LLMs Plan? Uncovering the Latent Horizon in Chain-of-Thought Reasoning
Liyan Xu, Mo Yu, Fandong Meng et al. (+1 more)
Chain-of-thought (CoT)
reasoning has become a central mechanism for eliciting multi-step
reasoning in Large
Language Models (LLMs). Yet recent evidence presents a tension: hidden states appear to already encode future
reasoning before CoT fully unfolds, while explicit steps still remain crucial for tasks requiring compositional computation. To deepen the understanding between LLM's internal states and its verbalized
reasoning trajectories, we investigate the latent planning strength of LLMs, through our probing method, Tele-Lens, applying to hidden states across diverse task domains. Our empirical results indicate that LLMs exhibit a myopic horizon, primarily conducting incremental transitions without precise global planning. Leveraging this characteristic, we propose a hypothesis on enhancing uncertainty estimation of CoT, which we validate that a sparse set of pivot positions can effectively represent the uncertainty of the entire path. We further underscore the significance of exploiting CoT dynamics, and demonstrate that automatic recognition of CoT bypass can be achieved without performance degradation. Our code, data and models are released at https://github.com/lxucs/tele-lens.
Read the full abstract ▾
All authors Liyan Xu, Mo Yu, Fandong Meng, Jie Zhou
Computation and Language
Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning
Keqin Peng, Yuanxin Ouyang, Xuebo Liu et al. (+4 more)
Reinforcement Learning with Verifiable Rewards (RLVR) can elicit strong multi-step
reasoning, yet it often encourages overly verbose traces. Moreover, naive length penalties in group-relative optimization can severely hurt accuracy. We attribute this failure to two structural issues: (i) Dilution of Length Baseline, where incorrect responses (with zero length reward) depress the group baseline and over-penalize correct solutions; and (ii) Difficulty-Penalty Mismatch, where a static penalty cannot adapt to problem difficulty, suppressing necessary
reasoning on hard instances while leaving redundancy on easy ones. We propose Dynamic Decoupled Conditional Advantage (DDCA) to decouple efficiency optimization from correctness. DDCA computes length advantages conditionally within the correct-response cluster to eliminate baseline dilution, and dynamically scales the penalty strength using the group pass rate as a proxy for difficulty. Experiments on GSM8K, MATH500, AMC23, and AIME25 show that DDCA consistently improves the efficiency–accuracy trade-off relative to adaptive baselines, reducing generated
tokens by approximately 60% on simpler tasks (e.g., GSM8K) versus over 20% on harder
benchmarks (e.g., AIME25), thereby maintaining or improving accuracy. Code is available at https://github.com/alphadl/DDCA.
Read the full abstract ▾
All authors Keqin Peng, Yuanxin Ouyang, Xuebo Liu, Zhiliang Tian, Ruijian Han, Yancheng Yuan, Liang Ding
Computation and Language
LEC-KG: An LLM-Embedding Collaborative Framework for Domain-Specific Knowledge Graph Construction – A Case Study on SDGs
Yikai Zeng, Yingchao Piao, Changhua Pei et al. (+1 more)
Constructing domain-specific knowledge graphs from unstructured text remains challenging due to heterogeneous entity mentions, long-tail relation distributions, and the absence of standardized schemas. We present LEC-KG, a bidirectional collaborative framework that integrates the semantic understanding of Large
Language Models (LLMs) with the structural
reasoning of Knowledge Graph
Embeddings (KGE). Our approach features three key components: (1) hierarchical coarse-to-fine relation extraction that mitigates long-tail bias, (2) evidence-guided Chain-of-Thought feedback that grounds structural suggestions in source text, and (3) semantic initialization that enables structural validation for unseen entities. The two modules enhance each other iteratively-KGE provides structure-aware feedback to refine LLM extractions, while validated triples progressively improve KGE representations. We evaluate LEC-KG on Chinese Sustainable Development Goal (SDG) reports, demonstrating substantial improvements over LLM baselines, particularly on low-frequency relations. Through iterative refinement, our framework reliably transforms unstructured policy text into validated knowledge graph triples.
Read the full abstract ▾
All authors Yikai Zeng, Yingchao Piao, Changhua Pei, Jianhui Li
Computation and Language
Closing the Loop: Universal Repository Representation with RPG-Encoder
Jane Luo, Chengyu Yin, Xin Zhang et al. (+10 more)
Current repository
agents encounter a
reasoning disconnect due to fragmented representations, as existing methods rely on isolated API documentation or dependency graphs that lack semantic depth. We consider repository comprehension and generation to be inverse processes within a unified cycle: generation expands intent into implementation, while comprehension compresses implementation back into intent. To address this, we propose RPG-Encoder, a framework that generalizes the Repository Planning Graph (RPG) from a static generative blueprint into a unified, high-fidelity representation. RPG-Encoder closes the
reasoning loop through three mechanisms: (1) Encoding raw code into the RPG that combines lifted semantic features with code dependencies; (2) Evolving the
topology incrementally to decouple maintenance costs from repository scale, reducing overhead by 95.7%; and (3) Operating as a unified interface for structure-aware
navigation. In evaluations, RPG-Encoder establishes state-of-the-art localization performance on SWE-bench Verified with 93.7% Acc@5 and exceeds the best baseline by over 10% in localization accuracy on SWE-bench Live Lite. These results highlight our superior fine-grained precision in complex codebases. Furthermore, it achieves 98.5% reconstruction coverage on RepoCraft, confirming RPG's high-fidelity capacity to mirror the original codebase and closing the loop between intent and implementation.
Read the full abstract ▾
All authors Jane Luo, Chengyu Yin, Xin Zhang, Qingtao Li, Steven Liu, Yiming Huang, Jie Wu, Hao Liu, Yangyu Huang, Yu Kang, Fangkai Yang, Ying Xin, Scarlett Li
Computation and Language
WildGraphBench: Benchmarking GraphRAG with Wild-Source Corpora
Pengyu Wang, Benfeng Xu, Licheng Zhang et al. (+4 more)
Graph-based
Retrieval-Augmented Generation (GraphRAG) organizes external knowledge as a hierarchical graph, enabling efficient
retrieval and aggregation of scattered evidence across multiple documents. However, many existing
benchmarks for GraphRAG rely on short, curated passages as external knowledge, failing to adequately evaluate systems in realistic settings involving long contexts and large-scale heterogeneous documents. To bridge this gap, we introduce WildGraphBench, a
benchmark designed to assess GraphRAG performance in the wild. We leverage Wikipedia's unique structure, where cohesive narratives are grounded in long and heterogeneous external reference documents, to construct a
benchmark reflecting real-word scenarios. Specifically, we sample articles across 12 top-level topics, using their external references as the
retrieval corpus and citation-linked statements as ground truth, resulting in 1,100 questions spanning three levels of complexity: single-fact QA, multi-fact QA, and section-level summarization. Experiments across multiple baselines reveal that current GraphRAG pipelines help on multi-fact aggregation when evidence comes from a moderate number of sources, but this aggregation paradigm may overemphasize high-level statements at the expense of fine-grained details, leading to weaker performance on summarization tasks. Project page:https://github.com/BstWPY/WildGraphBench.
Read the full abstract ▾
All authors Pengyu Wang, Benfeng Xu, Licheng Zhang, Shaohan Wang, Mingxuan Du, Chiwei Zhu, Zhendong Mao
Computation and Language · Cross-listed from cs.LG
Dissecting Outlier Dynamics in LLM NVFP4 Pretraining
Peijie Dong, Ruibo Fan, Yuechen Tao et al. (+11 more)
Training large
language models using 4-bit arithmetic enhances throughput and
memory efficiency. Yet, the limited dynamic range of FP4 increases sensitivity to outliers. While NVFP4 mitigates quantization error via hierarchical microscaling, a persistent loss gap remains compared to BF16. This study conducts a longitudinal analysis of outlier dynamics across architecture during NVFP4 pretraining, focusing on where they localize, why they occur, and how they evolve temporally. We find that, compared with Softmax
Attention (SA), Linear
Attention (LA) reduces per-tensor heavy tails but still exhibits persistent block-level spikes under block quantization. Our analysis attributes outliers to specific architectural components: Softmax in SA, gating in LA, and SwiGLU in FFN, with "post-QK" operations exhibiting higher sensitivity to quantization. Notably, outliers evolve from transient spikes early in training to a small set of persistent hot channels (i.e., channels with persistently large magnitudes) in later stages. Based on these findings, we introduce Hot-Channel Patch (HCP), an online compensation mechanism that identifies hot channels and reinjects residuals using hardware-efficient kernels. We then develop CHON, an NVFP4 training recipe integrating HCP with post-QK operation protection. On GLA-1.3B model trained for 60B
tokens, CHON reduces the loss gap to BF16 from 0.94% to 0.58% while maintaining downstream accuracy.
Read the full abstract ▾
All authors Peijie Dong, Ruibo Fan, Yuechen Tao, Di Mou, Wenhu Hu, Zhenheng Tang, Yinghao Yu, Jiamang Wang, Wenbo Su, Guodong Yang, Liping Zhang, Xiaowen Chu, Baochun Li, Bo Li
Computation and Language · Cross-listed from cs.AI
Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models
Wei Liu, Peijie Yu, Michele Orini et al. (+2 more)
The agency expected of Agentic Large
Language Models goes beyond answering correctly, requiring autonomy to set goals and decide what to explore. We term this investigatory intelligence, distinguishing it from executional intelligence, which merely completes assigned tasks. Data Science provides a natural testbed, as real-world analysis starts from raw data rather than explicit queries, yet few
benchmarks focus on it. To address this, we introduce Deep Data Research (DDR), an open-ended task where LLMs autonomously extract key insights from databases, and DDR-Bench, a large-scale, checklist-based
benchmark that enables verifiable evaluation. Results show that while frontier models display emerging agency, long-horizon exploration remains challenging. Our analysis highlights that effective investigatory intelligence depends not only on
agent scaffolding or merely scaling, but also on intrinsic strategies of agentic models.
Read the full abstract ▾
All authors Wei Liu, Peijie Yu, Michele Orini, Yali Du, Yulan He
Computation and Language · Cross-listed from cs.CV
Rethinking Genomic Modeling Through Optical Character Recognition
Hongxin Xiang, Pengsen Ma, Yunkang Cao et al. (+4 more)
Recent genomic foundation models largely adopt large
language model architectures that treat DNA as a one-dimensional
token sequence. However, exhaustive sequential reading is structurally misaligned with sparse and discontinuous genomic semantics, leading to wasted computation on low-information background and preventing understanding-driven compression for long contexts. Here, we present OpticalDNA, a
vision-based framework that reframes genomic modeling as Optical Character Recognition (OCR)-style document understanding. OpticalDNA renders DNA into structured visual layouts and trains an OCR-capable
vision–
language model with a visual DNA encoder and a document decoder, where the encoder produces compact, reconstructible visual
tokens for high-fidelity compression. Building on this representation, OpticalDNA defines
prompt-conditioned objectives over core genomic primitives-reading, region grounding, subsequence
retrieval, and masked span completion-thereby learning layout-aware DNA representations that retain fine-grained genomic information under a reduced effective
token budget. Across diverse genomic
benchmarks, OpticalDNA consistently outperforms recent baselines; on sequences up to 450k bases, it achieves the best overall performance with nearly 20$\times$ fewer effective
tokens, and surpasses models with up to 985$\times$ more activated parameters while tuning only 256k trainable parameters.
Read the full abstract ▾
All authors Hongxin Xiang, Pengsen Ma, Yunkang Cao, Di Yu, Haowen Chen, Xinyu Yang, Xiangxiang Zeng
Computation and Language
NEAT: Neuron-Based Early Exit for Large Reasoning Models
Kang Liu, Yongkang Liu, Xiaocui Yang et al. (+5 more)
Large
Reasoning Models (LRMs) often suffer from overthinking, a phenomenon in which redundant
reasoning steps are generated after a correct solution has already been reached. Existing early
reasoning exit methods primarily rely on output-level heuristics or trained probing models to skip redundant
reasoning steps, thereby mitigating overthinking. However, these approaches typically require additional rollout computation or externally labeled
datasets. In this paper, we propose NEAT, a
Neuron-based Early
reAsoning exiT framework that monitors
neuron-level activation dynamics to enable training-free early exits, without introducing additional test-time computation. NEAT identifies exit-associated
neurons and tracks their activation patterns during
reasoning to dynamically trigger early exit or suppress reflection, thereby reducing unnecessary
reasoning while preserving solution quality. Experiments on four
reasoning benchmarks across six models with different scales and architectures show that, for each model, NEAT achieves an average
token reduction of 22% to 28% when averaged over the four
benchmarks, while maintaining accuracy.
Read the full abstract ▾
All authors Kang Liu, Yongkang Liu, Xiaocui Yang, Peidong Wang, Wen Zhang, Shi Feng, Yifei Zhang, Daling Wang
Computation and Language
Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation
Zhanghao Hu, Qinglin Zhu, Runcong Zhao et al. (+4 more)
Standard
Retrieval Augmented Generation (RAG) is poorly matched to
agent memory. Unlike large heterogeneous corpora,
agent memory forms a bounded and coherent interaction stream in which many spans are highly correlated or near duplicates. As a result, flat top-$k$ similarity
retrieval often returns redundant context, while summary-centric hierarchies can blur the subtle details that distinguish one candidate from another. We argue that
agent memory should follow the principle of decoupling before aggregation: the system should first isolate reusable facts, updates, and distinguishing details from similar histories, and only then organise them for efficient
retrieval. Based on this principle, we propose xMemory, which constructs a revisable hierarchical
memory structure from original messages to segments,
memory components, and groups. xMemory segments interaction history into local events, decouples each segment into
memory components, aggregates related components into high-level groups using a sparsity–semantic faithfulness objective, and maintains this structure incrementally as
memory evolves. At inference time, xMemory retrieves top-down, first selecting a compact backbone of complementary groups and components, and then expanding to segments and raw messages only when additional evidence reduces the reader's uncertainty. Experiments on LoCoMo and PerLTQA across diverse open source and closed source LLMs show consistent gains in answer quality and inference
token efficiency, supported by analyses of redundancy, evidence density, and coverage.
Read the full abstract ▾
All authors Zhanghao Hu, Qinglin Zhu, Runcong Zhao, Di Liang, Hanqi Yan, Yulan He, Lin Gui
Computation and Language
From Latent Signals to Reflection Behavior: Tracing Meta-Cognitive Activation Trajectory in R1-Style LLMs
Yanrui Du, Yibo Gao, Sendong Zhao et al. (+6 more)
R1-style LLMs have attracted growing
attention for their capacity for self-reflection, yet the internal mechanisms underlying such behavior remain unclear. To bridge this gap, we anchor on the onset of reflection behavior and trace its layer-wise activation trajectory. Using the logit lens to read out
token-level semantics, we uncover a structured progression: (i) Latent-control layers, where an approximate linear direction encodes the semantics of thinking budget; (ii) Semantic-pivot layers, where discourse-level cues, including turning-point and summarization cues, surface and dominate the probability mass; and (iii) Behavior-overt layers, where the likelihood of reflection-behavior
tokens begins to rise until they become highly likely to be sampled. Moreover, our targeted interventions uncover a causal chain across these stages:
prompt-level semantics modulate the projection of activations along latent-control directions, thereby inducing competition between turning-point and summarization cues in semantic-pivot layers, which in turn regulates the sampling likelihood of reflection-behavior
tokens in behavior-overt layers. Collectively, our findings suggest a human-like meta-cognitive process-progressing from latent monitoring, to discourse-level regulation, and to finally overt self-reflection. Our analysis code can be found at https://github.com/DYR1/S3-CoT.
Read the full abstract ▾
All authors Yanrui Du, Yibo Gao, Sendong Zhao, Jiayun Li, Haochun Wang, Qika Lin, Kai He, Bing Qin, Mengling Feng
Computation and Language · Cross-listed from cs.AI
Think Like a Doctor: Conversational Diagnosis through the Exploration of Diagnostic Knowledge Graphs
Jeongmoon Won, Seungwon Kook and Yohan Jo
Conversational diagnosis requires multi-turn history-taking, where an
agent asks clarifying questions to refine differential diagnoses under incomplete information. Existing approaches often rely on the parametric knowledge of a model or assume that patients provide rich and concrete information, which is unrealistic. To address these limitations, we propose a conversational diagnosis system that explores a diagnostic knowledge graph to reason in two steps: (i) generating diagnostic hypotheses from the dialogue context, and (ii) verifying hypotheses through clarifying questions, which are repeated until a final diagnosis is reached. Since evaluating the system requires a realistic patient simulator that responds to the system's questions, we adopt PatientSim, a persona-driven patient simulator, together with patient profiles from MIMIC-IV. We further adapt it with low-specificity symptom reporting to reflect how real-world patients describe symptoms vaguely during early clinical encounters. Experiments show improved diagnostic accuracy and efficiency over strong baselines, and physician evaluations support the realism of our simulator and the clinical utility of the generated clarifying questions. Our code will be released upon publication.
Read the full abstract ▾
Computation and Language
S3-CoT: Self-Sampled Succinct Reasoning Enables Efficient Chain-of-Thought LLMs
Yanrui Du, Sendong Zhao, Yibo Gao et al. (+9 more)
Large
language models (LLMs) equipped with chain-of-thought (CoT) achieve strong performance and offer a window into LLM behavior. However, recent evidence suggests that improvements in CoT capabilities often come with redundant
reasoning processes, motivating a key question: Can LLMs acquire a fast-thinking mode analogous to human System 1
reasoning? To explore this, our study presents a self-sampling framework based on activation steering for efficient CoT learning. Our method can induce style-aligned and variable-length
reasoning traces from target LLMs themselves without any teacher guidance, thereby alleviating a central bottleneck of SFT-based methods-the scarcity of high-quality supervision data. Using filtered data by gold answers, we perform SFT for efficient CoT learning with (i) a human-like dual-cognitive system, and (ii) a progressive compression curriculum. Furthermore, we explore a self-
evolution regime in which SFT is driven solely by prediction-consistent data of variable-length variants, eliminating the need for gold answers. Extensive experiments on math
benchmarks, together with cross-domain generalization tests in medicine, show that our method yields stable improvements for both general and R1-style LLMs. Our data and model checkpoints can be found at https://github.com/DYR1/S3-CoT.
Read the full abstract ▾
All authors Yanrui Du, Sendong Zhao, Yibo Gao, Danyang Zhao, Qika Lin, Ming Ma, Jiayun Li, Yi Jiang, Kai He, Qianyi Xu, Bing Qin, Mengling Feng
Computation and Language
Beyond Local Edits: Embedding-Virtualized Knowledge for Broader Evaluation and Preservation of Model Editing
Shuainan Liu, Xuanang Chen, Ben He et al. (+1 more)
Knowledge editing methods for large
language models are commonly evaluated using predefined
benchmarks that assess edited facts together with a limited set of related or neighboring knowledge. While effective, such evaluations remain confined to finite,
dataset-bounded samples, leaving the broader impact of editing on the model's knowledge system insufficiently understood. To address this gap, we introduce
Embedding-Virtualized Knowledge (EVK) that characterizes model knowledge through controlled perturbations in
embedding space, enabling the exploration of a substantially broader and virtualized knowledge region beyond explicit data annotations. Based on EVK, we construct an
embedding-level evaluation
benchmark EVK-Bench that quantifies potential knowledge drift induced by editing, revealing effects that are not captured by conventional sample-based metrics. Furthermore, we propose a plug-and-play EVK-Align module that constrains
embedding-level knowledge drift during editing and can be seamlessly integrated into existing editing methods. Experiments demonstrate that our approach enables more comprehensive evaluation while significantly improving knowledge preservation without sacrificing editing accuracy.
Read the full abstract ▾
All authors Shuainan Liu, Xuanang Chen, Ben He, Le Sun
Computation and Language
Orthogonal Hierarchical Decomposition for Structure-Aware Table Understanding with Large Language Models
Bin Cao, Huixian Lu, Chenwen Ma et al. (+3 more)
Complex tables with multi-level headers, merged
cells and heterogeneous layouts pose persistent challenges for LLMs in both understanding and
reasoning. Existing approaches typically rely on table linearization or normalized grid modeling. However, these representations struggle to explicitly capture hierarchical structures and cross-dimensional dependencies, which can lead to misalignment between structural semantics and textual representations for non-standard tables. To address this issue, we propose an Orthogonal Hierarchical Decomposition (OHD) framework that constructs structure-preserving input representations of complex tables for LLMs. OHD introduces an Orthogonal Tree Induction (OTI) method based on spatial–semantic co-constraints, which decomposes irregular tables into a column tree and a row tree to capture vertical and horizontal hierarchical dependencies, respectively. Building on this representation, we design a dual-pathway association protocol to symmetrically reconstruct semantic lineage of each
cell, and incorporate an LLM as a semantic arbitrator to align multi-level semantic information. We evaluate OHD framework on two complex table question answering
benchmarks, AITQA and HiTab. Experimental results show that OHD consistently outperforms existing representation paradigms across multiple evaluation metrics.
Read the full abstract ▾
All authors Bin Cao, Huixian Lu, Chenwen Ma, Ting Wang, Ruizhe Li, Jing Fan
Computation and Language
Mixture-of-Experts with Intermediate CTC Supervision for Accented Speech Recognition
Wonjun Lee, Hyounghun Kim and Gary Geunbae Lee
Accented
speech remains a persistent challenge for automatic
speech recognition (ASR), as most models are trained on data dominated by a few high-resource English varieties, leading to substantial performance degradation for other accents. Accent-agnostic approaches improve robustness yet struggle with heavily accented or unseen varieties, while accent-specific methods rely on limited and often noisy labels. We introduce Moe-Ctc, a Mixture-of-Experts architecture with intermediate CTC supervision that jointly promotes expert specialization and generalization. During training, accent-aware routing encourages experts to capture accent-specific patterns, which gradually transitions to label-free routing for inference. Each expert is equipped with its own CTC head to align routing with transcription quality, and a routing-augmented loss further stabilizes optimization. Experiments on the Mcv-Accent
benchmark demonstrate consistent gains across both seen and unseen accents in low- and high-resource conditions, achieving up to 29.3% relative WER reduction over strong FastConformer baselines.
Read the full abstract ▾
Computation and Language
Breaking the Static Graph: Context-Aware Traversal for Robust Retrieval-Augmented Generation
Kwun Hang Lau, Fangyuan Zhang, Boyu Ruan et al. (+4 more)
Recent advances in
Retrieval-Augmented Generation (RAG) have shifted from simple vector similarity to structure-aware approaches like HippoRAG, which leverage Knowledge Graphs (KGs) and Personalized PageRank (PPR) to capture multi-hop dependencies. However, these methods suffer from a "Static Graph Fallacy": they rely on fixed transition probabilities determined during indexing. This
rigidity ignores the query-dependent nature of edge relevance, causing semantic drift where random walks are diverted into high-degree "hub" nodes before reaching critical downstream evidence. Consequently, models often achieve high partial recall but fail to retrieve the complete evidence chain required for multi-hop queries. To address this, we propose CatRAG, Context-Aware Traversal for robust RAG, a framework that builds on the HippoRAG 2 architecture and transforms the static KG into a query-adaptive
navigation structure. We introduce a multi-faceted framework to steer the random walk: (1) Symbolic Anchoring, which injects weak entity constraints to regularize the random walk; (2) Query-Aware Dynamic Edge Weighting, which dynamically modulates graph structure, to prune irrelevant paths while amplifying those aligned with the query's intent; and (3) Key-Fact Passage Weight Enhancement, a cost-efficient bias that structurally anchors the random walk to likely evidence. Experiments across four multi-hop
benchmarks demonstrate that CatRAG consistently outperforms state of the art baselines. Our analysis reveals that while standard Recall metrics show modest gains, CatRAG achieves substantial improvements in
reasoning completeness, the capacity to recover the entire evidence path without gaps. These results reveal that our approach effectively bridges the gap between retrieving partial context and enabling fully grounded
reasoning. Resources are available at https://github.com/kwunhang/CatRAG.
Read the full abstract ▾
All authors Kwun Hang Lau, Fangyuan Zhang, Boyu Ruan, Yingli Zhou, Qintian Guo, Ruiyuan Zhang, Xiaofang Zhou
Computation and Language
From Code-Centric to Concept-Centric: Teaching NLP with LLM-Assisted "Vibe Coding"
Hend Al-Khalifa
The rapid advancement of Large
Language Models (LLMs) presents both challenges and opportunities for Natural
Language Processing (NLP) education. This paper introduces “Vibe Coding,” a pedagogical approach that leverages LLMs as coding assistants while maintaining focus on conceptual understanding and critical thinking. We describe the implementation of this approach in a senior-level undergraduate NLP course, where students completed seven labs using LLMs for code generation while being assessed primarily on conceptual understanding through critical reflection questions. Analysis of end-of-course feedback from 19 students reveals high satisfaction (mean scores 4.4-4.6/5.0) across engagement, conceptual learning, and assessment
fairness. Students particularly valued the reduced cognitive load from debugging, enabling deeper focus on NLP concepts. However, challenges emerged around time constraints, LLM output verification, and the need for clearer task specifications. Our findings suggest that when properly structured with mandatory
prompt logging and reflection-based assessment, LLM-assisted learning can shift focus from syntactic fluency to conceptual mastery, preparing students for an AI-augmented professional landscape.
Read the full abstract ▾
Computation and Language
GuideWeb: A Benchmark for Automatic In-App Guide Generation on Real-World Web UIs
Chengguang Gan, Yoshihiro Tsujii, Yunhao Liang et al. (+3 more)
Digital Adoption Platform (DAP) provide web-based overlays that deliver operation guidance and contextual hints to help users navigate complex websites. Although modern DAP tools enable non-experts to author such guidance, maintaining these guides remains labor-intensive because website layouts and functionalities evolve continuously, which requires repeated manual updates and re-annotation. In this work, we introduce GuideWeb, a new
benchmark for automatic in-app guide generation on real-world web UIs. GuideWeb formulates the task as producing page-level guidance by selecting guide target elements grounded in the webpage and generating concise guide text aligned with user intent. We also propose a comprehensive evaluation suite that jointly measures the accuracy of guide target element selection and the quality of generated intents and guide texts. Experiments show that our proposed GuideWeb
Agent achieves 30.79% accuracy in guide target element prediction, while obtaining BLEU scores of 44.94 for intent generation and 21.34 for guide-text generation. Existing baselines perform substantially worse, which highlights that automatic guide generation remains challenging and that further advances are necessary before such systems can be reliably deployed in real-world settings.
Read the full abstract ▾
All authors Chengguang Gan, Yoshihiro Tsujii, Yunhao Liang, Tatsunori Mori, Shiwen Ni, Hiroki Itoh
Computation and Language
ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support
Tiantian Chen, Jiaqi Lu, Ying Shen et al. (+1 more)
Large
Language Models (LLMs) have shown strong potential as conversational
agents. Yet, their effectiveness remains limited by deficiencies in robust long-term
memory, particularly in complex, long-term web-based services such as online emotional support. However, existing long-term dialogue
benchmarks primarily focus on static and explicit fact
retrieval, failing to evaluate
agents in critical scenarios where user information is dispersed, implicit, and continuously evolving. To address this gap, we introduce ES-MemEval, a comprehensive
benchmark that systematically evaluates five core
memory capabilities: information extraction, temporal
reasoning, conflict detection, abstention, and user modeling, in long-term emotional support settings, covering question answering, summarization, and dialogue generation tasks. To support the
benchmark, we also propose EvoEmo, a multi-session
dataset for personalized long-term emotional support that captures fragmented, implicit user disclosures and evolving user states. Extensive experiments on open-source long-context, commercial, and
retrieval-augmented (RAG) LLMs show that explicit long-term
memory is essential for reducing
hallucinations and enabling effective personalization. At the same time, RAG improves factual consistency but struggles with temporal dynamics and evolving user states. These findings highlight both the potential and limitations of current paradigms and motivate more robust integration of
memory and
retrieval for long-term personalized dialogue systems.
Read the full abstract ▾
All authors Tiantian Chen, Jiaqi Lu, Ying Shen, Lin Zhang
Computation and Language
PretrainRL: Alleviating Factuality Hallucination of Large Language Models at the Beginning
Langming Liu, Kangtao Lv, Haibin Chen et al. (+8 more)
Large
language models (LLMs), despite their powerful capabilities, suffer from factual
hallucinations where they generate verifiable falsehoods. We identify a root of this issue: the imbalanced data distribution in the pretraining corpus, which leads to a state of "low-probability truth" and "high-probability falsehood". Recent approaches, such as teaching models to say "I don't know" or post-hoc knowledge editing, either evade the problem or face catastrophic forgetting. To address this issue from its root, we propose PretrainRL, a novel framework that integrates reinforcement learning into the pretraining phase to consolidate factual knowledge. The core principle of PretrainRL is "debiasing then learning." It actively reshapes the model's probability distribution by down-weighting high-probability falsehoods, thereby making "room" for low-probability truths to be learned effectively. To enable this, we design an efficient negative sampling strategy to discover these high-probability falsehoods and introduce novel metrics to evaluate the model's probabilistic state concerning factual knowledge. Extensive experiments on three public
benchmarks demonstrate that PretrainRL significantly alleviates factual
hallucinations and outperforms state-of-the-art methods.
Read the full abstract ▾
All authors Langming Liu, Kangtao Lv, Haibin Chen, Weidong Zhang, Yejing Wang, Shilei Liu, Xin Tong, Yujin Yuan, Yongwei Wang, Wenbo Su, Bo Zheng
Computation and Language
Read As Human: Compressing Context via Parallelizable Close Reading and Skimming
Jiwei Tang, Shilei Liu, Zhicheng Zhang et al. (+9 more)
Large
Language Models (LLMs) demonstrate exceptional capability across diverse tasks. However, their deployment in long-context scenarios is hindered by two challenges: computational inefficiency and redundant information. We propose RAM (Read As HuMan), a context compression framework that adopts an adaptive hybrid reading strategy, to address these challenges. Inspired by human reading behavior (i.e., close reading important content while skimming less relevant content), RAM partitions the context into segments and encodes them with the input query in parallel. High-relevance segments are fully retained (close reading), while low-relevance ones are query-guided compressed into compact summary vectors (skimming). Both explicit textual segments and implicit summary vectors are concatenated and fed into decoder to achieve both superior performance and natural
language format interpretability. To refine the decision boundary between close reading and skimming, we further introduce a contrastive learning objective based on positive and negative query-segment pairs. Experiments demonstrate that RAM outperforms existing baselines on multiple question answering and summarization
benchmarks across two backbones, while delivering up to a 12x end-to-end speedup on long inputs (average length 16K; maximum length 32K).
Read the full abstract ▾
All authors Jiwei Tang, Shilei Liu, Zhicheng Zhang, Qingsong Lv, Runsong Zhao, Tingwei Lu, Langming Liu, Haibin Chen, Yujin Yuan, Hai-Tao Zheng, Wenbo Su, Bo Zheng
Computation and Language
AXE: Low-Cost Cross-Domain Web Structured Information Extraction
Abdelrahman Mansour, Khaled W. Alshaer and Moataz Elsaban
Extracting structured data from the web is often a trade-off between the brittle nature of manual heuristics and the prohibitive cost of Large
Language Models. We introduce AXE (Adaptive X-Path Extractor), a pipeline that rethinks this process by treating the HTML DOM as a tree that needs
pruning rather than just a wall of text to be read. AXE uses a specialized "
pruning" mechanism to strip away boilerplate and irrelevant nodes, leaving behind a distilled, high-density context that allows a tiny 0.6B LLM to generate precise, structured outputs. To keep the model honest, we implement Grounded XPath Resolution (GXR), ensuring every extraction is physically traceable to a source node. Despite its low footprint, AXE achieves state-of-the-art zero-shot performance, outperforming several much larger, fully-trained alternatives with an F1 score of 88.1% on the SWDE
dataset. By releasing our specialized adaptors, we aim to provide a practical, cost-effective path for large-scale web information extraction. Our code and adaptors are publicly available at https://github.com/abdo-Mansour/axetract.
Read the full abstract ▾
Computation and Language
Sentence Curve Language Models
DongNyeong Heo, Taehwan Kim and Heeyoul Choi
Language models (LMs) are a central component of modern AI systems, and
diffusion language models (DLMs) have recently emerged as a competitive alternative. Both paradigms rely on word
embeddings not only to represent the input sentence, but also to represent the target sentence that backbone models are trained to predict. We argue that such static
embedding of the target word is insensitive to neighboring words, encouraging locally accurate word prediction while global sentence structure is less emphasized. To address this, we propose a continuous sentence representation, termed sentence curve, defined as a spline curve whose control points affect multiple words in the sentence. Based on this representation, we introduce sentence curve
language model (SCLM), which extends DLMs to predict sentence curves instead of the static word
embeddings. We theoretically show that sentence curve prediction induces a regularization effect that promotes global structure modeling, and characterize how different sentence curve types affect this behavior. Empirically, SCLM achieves state-of-the-art performance among DLMs on IWSLT14 and WMT14, shows stable training without burdensome knowledge
distillation, and demonstrates promising potential compared to discrete DLMs on LM1B.
Read the full abstract ▾
Computation and Language
Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understanding
Yuling Shi, Chaoxiang Xie, Zhensu Sun et al. (+7 more)
Large
Language Models (LLMs) have achieved remarkable success in source code understanding, yet as software systems grow in scale, computational efficiency has become a critical bottleneck. Currently, these models rely on a text-based paradigm that treats source code as a linear sequence of
tokens, which leads to a linear increase in context length and associated computational costs. The rapid advancement of
Multimodal LLMs (MLLMs) introduces an opportunity to optimize efficiency by representing source code as rendered images. Unlike text, which is difficult to compress without losing semantic meaning, the image modality is inherently suitable for compression. By adjusting resolution, images can be scaled to a fraction of their original
token cost while remaining recognizable to
vision-capable models. To explore the feasibility of this approach, we conduct the first systematic study on the effectiveness of MLLMs for code understanding. Our experiments reveal that: (1) MLLMs can effectively understand code with substantial
token reduction, achieving up to 8x compression; (2) MLLMs can effectively leverage visual cues such as syntax highlighting, improving code completion performance under 4x compression; and (3) Code-understanding tasks like clone detection exhibit exceptional resilience to visual compression, with some compression ratios even slightly outperforming raw text inputs. Our findings highlight both the potential and current limitations of MLLMs in code understanding, which points out a shift toward image-modality code representation as a pathway to more efficient inference.
Read the full abstract ▾
All authors Yuling Shi, Chaoxiang Xie, Zhensu Sun, Yeheng Chen, Chenxu Zhang, Longfei Yun, Chengcheng Wan, Hongyu Zhang, David Lo, Xiaodong Gu
Computation and Language
Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Model
Kangtao Lv, Jiwei Tang, Langming Liu et al. (+7 more)
The deployment of Large
Language Models (LLMs) in long-context scenarios is hindered by computational inefficiency and significant information redundancy. Although recent advancements have widely adopted context compression to address these challenges, existing research only focus on model-side improvements, the impact of the data distribution itself on context compression remains largely unexplored. To bridge this gap, we are the first to adopt a data-centric perspective to systematically investigate how data distribution impacts compression quality, including two dimensions: input data and intrinsic data (i.e., the model's internal pretrained knowledge). We evaluate the semantic integrity of compressed representations using an autoencoder-based framework to systematically investigate it. Our experimental results reveal that: (1) encoder-measured input
entropy negatively correlates with compression quality, while decoder-measured
entropy shows no significant relationship under a frozen-decoder setting; and (2) the gap between intrinsic data of the encoder and decoder significantly diminishes compression gains, which is hard to mitigate. Based on these findings, we further present practical guidelines to optimize compression gains.
Read the full abstract ▾
All authors Kangtao Lv, Jiwei Tang, Langming Liu, Haibin Chen, Weidong Zhang, Shilei Liu, Yongwei Wang, Yujin Yuan, Wenbo Su, Bo Zheng
Computation and Language
<SOG_k>: One LLM Token for Explicit Graph Structural Understanding
Jingyao Wu, Bin Lu, Zijun Di et al. (+5 more)
Large
language models show great potential in unstructured data understanding, but still face significant challenges with graphs due to their structural
hallucination. Existing approaches mainly either verbalize graphs into natural
language, which leads to excessive
token consumption and scattered
attention, or transform graphs into trainable continuous
embeddings (i.e., soft
prompt), but exhibit severe misalignment with original text
tokens. To solve this problem, we propose to incorporate one special
token <SOG_k> to fully represent the Structure Of Graph within a unified
token space, facilitating explicit
topology input and structural information sharing. Specifically, we propose a
topology-aware structural tokenizer that maps each graph
topology into a highly selective single
token. Afterwards, we construct a set of hybrid structure Question-Answering corpora to align new structural
tokens with existing text
tokens. With this approach, <SOG_k> empowers LLMs to understand, generate, and reason in a concise and accurate manner. Extensive experiments on five graph-level
benchmarks demonstrate the superiority of our method, achieving a performance improvement of 9.9% to 41.4% compared to the baselines while exhibiting interpretability and consistency. Furthermore, our method provides a flexible extension to node-level tasks, enabling both global and local structural understanding. The codebase is publicly available at https://github.com/Jingyao-Wu/SOG.
Read the full abstract ▾
All authors Jingyao Wu, Bin Lu, Zijun Di, Xiaoying Gan, Meng Jin, Luoyi Fu, Xinbing Wang, Chenghu Zhou
Computation and Language · Cross-listed from cs.AI
PRISM: Parametrically Refactoring Inference for Speculative Sampling Draft Models
Xuliang Wang, Yuetao Chen, Maochan Zhen et al. (+5 more)
Large
Language Models (LLMs), constrained by their auto-regressive nature, suffer from slow decoding. Speculative decoding methods have emerged as a promising solution to accelerate LLM decoding, attracting
attention from both systems and AI research communities. Recently, the pursuit of better draft quality has driven a trend toward parametrically larger draft models, which inevitably introduces substantial computational overhead. While existing work attempts to balance the trade-off between prediction accuracy and compute latency, we address this fundamental dilemma through architectural innovation. We propose PRISM, which disaggregates the computation of each predictive step across different parameter sets, refactoring the computational pathways of draft models to successfully decouple model capacity from inference cost. Through extensive experiments, we demonstrate that PRISM outperforms all existing draft architectures, achieving exceptional acceptance lengths while maintaining minimal draft latency for superior end-to-end speedup. We also re-examine scaling laws with PRISM, revealing that PRISM scales more effectively with expanding data volumes than other draft architectures. Through rigorous and fair comparison, we show that PRISM boosts the decoding throughput of an already highly optimized inference engine by more than 2.6x.
Read the full abstract ▾
All authors Xuliang Wang, Yuetao Chen, Maochan Zhen, Fang Liu, Xinzhou Zheng, Xingwu Liu, Hong Xu, Ming Li