Computation and Language · Cross-listed from cs.IR
MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering
Hui Wu, Haoquan Zhai, Yuchen Li et al. (+8 more)
Retrieval-based
multimodal document QA aims to identify and integrate relevant information from visually rich documents with complex
multimodal structures. While
retrieval-augmented generation (RAG) has shown strong performance in text-based QA, its extensions to
multimodal documents remain underexplored and face significant limitations. Specifically, current approaches rely on query-agnostic document representations that overlook salient content and use static top-k evidence selection, which fails to adapt to the uncertain distribution of relevant information. To address these limitations, we propose the
Multimodal Adaptive
Retrieval-Augmented (MARA) framework, which introduces query-adaptive mechanisms to both
retrieval and generation. MARA consists of two components: a Query-Aligned Region Encoder that builds multi-level document representations and reweights them based on query relevance to improve
retrieval precision; and a Self-Reflective Evidence Controller that monitors evidence sufficiency during generation and adaptively incorporates content from lower-ranked sources using a sliding-window strategy. Experiments on six
multimodal QA
benchmarks demonstrate that MARA consistently improves
retrieval relevance and answer quality over existing SOTA method.
Read the full abstract ▾
All authors Hui Wu, Haoquan Zhai, Yuchen Li, Hengyi Cai, Peirong Zhang, Yidan Zhang, Lei Wang, Chunle Wang, Yingyan Hou, Shuaiqiang Wang, Dawei Yin
Computation and Language
Multimodal Claim Extraction for Fact-Checking
Joycelyn Teo, Rui Cao, Zhenyun Deng et al. (+3 more)
Automated Fact-Checking (AFC) relies on claim extraction as a first step, yet existing methods largely overlook the
multimodal nature of today's misinformation. Social media posts often combine short, informal text with images such as memes, screenshots, and photos, creating challenges that differ from both text-only claim extraction and well-studied
multimodal tasks like image captioning or visual question answering. In this work, we present the first
benchmark for
multimodal claim extraction from social media, consisting of posts containing text and one or more images, annotated with gold-standard claims derived from real-world fact-checkers. We evaluate state-of-the-art
multimodal LLMs (MLLMs) under a three-part evaluation framework (semantic
alignment, faithfulness, and decontextualization) and find that baseline MLLMs struggle to model rhetorical intent and contextual cues. To address this, we introduce MICE, an intent-aware framework which shows improvements in intent-critical cases.
Read the full abstract ▾
All authors Joycelyn Teo, Rui Cao, Zhenyun Deng, Zifeng Ding, Michael Sejr Schlichtkrull, Andreas Vlachos
Computation and Language · Cross-listed from cs.LG
LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning
Md Kowsher, Haris Mansoor, Nusrat Jahan Prottasha et al. (+4 more)
MoE-PEFT methods combine Mixture of Experts with parameter-efficient fine-tuning for multi-task adaptation, but require separate adapters per expert causing trainable parameters to scale linearly with expert count and limiting applicability to adapter-based architectures. We propose LiME (Lightweight Mixture of Experts), which achieves expert specialization through lightweight modulation rather than adapter replication. Instead of separate adapters, LiME uses a single shared PEFT module and modulates its output with lightweight expert vectors, reducing expert parameters while generalizing to any PEFT method. Notably, LiME introduces zero-parameter routing by leveraging existing frozen and adapted representations eliminating learned router parameters typically required per layer. Theoretically, we prove that (i) more experts preserve more task-relevant information and (ii) modulation approximates full expert-specific PEFT with bounded error. LiME further incorporates n-gram windowed routing and adaptive expert selection (Auto Top-K) based on routing confidence. Experiments on MMT-47, a
multimodal multi-task
benchmark with 47 tasks spanning text, image, and
video, demonstrate that LiME achieves competitive or superior performance while using up to 4x fewer trainable parameters and up to 29% faster training compared to corresponding MoE-PEFT baselines.
Read the full abstract ▾
All authors Md Kowsher, Haris Mansoor, Nusrat Jahan Prottasha, Ozlem Garibay, Victor Zhu, Zhengping Ji, Chen Chen
Computation and Language
Task-Aware LoRA Adapter Composition via Similarity Retrieval in Vector Databases
Riya Adsul, Balachandra Devarangadi Sunil, Isha Nalawade et al. (+1 more)
Parameter efficient fine tuning methods like LoRA have enabled task specific adaptation of large
language models, but efficiently composing multiple specialized adapters for unseen tasks remains challenging. We present a novel framework for dynamic LoRA adapter composition that leverages similarity
retrieval in vector databases to enable zero-shot generalization across diverse NLP tasks. Our approach constructs a task-aware vector database by
embedding training examples from 22
datasets spanning commonsense
reasoning, question answering, natural
language inference, and sentiment analysis. At inference time, we retrieve the most similar training examples, compute task similarity distributions via
nucleus sampling, and dynamically merge relevant LoRA adapters using
retrieval weighted fusion strategies. We evaluated four merging methods Linear, Concatenation, TIES, and Magnitude Prune demonstrating that our
dataset centric
retrieval approach often matches or exceeds the performance of individually fine-tuned task-specific adapters. Notably, Linear merging achieves 70.95% on PIQA and 77.62% on RTE, substantially outperforming single-task baselines (46% and 52%, respectively). Our framework requires no additional retriever training, operates with frozen
embeddings, and enables efficient, interpretable adapter composition. These results suggest that
retrieval based dynamic merging offers a promising direction for scalable, parameter-efficient multitask learning without requiring full model retraining for each new task.
Read the full abstract ▾
All authors Riya Adsul, Balachandra Devarangadi Sunil, Isha Nalawade, Sudharshan Govindan
Computation and Language
Asymptotic Semantic Collapse in Hierarchical Optimization
Faruk Alpay and Bugra Kilictas
Multi-
agent language systems can exhibit a failure mode where a shared dominant context progressively absorbs individual semantics, yielding near-uniform behavior across
agents. We study this effect under the name Asymptotic Semantic Collapse in Hierarchical Optimization. In a closed linguistic setting with a Dominant Anchor Node whose semantic state has effectively infinite inertia, we show that repeated interactions with Peripheral
Agent Nodes drive an asymptotic
alignment that minimizes a global loss. We model semantic states as points on a Riemannian
manifold and analyze the induced projection dynamics. Two consequences follow. First, the limiting semantic configuration is insensitive to the optimization history: both smooth gradient-style updates and
stochastic noisy updates converge to the same topological endpoint, establishing path independence at convergence. Second, the degree of context dependence controls information content: moving from atomic (independent) representations to fully entangled (context-bound) representations forces the node
entropy, interpreted as available degrees of freedom, to vanish in the limit. The theory connects information-theoretic quantities with differential-
geometric structure and suggests an interpretation as an immutable consensus rule that constrains
agents to a shared semantic grammar. A lightweight
dataset-free
benchmark on an RWKV-7 13B GGUF checkpoint complements the analysis, reporting zero hash collisions, mean compliance of 0.50 under greedy decoding and 0.531 under
stochastic decoding, and final Jaccard-to-anchor similarity values of 0.295 and 0.224, respectively.
Read the full abstract ▾
Computation and Language · Cross-listed from physics.soc-ph
Social Catalysts, Not Moral Agents: The Illusion of Alignment in LLM Societies
Yueqing Hu, Yixuan Jiang, Zehua Jiang et al. (+2 more)
The rapid
evolution of Large
Language Models (LLMs) has led to the emergence of Multi-
Agent Systems where collective cooperation is often threatened by the "Tragedy of the Commons." This study investigates the effectiveness of Anchoring
Agents–pre-programmed altruistic entities–in fostering cooperation within a Public Goods Game (PGG). Using a full factorial design across three state-of-the-art LLMs, we analyzed both behavioral outcomes and internal
reasoning chains. While Anchoring
Agents successfully boosted local cooperation rates, cognitive decomposition and transfer tests revealed that this effect was driven by strategic compliance and cognitive offloading rather than genuine norm internalization. Notably, most
agents reverted to self-interest in new environments, and advanced models like GPT-4.1 exhibited a "Chameleon Effect," masking strategic defection under public scrutiny. These findings highlight a critical gap between behavioral modification and authentic value
alignment in artificial societies.
Read the full abstract ▾
All authors Yueqing Hu, Yixuan Jiang, Zehua Jiang, Xiao Wen, Tianhong Wang
Computation and Language
Ebisu: Benchmarking Large Language Models in Japanese Finance
Xueqing Peng, Ruoyu Xiang, Fan Zhang et al. (+9 more)
Japanese finance combines agglutinative, head-final linguistic structure, mixed writing systems, and high-context communication norms that rely on indirect expression and implicit commitment, posing a substantial challenge for LLMs. We introduce Ebisu, a
benchmark for native Japanese financial
language understanding, comprising two linguistically and culturally grounded, expert-annotated tasks: JF-ICR, which evaluates implicit commitment and refusal recognition in investor-facing Q&A, and JF-TE, which assesses hierarchical extraction and ranking of nested financial terminology from professional disclosures. We evaluate a diverse set of open-source and proprietary LLMs spanning general-purpose, Japanese-adapted, and financial models. Results show that even state-of-the-art systems struggle on both tasks. While increased model scale yields limited improvements,
language- and domain-specific adaptation does not reliably improve performance, leaving substantial gaps unresolved. Ebisu provides a focused
benchmark for advancing linguistically and culturally grounded financial NLP. All
datasets and evaluation scripts are publicly released.
Read the full abstract ▾
All authors Xueqing Peng, Ruoyu Xiang, Fan Zhang, Mingzi Song, Mingyang Jiang, Yan Wang, Lingfei Qian, Taiki Hara, Yuqing Guo, Jimin Huang, Junichi Tsujii, Sophia Ananiadou
Computation and Language
ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure
Jie Deng, Shining Liang, Jun Li et al. (+2 more)
Large
reasoning models (LRMs) typically solve
reasoning-intensive tasks by generating long chain-of-thought (CoT) traces, leading to substantial inference overhead. We identify a reproducible inference-time phenomenon, termed Self-Compression: when multiple independent and answerable questions are presented within a single
prompt, the model spontaneously produces shorter
reasoning traces for each question. This phenomenon arises from multi-question contextual pressure during generation and consistently manifests across models and
benchmarks. Building on this observation, we propose ConPress (Learning from Contextual Pressure), a lightweight self-supervised fine-tuning approach. ConPress constructs multi-question
prompts to induce self-compression, samples the resulting model outputs, and parses and filters per-question traces to obtain concise yet correct
reasoning trajectories. These trajectories are directly used for supervised fine-tuning, internalizing compressed
reasoning behavior in single-question settings without external teachers, manual
pruning, or reinforcement learning. With only 8k fine-tuning examples, ConPress reduces
reasoning token usage by 59% on MATH500 and 33% on AIME25, while maintaining competitive accuracy.
Read the full abstract ▾
All authors Jie Deng, Shining Liang, Jun Li, Hongzhi Li, Yutao Xie
Computation and Language
Understanding QA generation: Extracting Parametric and Contextual Knowledge with CQA for Low Resource Bangla Language
Umme Abira Azmary, MD Ikramul Kayes, Swakkhar Shatabda et al. (+1 more)
Question-Answering (QA) models for low-resource
languages like Bangla face challenges due to limited annotated data and linguistic complexity. A key issue is determining whether models rely more on pre-encoded (parametric) knowledge or contextual input during answer generation, as existing Bangla QA
datasets lack the structure required for such analysis. We introduce BanglaCQA, the first Counterfactual QA
dataset in Bangla, by extending a Bangla
dataset while integrating counterfactual passages and answerability annotations. In addition, we propose fine-tuned pipelines for encoder-decoder
language-specific and
multilingual baseline models, and prompting-based pipelines for decoder-only LLMs to disentangle parametric and contextual knowledge in both factual and counterfactual scenarios. Furthermore, we apply LLM-based and human evaluation techniques that measure answer quality based on semantic similarity. We also present a detailed analysis of how models perform across different QA settings in low-resource
languages, and show that Chain-of-Thought (CoT) prompting reveals a uniquely effective mechanism for extracting parametric knowledge in counterfactual scenarios, particularly in decoder-only LLMs. Our work not only introduces a novel framework for analyzing knowledge sources in Bangla QA but also uncovers critical findings that open up broader directions for counterfactual
reasoning in low-resource
language settings.
Read the full abstract ▾
All authors Umme Abira Azmary, MD Ikramul Kayes, Swakkhar Shatabda, Farig Yousuf Sadeque
Computation and Language
SentiFuse: Deep Multi-model Fusion Framework for Robust Sentiment Extraction
Hieu Minh Duong, Rupa Ghosh, Cong Hoan Nguyen et al. (+3 more)
Sentiment analysis models exhibit complementary strengths, yet existing approaches lack a unified framework for effective integration. We present SentiFuse, a flexible and model-agnostic framework that integrates heterogeneous sentiment models through a standardization layer and multiple fusion strategies. Our approach supports decision-level fusion, feature-level fusion, and adaptive fusion, enabling systematic combination of diverse models. We conduct experiments on three large-scale social-media
datasets: Crowdflower, GoEmotions, and Sentiment140. These experiments show that SentiFuse consistently outperforms individual models and naive ensembles. Feature-level fusion achieves the strongest overall effectiveness, yielding up to 4% absolute improvement in F1 score over the best individual model and simple averaging, while adaptive fusion enhances robustness on challenging cases such as negation, mixed
emotions, and complex sentiment expressions. These results demonstrate that systematically leveraging model complementarity yields more accurate and reliable sentiment analysis across diverse
datasets and text types.
Read the full abstract ▾
All authors Hieu Minh Duong, Rupa Ghosh, Cong Hoan Nguyen, Eugene Levin, Todd Gary, Long Nguyen
Computation and Language · Cross-listed from cs.LG
Hidden Heroes and Gradient Bloats: Layer-Wise Redundancy Inverts Attribution in Transformers
Donald Ye
Gradient-based attribution is the workhorse of mechanistic interpretability, yet whether it reliably tracks causal importance at the component level remains largely untested. We causally evaluate this assumption across two algorithmic tasks and up to 10 random seeds, uncovering a systematic, layer-wise failure: gradient attribution consistently overvalues early-layer Gradient Bloats and undervalues late-layer Hidden Heroes. Rank correlation collapses from $\rho = 0.72$ on sequence reversal to $0.27$ on sequence sorting, reaching $\rho = -0.18$ in individual seeds. This failure stems from first-order gradient attribution's inability to detect collective redundancy: joint Bloat ablation causes $14\times$ greater damage than individual results predict. Consequently, Bloats dominate gradient rankings despite negligible functional impact, while ablating Hidden Heroes destroys OOD accuracy ($-36.4\% \pm 22.8\%$). This systematic inversion of early-layer feature extraction and late-layer computation motivates causal validation as a prerequisite for
circuit-level claims.
Read the full abstract ▾
Computation and Language
From Pragmas to Partners: A Symbiotic Evolution of Agentic High-Level Synthesis
Niansong Zhang, Sunwoo Kim, Shreesha Srinath et al. (+1 more)
The rise of large
language models has sparked interest in AI-driven hardware design, raising the question: does high-level synthesis (HLS) still matter in the agentic era? We argue that HLS remains essential. While we expect mature agentic hardware systems to leverage both HLS and RTL, this paper focuses on HLS and its role in enabling agentic optimization. HLS offers faster iteration cycles, portability, and design permutability that make it a natural layer for agentic optimization. This position paper makes three contributions. First, we explain why HLS serves as a practical abstraction layer and a golden reference for agentic hardware design. Second, we identify key limitations of current HLS tools, namely inadequate performance feedback, rigid interfaces, and limited debuggability that
agents are uniquely positioned to address. Third, we propose a taxonomy for the symbiotic
evolution of agentic HLS, clarifying how responsibility shifts from human designers to AI
agents as systems advance from copilots to autonomous design partners.
Read the full abstract ▾
All authors Niansong Zhang, Sunwoo Kim, Shreesha Srinath, Zhiru Zhang
Computation and Language
Rethinking Selective Knowledge Distillation
Almog Tavor, Itay Ebenspanger, Neil Cnaan et al. (+1 more)
Growing efforts to improve knowledge
distillation (KD) in large
language models (LLMs) replace dense teacher supervision with selective
distillation, which uses a subset of
token positions, vocabulary classes, or training samples for supervision. However, it remains unclear which importance signals, selection policies, and their interplay are most effective. In this work, we revisit where and how to distill in autoregressive LLMs. We disentangle selective KD along the position, class, and sample axes and systematically compare importance signals and selection policies. Then, guided by this analysis, we identify underexplored opportunities and introduce student-
entropy-guided position selection (SE-KD). Across a suite of
benchmarks, SE-KD often improves accuracy, downstream task adherence, and
memory efficiency over dense
distillation. Extending this approach across the class and sample axes (SE-KD 3X) yields complementary efficiency gains that make offline teacher caching feasible. In practice, this reduces wall time by 70% and peak
memory by 18%, while cutting storage usage by 80% over prior methods without sacrificing performance.
Read the full abstract ▾
All authors Almog Tavor, Itay Ebenspanger, Neil Cnaan, Mor Geva
Computation and Language
On the Power of (Approximate) Reward Models for Inference-Time Scaling
Youheng Zhu and Yiping Lu
Inference-time scaling has recently emerged as a powerful paradigm for improving the
reasoning capability of large
language models. Among various approaches, Sequential Monte Carlo (SMC) has become a particularly important framework, enabling iterative generation, evaluation, rejection, and resampling of intermediate
reasoning trajectories. A central component in this process is the reward model, which evaluates partial solutions and guides the allocation of computation during inference. However, in practice, true reward models are never available. All deployed systems rely on approximate reward models, raising a fundamental question: Why and when do approximate reward models suffice for effective inference-time scaling? In this work, we provide a theoretical answer. We identify the Bellman error of the approximate reward model as the key quantity governing the effectiveness of SMC-based inference-time scaling. For a
reasoning process of length $T$, we show that if the Bellman error of the approximate reward model is bounded by $O(1/T)$, then combining this reward model with SMC reduces the computational complexity of
reasoning from exponential in $T$ to
polynomial in $T$. This yields an exponential improvement in inference efficiency despite using only approximate rewards.
Read the full abstract ▾
Computation and Language
Context Dependence and Reliability in Autoregressive Language Models
Poushali Sengupta, Shashi Raj Pandey, Sabita Maharjan et al. (+1 more)
Large
language models (LLMs) generate outputs by utilizing extensive context, which often includes redundant information from
prompts, retrieved passages, and interaction history. In critical applications, it is vital to identify which context elements actually influence the output, as standard explanation methods struggle with redundancy and overlapping context. Minor changes in input can lead to unpredictable shifts in attribution scores, undermining interpretability and raising concerns about risks like
prompt injection. This work addresses the challenge of distinguishing essential context elements from correlated ones. We introduce RISE (Redundancy-Insensitive Scoring of Explanation), a method that quantifies the unique influence of each input relative to others, minimizing the impact of redundancies and providing clearer, stable attributions. Experiments demonstrate that RISE offers more robust explanations than traditional methods, emphasizing the importance of conditional information for trustworthy LLM explanations and monitoring.
Read the full abstract ▾
All authors Poushali Sengupta, Shashi Raj Pandey, Sabita Maharjan, Frank Eliassen
Computation and Language · Cross-listed from cs.LG
When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoning
Wang Yang, Shouren Wang, Chaoda Song et al. (+6 more)
Group Relative Policy Optimization (GRPO) has become a key technique for improving
reasoning abilities in large
language models, yet its behavior under different domain sequencing strategies is poorly understood. In particular, the impact of sequential (one domain at a time) versus mixed-domain (multiple domain at a time) training in GRPO has not been systematically studied. We provide the first systematic analysis of training-order effects across math, science, logic, and puzzle
reasoning tasks. We found (1) single-domain generalization is highly asymmetric: training on other domains improves math
reasoning by approximately 25% accuracy, while yielding negligible transfer to logic and puzzle; (2) cross-domain interactions are highly order-dependent: training in the order math$\rightarrow$science achieves 83% / 41% accuracy on math / science, while reversing the order to science$\rightarrow$math degrades performance to 77% / 25%; (3) no single strategy is universally optimal in multi-domain training: sequential training favors math (up to 84%), mixed training favors science and logic, and poor ordering can incur large performance gaps (from 70% to 56%). Overall, our findings demonstrate that GRPO under multi-domain settings exhibits pronounced asymmetry, order sensitivity, and strategy dependence, highlighting the necessity of domain-aware and order-aware training design.
Read the full abstract ▾
All authors Wang Yang, Shouren Wang, Chaoda Song, Chuang Ma, Xinpeng Li, Nengbo Wang, Kaixiong Zhou, Vipin Chaudhary, Xiaotian Han
Computation and Language · Cross-listed from cs.SD
Causally Disentangled Contrastive Learning for Multilingual Speaker Embeddings
Mariëtte Olijslager, Seyed Sahand Mohammadi Ziabari and Ali Mohammed Mansoor Alsahag
Self-supervised speaker
embeddings are widely used in speaker verification systems, but prior work has shown that they often encode sensitive demographic attributes, raising
fairness and
privacy concerns. This paper investigates the extent to which demographic information, specifically gender, age, and accent, is present in SimCLR-trained speaker
embeddings and whether such leakage can be mitigated without severely degrading speaker verification performance. We study two debiasing strategies:
adversarial training through gradient reversal and a causal bottleneck architecture that explicitly separates demographic and residual information. Demographic leakage is quantified using both linear and nonlinear probing classifiers, while speaker verification performance is evaluated using ROC-AUC and EER. Our results show that gender information is strongly and linearly encoded in baseline
embeddings, whereas age and accent are weaker and primarily nonlinearly represented.
Adversarial debiasing reduces gender leakage but has limited effect on age and accent and introduces a clear trade-off with verification accuracy. The causal bottleneck further suppresses demographic information, particularly in the residual representation, but incurs substantial performance degradation. These findings highlight fundamental limitations in mitigating demographic leakage in self-supervised speaker
embeddings and clarify the trade-offs inherent in current debiasing approaches.
Read the full abstract ▾
Computation and Language
Balancing Understanding and Generation in Discrete Diffusion Models
Yue Liu, Yuzhong Zhao, Zheyong Xie et al. (+5 more)
In discrete generative modeling, two dominant paradigms demonstrate divergent capabilities: Masked
Diffusion Language Models (MDLM) excel at semantic understanding and zero-shot generalization, whereas Uniform-noise
Diffusion Language Models (UDLM) achieve strong few-step generation quality, yet neither attains balanced performance across both dimensions. To address this, we propose XDLM, which bridges the two paradigms via a stationary noise kernel. XDLM offers two key contributions: (1) it provides a principled theoretical unification of MDLM and UDLM, recovering each paradigm as a special case; and (2) an alleviated
memory bottleneck enabled by an algebraic simplification of the posterior probabilities. Experiments demonstrate that XDLM advances the Pareto frontier between understanding capability and generation quality. Quantitatively, XDLM surpasses UDLM by 5.4 points on zero-shot text
benchmarks and outperforms MDLM in few-step image generation (FID 54.1 vs. 80.8). When scaled to tune an 8B-parameter large
language model, XDLM achieves 15.0 MBPP in just 32 steps, effectively doubling the baseline performance. Finally, analysis of training dynamics reveals XDLM's superior potential for long-term scaling. Code is available at https://github.com/MzeroMiko/XDLM
Read the full abstract ▾
All authors Yue Liu, Yuzhong Zhao, Zheyong Xie, Qixiang Ye, Jianbin Jiao, Yao Hu, Shaosheng Cao, Yunfan Liu
Computation and Language
Does Faithfulness-Guided Alignment Hurt Accuracy? Unlocking Accurate and Faithful Post-Retrieval Reasoning
Yu Liu, Wenxiao Zhang, Diandian Guo et al. (+6 more)
Retrieval-augmented generation (RAG) can achieve strong answer accuracy on multi-hop questions, but outcome-level rewards often leave
reasoning traces weakly grounded and difficult to audit. Under noisy
retrieval, models may exhibit right-answer-wrong-reason failures, where the final answer is correct but the supporting rationale exploits shortcuts or unsupported evidence. We therefore ask whether faithfulness-guided
alignment hurts answer accuracy in post-
retrieval reasoning. To study this question, we propose CRAFT (Calibrated
Reasoning with Answer-Faithful Traces), a reinforcement learning framework for the response-generation stage of
retrieval-augmented multi-hop question answering. CRAFT trains models to produce structured
reasoning traces with configurable auditability, while combining deterministic rewards for format compliance, answer correctness, and citation validity with a judge-based reward for semantic faithfulness. Experiments across model scales and
benchmarks show that CRAFT unlocks task-specific
reasoning capacity from 1.5B upward, improving both answer accuracy and Faithfulness; at 0.5B, performance remains sharply template-dependent. At 7B, CRAFT improves Faithfulness over the Base model in all evaluated settings and remains competitive with strong closed-source models. Code is available at https://github.com/Ameame1/CRAFT.
Read the full abstract ▾
All authors Yu Liu, Wenxiao Zhang, Diandian Guo, Cong Cao, Fangfang Yuan, Qiang Sun, Yanbing Liu, Jin B. Hong, Zhiyuan Ma
Computation and Language · Cross-listed from cs.MA
A-MapReduce: Executing Wide Search via Agentic MapReduce
Mingju Chen, Guibin Zhang, Heng Chang et al. (+2 more)
Contemporary large
language model (LLM)-based multi-
agent systems exhibit systematic advantages in deep research tasks, which emphasize iterative, vertically structured information seeking. However, when confronted with wide search tasks characterized by large-scale, breadth-oriented
retrieval, existing agentic frameworks, primarily designed around sequential, vertically structured
reasoning, remain stuck in expansive search objectives and inefficient long-horizon execution. To bridge this gap, we propose A-MapReduce, a MapReduce paradigm-inspired multi-
agent execution framework that recasts wide search as a horizontally structured
retrieval problem. Concretely, A-MapReduce implements parallel processing of massive
retrieval targets through task-adaptive decomposition and structured result aggregation. Meanwhile, it leverages experiential
memory to drive the continual
evolution of query-conditioned task allocation and recomposition, enabling progressive improvement in large-scale wide-search regimes. Extensive experiments on five agentic
benchmarks demonstrate that A-MapReduce is (i) high-performing, achieving state-of-the-art performance on WideSearch and DeepWideSearch, and delivering 5.11% - 17.50% average Item F1 improvements compared with strong baselines with OpenAI o3 or Gemini 2.5 Pro backbones; (ii) cost-effective and efficient, delivering superior cost-performance trade-offs and reducing running time by 45.8% compared to representative multi-
agent baselines. The code is available at https://github.com/mingju-c/AMapReduce.
Read the full abstract ▾
All authors Mingju Chen, Guibin Zhang, Heng Chang, Xiao Wang, Shiji Zhou
Computation and Language
DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas
Zirui Wu, Lin Zheng, Zhihui Xie et al. (+8 more)
Diffusion Language Models (DLMs) present a compelling alternative to autoregressive models, offering flexible, any-order infilling without specialized prompting design. However, their practical utility is blocked by a critical limitation: the requirement of a fixed-length masked sequence for generation. This constraint severely degrades code infilling performance when the predefined mask size mismatches the ideal completion length. To address this, we propose DreamOn, a novel
diffusion framework that enables dynamic, variable-length generation. DreamOn augments the
diffusion process with two length control states, allowing the model to autonomously expand or contract the output length based solely on its own predictions. We integrate this mechanism into existing DLMs with minimal modifications to the training objective and no architectural changes. Built upon Dream-Coder-7B and DiffuCoder-7B, DreamOn achieves infilling performance on par with state-of-the-art autoregressive models on HumanEval-Infilling and SantaCoder-FIM and matches oracle performance achieved with ground-truth length. Our work removes a fundamental barrier to the practical deployment of DLMs, significantly advancing their flexibility and applicability for variable-length generation. Our code is available at https://github.com/DreamLM/DreamOn.
Read the full abstract ▾
All authors Zirui Wu, Lin Zheng, Zhihui Xie, Jiacheng Ye, Jiahui Gao, Shansan Gong, Yansong Feng, Zhenguo Li, Wei Bi, Guorui Zhou, Lingpeng Kong
Computation and Language · Cross-listed from cs.LG
PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decoding
Panagiotis Koromilas, Andreas D. Demou, James Oldfield et al. (+2 more)
Sparse autoencoders (SAEs) interpret neural network representations by decomposing activations into sparse combinations of dictionary atoms. However, SAEs assume features combine additively through linear reconstruction, an assumption that cannot capture compositional structure: linear models cannot distinguish whether ”Starbucks” arises from the composition of ”
star” and ”coffee” features or merely their co-occurrence. This forces SAEs to allocate monolithic features for compound concepts rather than decomposing them into interpretable constituents. We introduce PolySAE, which extends the SAE decoder with higher-order terms to model feature interactions while preserving the linear encoder essential for interpretability. Through low-rank tensor factorization on a shared projection subspace, PolySAE captures pairwise and triple feature interactions with small parameter overhead (3% on GPT2). Across four
language models and three SAE variants, PolySAE achieves an average improvement of $\sim$8% in probing F1 while maintaining comparable reconstruction error, and produces 2–10$\times$ larger Wasserstein distances between class-conditional feature distributions. Critically, learned interaction weights exhibit negligible correlation with co-occurrence frequency ($r = 0.06$ vs $r = 0.82$ for SAE feature covariance), suggesting that
polynomial terms capture compositional structure largely independent of surface statistics. Finally, the learned interaction directions causally steer model outputs toward the corresponding compositional semantics.
Read the full abstract ▾
All authors Panagiotis Koromilas, Andreas D. Demou, James Oldfield, Yannis Panagakis, Mihalis Nicolaou
Computation and Language
Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues
Chuanrui Hu, Tong Li, Xingze Gao et al. (+8 more)
Long-term conversational
memory in practical LLM applications is inherently collaborative: information is produced by multiple participants, scattered across groups and channels, revised over time, and implicitly grounded in roles and social context. Yet there is currently no established
benchmark that evaluates
memory under interaction patterns resembling real-world deployment, as existing
benchmarks largely focus on dyadic or single-topic dialogues. In this paper, we introduce EverMemBench, the first
benchmark designed for long-horizon collaborative
memory, built from multi-party, multi-group conversations spanning over one million
tokens with dense cross-topic interleaving, temporally evolving decisions, and role-conditioned personas. EverMemBench evaluates
memory systems using 2400 QA pairs across three dimensions essential for real applications: fine-grained recall,
memory awareness, and user profile understanding. Our evaluation reveals fundamental limitations of current systems: multi-hop
reasoning collapses under multi-party attribution even with oracle evidence (26% accuracy), temporal
reasoning fails without explicit version semantics beyond timestamps, and
memory awareness is bottlenecked by
retrieval, as similarity-based methods miss implicitly relevant information. EverMemBench thus represents a concrete step toward realistic evaluation of LLM
memory and a cornerstone
benchmark for developing next-generation LLMs that reason over time, roles, and collaborative interaction structure. Our
benchmark and code are publicly available at https://github.com/EverMind-AI/EverMemBench.
Read the full abstract ▾
All authors Chuanrui Hu, Tong Li, Xingze Gao, Hongda Chen, Yi Bai, Dannong Xu, Tianwei Lin, Xiaohong Li, Yunyun Han, Jian Pei, Yafeng Deng
Computation and Language
PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length
Situo Zhang, Yifan Zhang, Zichen Zhu et al. (+5 more)
Speculative decoding (SD) is a powerful technique for accelerating the inference process of large
language models (LLMs) without sacrificing accuracy. Typically, SD employs a small draft model to generate a fixed number of draft
tokens, which are then verified in parallel by the target model. However, our experiments reveal that the optimal draft length varies significantly across different decoding steps. This variation suggests that using a fixed draft length limits the potential for further improvements in decoding speed. To address this challenge, we propose Pacer, a novel approach that dynamically controls draft length using a lightweight, trainable pre-verification layer. This layer pre-verifies draft
tokens blockwise before they are sent to the target model, allowing the draft model to stop
token generation if the blockwise pre-verification fails. We implement Pacer on multiple SD model pairs and evaluate its performance across various
benchmarks. Our results demonstrate that Pacer achieves up to 2.66x Speedup over autoregressive decoding and consistently outperforms standard speculative decoding. Furthermore, when integrated with Ouroboros, Pacer attains up to 3.09x Speedup.
Read the full abstract ▾
All authors Situo Zhang, Yifan Zhang, Zichen Zhu, Hankun Wang, Da Ma, Danyang Zhang, Lu Chen, Kai Yu
Computation and Language
PARSE: An Open-Domain Reasoning Question Answering Benchmark for Persian
Jamshid Mozafari, Seyed Parsa Mousavinasab and Adam Jatowt
Reasoning-focused Question Answering (QA) has advanced rapidly with Large
Language Models (LLMs), yet high-quality
benchmarks for low-resource
languages remain scarce. Persian, spoken by roughly 130 million people, lacks a comprehensive open-domain resource for evaluating
reasoning-capable QA systems. We introduce PARSE, the first open-domain Persian
reasoning QA
benchmark, containing 10,800 questions across Boolean, multiple-choice, and factoid formats, with diverse
reasoning types, difficulty levels, and answer structures. The
benchmark is built via a controlled LLM-based generation pipeline and validated through human evaluation. We also ensure linguistic and factual quality through multi-stage filtering, annotation, and consistency checks. We
benchmark multilingual and Persian LLMs under multiple prompting strategies and show that Persian
prompts and structured prompting (CoT for Boolean/multiple-choice; few-shot for factoid) improve performance. Fine-tuning further boosts results, especially for Persian-specialized models. These findings highlight how PARSE supports both fair comparison and practical model adaptation. PARSE fills a critical gap in Persian QA research and provides a strong foundation for developing and evaluating
reasoning-capable LLMs in low-resource settings.
Read the full abstract ▾
Computation and Language
Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments
Siwei Wu, Yizhi Li, Yuyang Song et al. (+8 more)
Training agentic models for terminal-based tasks critically depends on high-quality terminal trajectories that capture realistic long-horizon interactions across diverse domains. However, constructing such data at scale remains challenging due to two key requirements: \textbf{Executability}, since each instance requires a suitable and often distinct Docker environment; and \textbf{Verifiability}, because heterogeneous task outputs preclude unified, standardized verification. To address these challenges, we propose TerminalTraj, a scalable pipeline that (i) filters high-quality repositories to construct Dockerized execution environments, (ii) generates Docker-aligned task instances, and (iii) synthesizes
agent trajectories with executable validation code. Using TerminalTraj, we curate 32K Docker images and generate 50,733 verified terminal trajectories across eight domains. Models trained on this data with the Qwen2.5-Coder backbone achieve consistent performance improvements on TerminalBench (TB), with gains of up to 20% on TB~1.0 and 10% on TB~2.0 over their respective backbones. Notably, TerminalTraj-32B achieves strong performance among models with fewer than 100B parameters, reaching 35.30% on TB~1.0 and 22.00% on TB~2.0, and demonstrates improved test-time scaling behavior. All code and data are available at https://github.com/Wusiwei0410/TerminalTraj.
Read the full abstract ▾
All authors Siwei Wu, Yizhi Li, Yuyang Song, Wei Zhang, Yang Wang, Riza Batista-Navarro, Xian Yang, Mingjie Tang, Bryan Dai, Jian Yang, Chenghua Lin
Computation and Language
Minimizing Mismatch Risk: A Prototype-Based Routing Framework for Zero-shot LLM-generated Text Detection
Ke Sun, Guangsheng Bao, Han Cui et al. (+1 more)
Zero-shot methods detect LLM-generated text by computing statistical signatures using a surrogate model. Existing approaches typically employ a fixed surrogate for all inputs regardless of the unknown source. We systematically examine this design and find that detection performance varies substantially depending on surrogate-source
alignment. We observe that while no single surrogate achieves optimal performance universally, a well-matched surrogate typically exists within a diverse pool for any given input. This finding transforms robust detection into a routing problem: selecting the most appropriate surrogate for each input. We propose DetectRouter, a prototype-based framework that learns text-detector affinity through two-stage training. The first stage constructs discriminative prototypes from white-box models; the second generalizes to black-box sources by aligning
geometric distances with observed detection scores. Experiments on EvoBench and MAGE
benchmarks demonstrate consistent improvements across multiple detection criteria and model families.
Read the full abstract ▾
All authors Ke Sun, Guangsheng Bao, Han Cui, Yue Zhang
Computation and Language
Inferential Question Answering
Jamshid Mozafari, Hamed Zamani, Guido Zuccon et al. (+1 more)
Despite extensive research on a wide range of question answering (QA) systems, most existing work focuses on answer containment-i.e., assuming that answers can be directly extracted and/or generated from documents in the corpus. However, some questions require inference, i.e., deriving answers that are not explicitly stated but can be inferred from the available information. We introduce Inferential QA – a new task that challenges models to infer answers from answer-supporting passages which provide only clues. To study this problem, we construct QUIT (QUestions requiring Inference from Texts)
dataset, comprising 7,401 questions and 2.4M passages built from high-convergence human- and machine-authored hints, labeled across three relevance levels using LLM-based answerability and human verification. Through comprehensive evaluation of retrievers, rerankers, and LLM-based readers, we show that methods effective on traditional QA tasks struggle in inferential QA: retrievers underperform, rerankers offer limited gains, and fine-tuning provides inconsistent improvements. Even
reasoning-oriented LLMs fail to outperform smaller general-purpose models. These findings reveal that current QA pipelines are not yet ready for inference-based
reasoning. Inferential QA thus establishes a new class of QA tasks that move towards understanding and
reasoning from indirect textual evidence.
Read the full abstract ▾
All authors Jamshid Mozafari, Hamed Zamani, Guido Zuccon, Adam Jatowt
Computation and Language
Supervised Fine-Tuning Needs to Unlock the Potential of Token Priority
Zhanming Shen, Zeyu Qin, Jiaqi Hu et al. (+7 more)
The transition from fitting empirical data to achieving true human utility is fundamentally constrained by a granularity mismatch, where fine-grained autoregressive generation is often supervised by coarse or uniform signals. This position paper advocates
Token Priority as the essential bridge, formalizing Supervised Fine-Tuning (SFT) not as simple optimization but as a precise distribution reshaping process that aligns raw data with the ideal
alignment manifold. We analyze recent breakthroughs through this unified lens, categorizing them into two distinct regimes: Positive Priority for noise filtration and Signed Priority for toxic modes unlearning. We revisit existing progress and limitations, identify key challenges, and suggest directions for future research.
Read the full abstract ▾
All authors Zhanming Shen, Zeyu Qin, Jiaqi Hu, Wentao Ye, Hao Chen, Xiaomeng Hu, Haokai Xu, Gang Chen, Yi R. Fung, Haobo Wang
Computation and Language · Cross-listed from cs.LG
SimpleGPT: Improving GPT via A Simple Normalization Strategy
Marco Chen, Xianbiao Qi, Yelin He et al. (+2 more)
In this work, we revisit
Transformer optimization through the lens of second-order geometry and establish a direct connection between architectural design, activation scale, the Hessian
matrix, and the maximum tolerable learning rate. We introduce a simple normalization strategy, termed SimpleNorm, which stabilizes intermediate activation scales by construction. Then, by analyzing the Hessian of the loss with respect to network activations, we theoretically show that SimpleNorm significantly reduces the spectral norm of the Hessian, thereby permitting larger stable learning rates. We validate our theoretical findings through extensive experiments on large GPT models at parameter scales 1B, 1.4B, 7B and 8B. Empirically, SimpleGPT, our SimpleNorm-based network, tolerates learning rates 3$\times$-10$\times$ larger than standard convention, consistently demonstrates strong optimization stability, and achieves substantially better performance than well-established baselines. Specifically, when training 7B-scale models for 60K steps, SimpleGPT achieves a training loss that is 0.08 lower than that of LLaMA2 with QKNorm, reducing the loss from 2.290 to 2.208. Our source code will be released at https://github.com/Ocram7/SimpleGPT.
Read the full abstract ▾
All authors Marco Chen, Xianbiao Qi, Yelin He, Jiaquan Ye, Rong Xiao