Computation and Language
Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Head
Zizhuo Fu, Runsheng Wang and Meng Li
Scaling
attention parameters can improve
language model quality, but retaining full
token histories makes additional heads costly at long contexts. Furthermore, since
attention retrieves and combines contextual information, parameter scaling should also support longer contexts. We therefore ask whether
attention parameter scaling can directly enable efficient and effective context scaling. We introduce NAMOH, an architecture-native sparse
attention mechanism that activates $K$ of $H$ heads per
token. Each head retains only its assigned
tokens and performs causal
attention within this subsequence. Head selection thus jointly determines active parameters and available context without scanning the full history. Under balanced assignments, increasing $H$ at fixed $K$ shortens head histories and reduces per-
token key-value (KV) access without increasing total KV storage. We further support head-relative rotary position
embeddings to shorten positional spans within routed subsequences, aiming to mitigate position-induced
attention noise. Experiments show that NAMOH can outperform fully activated models with the same total parameters, while enabling more efficient long-context inference than smaller dense models with matched active parameter counts. It remains compatible with GQA and existing sparse
attention mechanisms. We hope this work offers a new path for scaling
attention, with parameter scaling directly enabling context scaling.
Read the full abstract ▾
Computation and Language
Forging LLM Authorship Fingerprints with Targeted Rewriting
Haohan Yuan, Simin Chen, Xi Niu et al. (+3 more)
Model-attribution classifiers can often identify which
language model produced a text, making model-specific writing patterns a signal of provenance. Accurate attribution on unmodified text, however, does not show whether the prediction still identifies the original source after deliberate rewriting. We formulate this problem as targeted fingerprint transfer: rewriting one model's output so that attribution classifiers assign it to a chosen target model. We study summarization, where different models receive the same document and express the same underlying content, providing a controlled setting for conditional generation. We introduce ForgePrint, a search-then-distil framework that first searches for rewrites that move attribution toward a target fingerprint, then distils the selected rewrites into a one-pass 4B Student model. On CNN/DM, the Student reaches 70.2% target success rate, outperforming both its Teacher (54.1%) and the strongest of six published rewriting baselines (39.3%), against held-out classifiers that are never queried by the attack. It also reaches 68.3% target success when transferring summaries from an open model toward chosen commercial models. These results show that fingerprint detectability should not be conflated with source authenticity, and that text-only attribution can provide misleading evidence of model identity under targeted rewriting, even when it is accurate on unmodified text.
Read the full abstract ▾
All authors Haohan Yuan, Simin Chen, Xi Niu, Hanqing Guo, Depeng Xu, Haopeng Zhang
Machine Learning
SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUs
Fahao Chen, Linkang Du, Jinhao Zhou et al. (+2 more)
Sparse
attention is widely used to accelerate long-context inference in modern large
language models (LLMs), but its input-dependent execution behavior introduces previously unexplored
privacy risks. We identify a new GPU micro-architectural side channel, termed Sparsity-Induced
Memory Access (SIMA), which arises from secret-dependent key-value cache access patterns induced by sparse
attention. Based on this observation, we present SparLeak, a phase-aware side-channel attack that extracts SIMA traces during LLM inference and enables two practical
privacy extractions: query attribute inference from prefill-phase traces and autoregressive response reconstruction from decoding-phase traces. By reconstructing approximate
token-level sparsity profiles from page-level observations and applying profiling-based learning, SparLeak accurately recovers sensitive information, including user-query attributes and private LLM response content. Extensive evaluation across three LLM architectures, three sparse
attention mechanisms, and three
privacy-sensitive
datasets shows that SparLeak achieves average attack success rates of 90.9% for attribute inference and 87.3% for response reconstruction under real-world LLM serving settings, highlighting the significance to account for SIMA leakage when deploying sparse-
attention-based LLM systems. We provide anonymized SIMA traces, trained attack models, evaluation scripts, and documentation as artifacts at https://anonymous.4open.science/r/Janus_artifacts/.
Read the full abstract ▾
All authors Fahao Chen, Linkang Du, Jinhao Zhou, Peng Li, Zhou Su
Artificial Intelligence
Diversity Combining for Multi-Path LLM Reasoning
Guangsheng Yu, Litianyi Zhang, Qin Wang et al. (+5 more)
Multi-path
reasoning methods such as self-consistency (SC) sample $K$
reasoning paths and choose the most frequent answer. However, their gains quickly plateau as $K$ increases, and locating this saturation point currently requires fitting a scaling model to accuracies measured at several values of $K$ or stopping online per query. We formalize multi-path LLM
reasoning as a diversity combining problem from wireless communications: each path is a noisy channel observation, and the pairwise correlation of path correctness caps the design-effect effective sample size of the vote at a finite ceiling. Generalized least squares (GLS) analysis shows that, for paths with equal gains, the optimal linear combiner of latent
embeddings is uniform exactly when every row of the path covariance has the same sum, as under exchangeability, supporting majority vote as the natural default in standard SC while leaving room for weighting or
pruning under heterogeneous
prompt-template branches. Across 5 models and 12
benchmarks,
prompt-template diversity reduces path correlation in $55$ of $60$ model-
benchmark cells, with a mean relative reduction of $16\%$ and reductions in every task domain. We derive an Adaptive-K rule that uses a four-path pilot to select $K^*$, retaining $96$–$103\%$ of MV@$K{=}32$ accuracy across Math, QA, and NLU.
Read the full abstract ▾
All authors Guangsheng Yu, Litianyi Zhang, Qin Wang, Xu Wang, Mingyuan Li, Shaoxiong Ji, Ren Ping Liu, Massimo Piccardi
Artificial Intelligence
More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models
Tianxiang Gao, Jinzhe Li, Zhiyuan Li et al. (+2 more)
Direct-decision models enable efficient classification and evaluation by selecting from predefined candidates, but their reliability depends on more than prediction accuracy. Across 40
datasets and nine models from the JEV, KEV, and StartLux Decision families, we identify a systematic tendency to underuse ordinal rating scales. We call this phenomenon ordinal scale-utilization bias: final decisions concentrate on fewer effective labels than the gold distribution. The bias is substantially weaker on tasks with unordered semantic categories. Evaluating the same items on finer ordinal scales strengthens compression despite controls for gold imbalance and candidate position. Randomizing candidate order reduces but does not eliminate compression, while widening ordinal gaps at fixed candidate count can improve accuracy without restoring utilization. For most models, concentrated decisions coexist with broad average candidate probabilities. Greater model size improves accuracy without consistently restoring scale utilization. Together, these findings characterize compression as strongly associated with ordinal structure and intensified by scale refinement, rather than fully explained by presentation position or local spacing. This bias can be effectively mitigated through targeted training, making scale utilization an actionable reliability target alongside accuracy. The code and data are available at https://anonymous.4open.science/r/jev_ordinal_scale_bia/.
Read the full abstract ▾
All authors Tianxiang Gao, Jinzhe Li, Zhiyuan Li, Yi Chang, Yuan Wu
Robotics
PhaseSync-Exo: Human Clock Anchored Reference Adaptation for Dynamic Gait Tracking
Kaijie Qi, Yuehan Wang, Kaiming Xu et al. (+2 more)
Human-aware exoskeleton walking requires reconstructing gait, tracking diverse motions under dynamic constraints, and preserving human timing. We present PhaseSync-Exo, which combines two-IMU CNN-
Transformer reconstruction, factorized amplitude-cadence retargeting with curriculum-trained recurrent control, and a human-clock-anchored adapter (HCA). HCA combines human-clock attraction with
robot-relative feedback to adjust reference rate while preserving forward progression and continuity. The reconstruction module achieves a mean absolute error (MAE) of 3.24 degrees on a held-out recording, and the frozen tracking policy completes 356 of 357 amplitude-frequency trials. Two complementary dynamic comparisons isolate HCA's timing benefit without retraining. Against
robot-relative correction, HCA reduces human-clock phase MAE from 95.55 degrees to 10.99 degrees, limiting reference drift. When human and delivered phases initially differ, it reduces phase MAE from 72.02 degrees to 13.90 degrees versus fixed-clock continuation. Compared with immediate phase reset, HCA reduces transient reference-tracking hip RMSE by 22.7% without reference jumps. All 420 timing rollouts complete without falls. These simulation results support continuous phase acquisition and sustained
alignment to an independent human clock.
Read the full abstract ▾
All authors Kaijie Qi, Yuehan Wang, Kaiming Xu, Chong Li, Jiakuo Yu
Computer Vision
DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference
Xudong Tan, Peng Ye, Ming Xie et al. (+4 more)
Multimodal mixture-of-experts (MoE) models combine sparse expert activation with visual-
language capabilities, yet their inference remains costly because long visual-
token sequences repeatedly incur
attention, routing, dispatch, and expert-MLP computation. Existing methods typically compress either the
token or expert dimension, leaving redundancy along the other. Our analysis reveals two complementary regularities: the depth required for visual propagation varies across inputs, while text-
token routing exhibits concentrated and recurrent expert-importance patterns. Based on these observations, we propose DecoMoE, a two-dimensional structured compression framework that decouples visual propagation from expert computation. The Sample-Adaptive Visual Boundary (SAVB) predicts an input-dependent visual-exit layer at which the visual-
token block is removed. The Routing-Calibrated Expert Prefix (RCEP) reorders experts offline using text-
token routed mass and, from this predicted exit layer onward, retains at each MoE layer the shortest contiguous prefix covering a target routed-mass fraction. We evaluate DecoMoE on Qwen3-VL-MoE and InternVL3.5-30B-A3B across six
benchmarks. On Qwen3-VL-MoE, DecoMoE retains 97.91% of dense-baseline performance while reducing computation from 27.06 to 16.73 TFLOPs and latency from 0.44 to 0.26 seconds, yielding a 1.69x speedup. Code will be available at https://github.com/ShawnTan86/DecoMoE.
Read the full abstract ▾
All authors Xudong Tan, Peng Ye, Ming Xie, Chenyu Huang, Yaoxin Yang, Jiayuan Fan, Tao Chen
Information Retrieval
SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale
Guanqun Yang, Wenlong Zhang, Tian Shi et al. (+1 more)
Anthropic's
Agent Skills package reusable procedural know-how for an LLM
agent into SKILL.md directories, and open-source aggregations have grown past 230,000 skills, making selection rather than authoring the bottleneck. The standing answer in the literature outsources selection to the
agent itself: an LLM-mediated
retrieval loop that rewrites queries and refines candidates inside the
agent's decision loop, paying LLM
tokens on every task. We present SkillSeek, an open-source two-stage skill retriever built from the standard IR recipe (a BGE-base bi-encoder feeding a small cross-encoder, exposed over MCP). Across a $4 \times 11$ grid of pool, backbone, and method on the 89-task SkillsBench
benchmark, SkillSeek reaches observed parity with the LLM-mediated loop of Liu et al. at essentially no extra cost: plain bm25 alone records a pass rate at or above their refined loop on three of four settings, and a small cross-encoder covers the remaining difference on the fourth. A first-stage recall ceiling explains the pattern, and total per-trial spend drops from USD 51.30 to USD 27.54 (within fifty cents of the no-skill baseline). Under the SkillsBench tasks and OpenHands harness we tested, this positions the standard IR recipe as a strong default for
agent-skill
retrieval, with LLM-mediated alternatives a natural fit for cases where deterministic methods fall short.
Read the full abstract ▾
All authors Guanqun Yang, Wenlong Zhang, Tian Shi, Ping Wang
Computation and Language
BARRAC: Adaptation of an English Aspect-based Sentiment Analysis Approach for Classification Tasks in Arabic Dialects
Ali Almutairi, Gelareh Mohammadi, Imran Razzak et al. (+1 more)
With the rapid growth of Arabic NLP, several models,
datasets and
benchmarks have been reported. This paper asks whether approaches developed for majority
languages like English can be adapted to Arabic tasks. We adapt an English aspect-based sentiment analysis framework to Arabic classification tasks and present the adaptation as BARRAC: Brainstorming
Alignment and Replaced Representation learning for ArabiC tasks. BARRAC replaces consumer-review attribute pools with Arabic linguistic devices and markers for dialectal sentiment, sarcasm, and dialect identification, and replaces noisy self-training with two-stage training. Evaluated on five Arabic dialect
datasets, BARRAC achieves a mean macro-F1 of 63.93%, outperforming the best few-label SOTA by 3%, and outperforming GPT-4o on four out of five tasks. Error analysis provides insights into remaining challenges. These results demonstrate that adapting task-specific approaches is a promising direction for Arabic NLP alongside adapting models,
datasets and
benchmarks.
Read the full abstract ▾
All authors Ali Almutairi, Gelareh Mohammadi, Imran Razzak, Aditya Joshi
Computer Vision
Future Video Generation Better Aligns with the Human Visual Cortex than Observed Video
Chang-Bae Bang, Hyungjin Chung and Byung-Hoon Kim
Studying the
alignment between the internal representations of
vision models and the responses of the visual cortex to the same observed visual stimuli has enabled us to better understand human visual processing. However, studies so far have largely overlooked the fact that the human
brain not only processes observed visual stimuli, but also predicts upcoming stimuli based on what has been observed. Accordingly, we hypothesize that internal representations for generating future
video frames are better aligned with the predictive nature of human visual processing than representations of the observed
video itself. To this end, we compare the
alignment between human
video-watching fMRI responses in the visual cortex and the internal representations from two types of
video diffusion models, an autoregressive (AR) model and its non-AR base model. We first conduct a within-model analysis of the AR
video diffusion model and show that the representations for future
video generation align better with the visual cortex than the representations of the observed
video. We then compare the internal representations of the AR model with those of its non-AR base model and again show that the representations for future
video generation align better with the visual cortex than the representations for observed
video reconstruction by the base model. Specifically, the
alignment of observed
video reconstruction is concentrated in lower-order visual cortex, whereas that of future
video generation is concentrated in higher-order visual cortex. Finally, we show in a human behavioral experiment that humans prefer
videos generated by amplifying the contributions of individual layers that align better with the visual cortex.
Read the full abstract ▾
Artificial Intelligence
Whose Voice Survives the Summary? A Voice-Retention Audit of LLM Employee Listening
Thilo Tamme, Anton Hantel and Bijan Khosrawi-Rad
Organizations increasingly route employee feedback to leaders through large
language model (LLM) summaries, an unaudited layer that silences already-spoken voice. We introduce a Voice Retention / Representation Ratio metric for representational bias in summarization and apply it to a bilingual (English/German) corpus of 2,586 free-text responses from a global professional service company. First, employees supply criticism more reliably than praise (withholding praise is 82 times more common). Second, across 45 leader-summaries the pipeline filters by popularity, not sentiment: criticism survives, yet a concern voiced once is dropped 86% of the time, with short and German-only content lost on the same axis (theme retention 0.14 vs 0.74; German directional). Controlling for frequency, sentiment has no independent effect; the harm is prevalence-driven, which sentiment-only audits miss. A targeted
prompt recovers only named themes. We contribute the metric, field evidence, and a disaggregated voice-retention card.
Read the full abstract ▾
Artificial Intelligence
Yuanhe Zhang, Ziwei Wang, Jie Ren et al. (+6 more)
Large
reasoning models (LRMs) improve performance on complex tasks through extended
reasoning, yet the same process can degenerate into redundant verification and persistent generation loops. Such uncontrolled
reasoning increases inference cost and creates risks of resource exhaustion and service degradation. However, existing mitigations largely truncate long outputs or react to surface repetition, and thus fail to distinguish normal thinking from uncontrolled
reasoning or explain how benign
reasoning degenerates into harmful behavior. In this paper, we operationalize LRM generation as four states and further introduce
Reasoning-state Analysis via Dynamic
Attention Responses (RADAR), which identifies the current
reasoning state in real time and characterizes how effective reflection can develop into uncontrolled generation. Guided by RADAR's analysis, we further realign abnormal
attention distributions toward patterns observed in normal requests and examine how this correction affects excessive reflection and persistent looping. Temporal analyses show that uncontrolled
reasoning is characterized by
attention distributions that deviate from normal generation, with abnormal trends becoming detectable before repetition begins. Correcting these deviations through
Attention Realignment consistently reduces looping while largely preserving benign performance. Together, RADAR provide a mechanistic account of how
reasoning becomes uncontrolled, offering actionable guidance for identifying critical failure stages and designing targeted runtime interventions.
Read the full abstract ▾
All authors Yuanhe Zhang, Ziwei Wang, Jie Ren, Haoran Gao, Zhenhong Zhou, Fanyu Meng, Cong Wu, Li Sun, Sen Su
Computation and Language
You're Hired: Strategic Model Selection for LLM Collaboration
Zongwan Cao, Ziyuan Yang, Shangbin Feng et al. (+5 more)
While multi-
agent and model collaboration
algorithms gain traction to combine the strengths of diverse Large
Language Models (LLMs), existing systems remain bottlenecked on pre-defined and hand-crafted model pools. In this work, we investigate the problem of model selection in multi-LLM systems. We propose and systematically evaluate a taxonomy of 9 selection
algorithms ranging from diversity of model descriptions, capability-aware behavioral diversity, and LLM-based recruiters. We conduct extensive experiments across two candidate pools of 10 and 32 models, deployed in four model collaboration
algorithms, and evaluated across tasks spanning math, coding, QA, and
reasoning. Results demonstrate that successful selection
algorithms greatly outperform random or heuristics-based teams such as merely selecting the models with top individual performance, by up to 36.1% across settings. Specifically, capability- and training-based selection strategies alleviate selection variance and achieve the best performance, which we recommend to employ before deploying real-world multi-LLM systems. Further analysis reveals that larger candidate pools pose greater challenges to shallow selection heuristics, while
algorithms grounded in interacting with candidate models and understanding model capability robustly filter out misaligned, unsafe models, as well as generalizing to novel, out-of-distribution tasks. Together, we establish that principled and informed team selection is critical and present strong model selection
algorithms for assembling effective multi-LLM systems.
Read the full abstract ▾
All authors Zongwan Cao, Ziyuan Yang, Shangbin Feng, Michael Duan, Skyler Hallinan, Bingbing Wen, Lucy Lu Wang, Yulia Tsvetkov
Machine Learning
Learning Chaos Without Seeing Chaos: Extrapolation of Global Dynamics in Autoregressive Transformers
Yilun Liu, Yi Zhang, Ganyu Wu et al. (+5 more)
Autoregressive models are trained to predict a system's behavior one step at a time, and recursive generation allows the learned dynamics to unfold over long horizons. To what extent can such dynamics learned from local observations recover broader organization of an underlying system that was only partially observed during training? Here we study small autoregressive
transformers trained from scratch on trajectories sampled from restricted parameter regimes of several non-linear dynamical systems, including logistic and sine maps, the Lorenz system, and the generalized
Hopf system, with control parameters and state trajectories represented as sequences of continuous
tokens. Under closed-loop evaluation at parameters far outside the training distribution, the models can recover self-similar period-doubling cascades, chaotic dynamics, and attractor structures with remarkable visual and numerical fidelity. For the logistic map, a
transformer reproduces successive period doublings up to period 128, yielding a finite-order scaling ratio of 4.6687, matching the Feigenbaum constant to within $5\times10^{-4}$. We further investigate how these structures emerge over the course of training, and reveal with causal interventions how control-parameter information is processed through
attention into state prediction and shapes the resulting closed-loop dynamics. These results suggest that a surprisingly narrow window into a system's local behavior may suffice for autoregressive
transformers to generalize to its unseen global dynamical organization.
Read the full abstract ▾
All authors Yilun Liu, Yi Zhang, Ganyu Wu, Sikuan Yan, Mengyue Wang, Alois Knoll, Volker Tresp, Yunpu Ma
Computation and Language
Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification
Yingfeng Luo, Shaowei Wei, Daixin Wang et al. (+7 more)
Terminal
agents rely on self-verification to assess and correct their solutions as they solve tasks through interaction with command-line environments. Yet how trustworthy such self-verification is remains poorly understood. To investigate this question systematically, we introduce a diagnostic framework that identifies the first complete solution in each trajectory, determines whether it is objectively correct, and uses this ground truth to quantify the
agent's subsequent verification and recovery behavior. Applying it to ten terminal
agents on TerminalBench2.1, we find that verification is nearly universal after a complete candidate is formed, yet only 61.43% of incorrect candidates are detected and only 49.36% of detected errors are successfully repaired. These results show that the main weakness in self-verification lies not in initiating verification, but in detecting and repairing errors. Motivated by these findings, we propose Student-Conditioned Verification
Distillation (SCVD), which lets the student first produce a candidate solution and distills a stronger teacher's subsequent verification and recovery from the same interaction context. Across three Qwen3.5 backbones, SCVD improves Pass@1 on TerminalBench2.1 by 9.74–16.85 percentage points over the corresponding base models and by 4.49–8.61 points over the standard full-trajectory
distillation, while avoiding the pronounced out-of-distribution degradation of full-trajectory
distillation on SWE-bench Verified.
Read the full abstract ▾
All authors Yingfeng Luo, Shaowei Wei, Daixin Wang, Dingyang Lin, Kaiyan Chang, Weiqiao Shan, Tong Zheng, Zhiqiang Zhang, Jingbo Zhu, Tong Xiao
Computer Vision
DCM-SAM: Defect-Conditioned Mixture of LoRA Experts for NPU-Deployed AM Defect Segmentation
Md Mushfiqur Rahaman, Md Mahedi Hasan, Imtiaz Ahmed et al. (+1 more)
Metal additive manufacturing parts are inspected by X-ray computed
tomography, where labelled data is scarce, the pores and inclusions that matter span a few pixels, and inspection must happen at the machine. We present DCM-SAM, a defect-conditioned adaptive mixture of LoRA experts: one frozen Segment Anything backbone carries a separate Conv-LoRA expert bank and mask decoder per defect class, each trained in its own pass, without
prompts, on synthetic slices alone, updating only 4.4% of the parameters. On
benchmarks that XCT-SAM reports, DCM-SAM improves on every baseline for both classes from a ViT-B backbone against their ViT-H, and reaches 64.2% pore IoU on real NIST scans having seen no real images during training. Deployment then exposes what adaptation work rarely measures: on a Qualcomm Hexagon NPU, ViT-H and ViT-L compile yet cannot allocate at 1024x1024 image resolution, since activations rather than weights exceed the device ceiling, and quantizing weights does not help. ViT-B alone runs, but the adapted encoder then fails to allocate where the stock one succeeds, until a numerically identical rewrite of the
attention lets the complete DCM-SAM run in FP16 at 1024x1024, with no operator falling back to the CPU, masks within 0.01% of pixels of the FP32 reference. Code: https://github.com/MushfiqShovon/DCM-SAM.
Read the full abstract ▾
All authors Md Mushfiqur Rahaman, Md Mahedi Hasan, Imtiaz Ahmed, Srinjoy Das
Computer Vision
CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models
Chunzheng Zhu, Jiaqi Zeng, Hongbo Zhao et al. (+3 more)
As
vision language models are increasingly deployed in clinical diagnosis, under standing how they internally resolve competing visual and textual signals becomes a safety imperative. Existing mechanistic analyses remain confined to unimodal text and offer no explanation for why a single misleading sentence can override a correct image based diagnosis, or why a model commits to a confident answer despite insufficient visual evidence. We find that these two safety risks, arbitra tion failure where textual context overrides visual grounding and brake failure where the model commits without adequate evidence, are mediated by spatially disjoint
attention head populations: arbitration heads form a mid-to-deep wideband reflecting cross-layer evidence competition, while brake heads concentrate in a narrow middle-to-late layer band that regulates evidence sufficiency and abstention behavior. To ground these observations in causal circuitry, we introduce CRAFT, which localizes each failure mode to a minimal causal head set via dual criteria and verifies necessity and sufficiency through temporal probes and Tuned Lens trajectory analysis. Excising arbitration heads sharply reduces conflict following with negligible degradation on clean inputs, while excising brake heads restores ap propriate abstention under degraded visual evidence. The two interventions target spatially disjoint head sets and produce distinct corrective effects, underscoring the mechanistic separability of the failure modes. Experiments across multiple medical VQA
benchmarks and VLM architectures validate both the localization and inter ventions, demonstrating that the identified heads causally drive each failure mode and that targeted modulation generalises without retraining. The code is available at https://github.com/zhcz328/CRAFT.
Read the full abstract ▾
All authors Chunzheng Zhu, Jiaqi Zeng, Hongbo Zhao, Yihang Chen, Yijun Wang, Jianxin Lin
Computation and Language
StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learning
Naen Xu, Wanqing Cui, Yibo Hu et al. (+5 more)
Large
language models deployed as personalized assistants must reason over long, evolving interaction histories. However, in long-term dialogue
reasoning, relevant evidence is scattered across sessions, preferences may be revised over time, and standard long-context training fails to address these challenges under data scarcity and prohibitive computational costs. We propose StateTree, a data-driven RL method that constructs a challenging auxiliary task from scarce dialogues with verifiable ground truth. StateTree augments multi-session dialogues with a tree-structured path-tracing task: key-value records are embedded across sessions to form a binary tree. Solving the task requires the model to traverse from root to leaf by retrieving records across sessions and comparing timestamps to resolve branches, then recover the hidden target question among distractor leaves. We apply curriculum RL training progressively increasing tree depth and introduce a compositional variant whose edges carry step-level
reasoning fragments, training the model to compose partial cues into coherent queries. Trained on 10K-
token contexts, StateTree generalizes to 128K
tokens without full-length RL costs and exhibits capabilities including cross-session
retrieval, temporal
reasoning, knowledge update, and compositional multi-hop
reasoning. StateTree outperforms both SFT and RL-based baselines while preserving short-context general
reasoning. StateTree-7B achieves gains up to +23.60% on LongMemEval (128k), and StateTree-14B reaches 59.00% accuracy on LongMemEval, surpassing QwenLong-L1-32B (45.20%).
Read the full abstract ▾
All authors Naen Xu, Wanqing Cui, Yibo Hu, Shixin Hong, Hengyu An, Meiguang Jin, Junfeng Ma, Tianyu Du
Numerical Analysis · Cross-listed from math.OC
Halpern-Accelerated Majorized ADMM with Optimal Non-Ergodic Convergence under Degenerate Preconditioning
Ning Zhang, Benqi Liu and Liwei Zhang
We develop a Halpern-accelerated majorized alternating direction method of multipliers (ADMM) with possibly indefinite proximal terms for linearly constrained convex composite optimization. Majorization simplifies the subproblems but introduces a forward perturbation to the standard degenerate proximal point representation. We reformulate majorized ADMM as a perturbed degenerate proximal point method and characterize the fixed points of the induced mapping as Karush–Kuhn–Tucker (KKT) solutions. Under range compatibility and metric cocoercivity conditions, the mapping is nonexpansive in the seminorm induced by a possibly singular preconditioner. In finite dimensions, a verifiable block-
matrix condition permits indefinite proximal regularization while retaining the required metric properties. A relaxed Halpern iteration then yields an $\mathcal O(1/k)$ fixed-point residual bound, sharp in the general nonexpansive setting, and an $\mathcal O(1/k)$ nonergodic KKT residual bound at the intermediate iterates. Numerical experiments illustrate the predicted residual decay and the benefits of majorization and indefinite proximal terms.
Read the full abstract ▾
Information Retrieval
PatchHolmes: Agentic Patch Retrieval via Listwise Selection
Guanqun Yang, Yingming Zhou, Jiangrui Zheng et al. (+2 more)
Patch
retrieval, the task of finding the commit that fixes a known vulnerability, is the foundation of vulnerability management workflows, yet 60% to 63% of CVEs in the major advisory databases lack a patch link. We present PatchHolmes, a two-phase patch
retrieval system that pairs a hybrid first-stage retriever with an agentic second-stage inspection loop. Unlike pointwise prior work that scores each candidate independently, the Phase 2
agent reads the top-100 listwise: it sees the full candidate list at once and selectively reads 3 to 10 commits through four budgeted tools before submitting a single best commit. On GitHubAD, PatchHolmes beats the pointwise binary classifier Favia by 25.34% Recall@1 and the retrieve-and-CoT baseline IRCoT by 31.40%, at one
agent conversation per CVE versus Favia's ten; with the candidate set held identical, the
agent adds 27.32% Recall@1 over taking the retriever's top candidate, and the same
agent, transferred unchanged to PatchFinder_top10, lifts Recall@1 from PatchFinder's own top-1 pick (24.28%) to 39.86%. Swapping the LLM backbone within the Qwen family changes Recall@1 by under 1%, and a second model family (gpt-oss) stays far above the no-
agent floor, so the gain comes from the listwise
agent loop; the entire system runs on a frozen open-weight model over a local Git repository, without fine-tuning or external search APIs.
Read the full abstract ▾
All authors Guanqun Yang, Yingming Zhou, Jiangrui Zheng, Shudong Hao, Xueqing Liu
Machine Learning
Blackboard Intelligence Can Surpass Autoregressive on Globally Constrained Problems
Woosang Jeon, Jaeyeon Kim, Sham Kakade et al. (+6 more)
Next-
token prediction has driven remarkable progress in large
language models, yet a growing body of evidence suggests that they can struggle on problems governed by complex global constraints. In this work, we focus on this regime and ask whether some of these limitations arise from the inference interface induced by next-
token prediction itself. We study this question through blackboard intelligence: an inference-time perspective in which a model works on a fixed, revisable canvas and searches over candidate solution states rather than committing to a causal, left-to-right trajectory. We instantiate this idea with
diffusion language models, whose any-order prediction interface naturally exposes predictions over partially filled solution states. Our key observation is that mean confidence, a simple model-internal quantity available from the standard masked
diffusion objective, provides a useful proxy for global coherence and can guide inference-time search and revision. Empirically, across ZebraLogic, Nurse Rostering, and Job-Shop Scheduling, Blackboard consistently improves inference while holding the fine-tuned LLaDA-8B-Instruct checkpoint fixed and substantially outperforms same-scale autoregressive baselines, reaching 90.4% accuracy on ZebraLogic-Hard, 76.4% exact feasibility on Nurse Rostering, and 80.2% optimality on JSSP. Stronger autoregressive search and refinement also fail to close the gap on ZebraLogic-Hard, while Blackboard surpasses tested frontier LLMs there and on JSSP despite their substantially greater scale and strong test-time
reasoning. We open-source our codebase at https://github.com/jwoosang1/blackboard-intelligence.
Read the full abstract ▾
All authors Woosang Jeon, Jaeyeon Kim, Sham Kakade, Yilun Du, Amrit Singh Bedi, Arun Kumar Chithanar, Chul Lee, Taehyeong Kim, Sitan Chen
Machine Learning
Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agents
Huaiyu Fu, Heng Cao, Hao Wang et al. (+2 more)
LLM
agents often admit multiple high-quality solutions to the same task, differing in
reasoning structure, tool-use pattern, or interaction trajectory. Yet existing notions of diversity in LLM post-training are mostly implicit, arising from general stochasticity and regularization mechanisms rather than explicitly targeting task-relevant behavioral variation. While such implicit diversity can be useful, it does not directly specify which forms of behavioral variation should be encouraged for a given task. In this work, we study explicit trajectory diversity in RL-based post-training for LLMs. Our key idea is to define diversity through user-specified, task-specific trajectory descriptors, which map each sampled trajectory to an interpretable behavioral representation, and then measure diversity as a set-level functional over the resulting descriptor
matrix. Building on this formulation, we introduce Trajectory-guided Joint Policy Optimization(TJPO), a single-policy framework that optimizes explicit diversity over sampled trajectory groups, avoiding the need for population-based policy training, and instantiate it within group-based policy optimization through trajectory-level learning signals. This design makes the diversity objective both interpretable and controllable. Experiments on Sokoban and ALFWorld show that TJPO improves task-specific trajectory diversity while maintaining competitive task performance. Descriptor and trajectory analyses show that the learned variation follows the specified behavioral dimensions and includes distinct successful strategies. Extra experiment results suggest that explicitly shaping trajectory diversity can help LLM
agents satisfy user requirements and remain effective when task conditions change.
Read the full abstract ▾
All authors Huaiyu Fu, Heng Cao, Hao Wang, Jian Ya, Tao Chen
Computation and Language
Uncovering Uncontrolled Repetition through Residual Stream Dynamics
Yuanhe Zhang, Xinyao Zhou, Haoran Gao et al. (+5 more)
Uncontrolled repetition can prolong autoregressive generation in large
language models (LLMs) and enable resource consumption attacks. Prior analyses of repetitive generation have identified strongly activated features in intermediate and late layers. However, how uncontrolled repetition activity emerges and develops before becoming prominent in these layers remains insufficiently understood. In this paper, we investigate this question primarily in large
vision-
language models (LVLMs), which support a richer set of uncontrolled repetitions through both visual and textual inputs. We propose Tokenwise Residual Comparison (TRC), a method that identifies and localizes anomalies associated with repetition from residual dynamics during generation. TRC compares
attention and multilayer perceptron writes to the residual stream across generated
tokens to identify patterns associated with repetition. It then selectively suppresses coordinates in the residual stream at the identified layer. Experiments show that TRC effectively mitigates uncontrolled repetition, reducing loop rates by 57% on average. Our analysis further shows that repetition semantics emerge in shallow layers and propagate through the residual stream, disrupting normal representations. TRC also generalizes to large
language models (LLMs) and large
reasoning models (LRMs), where it consistently captures analogous repetition dynamics and achieves effective mitigation. Our work broadens the study of repetitive generation from its prominent internal representations to earlier opportunities for intervention, providing insights for mitigating resource consumption attacks.
Read the full abstract ▾
All authors Yuanhe Zhang, Xinyao Zhou, Haoran Gao, Yuyao Zhang, Zhenhong Zhou, Fanyu Meng, Li Sun, Sen Su
Computation and Language
Overlap, Unique and Conflict: Can LLMs Extract What They Can Recognize?
Eftekhar Hossain and Santu Karmaker
Understanding multi-perspective alternative narratives requires identifying how their information agrees, conflicts, or differs across sources. Existing work on cross-text relations largely focuses on categorizing relations between predefined text pairs, such as entailment or contradiction, rather than directly extracting such information from full narratives. To address this gap, we introduce Overlap-Unique-Conflict (OUC) extraction, a cross-narrative task that extracts all overlapping, conflicting, and unique clauses from two narratives. To support this study, we construct a
benchmark of approximately 22K narrative pairs and 140K OUC instances spanning factual, argumentative, and political discourse. Evaluating 14 open-source LLMs (0.6B-35B), we find that unique information is far easier to extract than overlap and conflict: the strongest model, Gemma-4-31B, reaches only 61.13% F1-score on overlap and 48.58% on conflict, against more than 75% on unique. Further diagnostic analysis reveals that this difficulty does not stem from relation recognition alone, but rather from a failure to pair and extract the corresponding clauses from full narratives, especially in smaller models. Nevertheless, learning these extractions with task-specific supervision narrows the gap considerably: a fine-tuned Qwen-3-8B gains 15-28% absolute over its baseline and surpasses models roughly four times its size (e.g., Qwen-3.6-35B) on several tasks. Even so, overlap and conflict remain well below satisfactory, leaving cross-narrative clause extraction an open challenge.
Read the full abstract ▾
Artificial Intelligence
GraphCert: Bootstrap Agentic Graph Reasoning with Certified Evidence Rubrics
Weiqi Jiang, Yuchen Ying, Rui Wang et al. (+5 more)
Graph
agents extend large
language models (LLMs) with the ability to actively explore and reason over knowledge graphs through multi-step interactions with graph tools. However, training capable graph
agents typically requires large collections of question-answer pairs and
reasoning trajectories, whose manual construction is costly and difficult to scale. Moreover, employing proprietary LLMs to generate such supervision further risks exposing sensitive graph data to external services. Therefore, we propose GraphCert to bootstrap agentic graph
reasoning with certified evidence rubrics during post-training. Specifically, the Bootstrapped Graph Quizzer guided by generation controls produces graph-grounded QA pairs and marks supporting evidence, which undergo execution certification and semantic curation. The accepted evidence is then canonicalized into certified evidence rubrics that later reward Graph Solver evidence
alignment alongside answer correctness during GRPO training. Experiments on five graph
reasoning domains in GRBENCH demonstrate that GraphCert consistently outperforms substantially larger LLM
agents and post-training method. Furthermore, our analysis demonstrates that the learned policy transfers robustly across heterogeneous graph domains, suggesting that GraphCert acquires reusable graph-
reasoning capabilities rather than domain-specific patterns. These results establish executable self-certification as an effective approach to self-training compact graph
reasoning agents. Our code will be made publicly available.
Read the full abstract ▾
All authors Weiqi Jiang, Yuchen Ying, Rui Wang, Kaixuan Chen, Bingde Hu, Shunyu Liu, Yu Wang, Tongya Zheng
Machine Learning
Evaluating Persistent Calibration under Evolving Model Knowledge
Victor Wang, Thomas Hofweber, Mohit Bansal et al. (+1 more)
As AI systems move from static repositories to
agents that are capable of continual adaptation and learning, maintaining their trustworthiness means equipping the models backing them with the ability to produce confidence estimates that dynamically reflect their changing skills and knowledge. We introduce the problem of persistent calibration, which requires a confidence
estimator to faithfully reflect the knowledge contained in a model as that knowledge changes, without recurring supervision. We operationalize this by examining persistent calibration across checkpoints of open models, asking whether confidence
estimators trained on earlier checkpoints can generalize to later ones. Specifically, we aim to shed light on whether confidence is dependent on knowledge, a question with implications for the reliability of confidence estimates. To measure this relationship, we define and evaluate calibration on knowledge contrast sets: subsets containing questions that one checkpoint answers correctly and another checkpoint answers incorrectly, reflecting a change in knowledge. We show that both inference-time and fine-tuning methods fall short on contrast-set calibration compared to oracle methods trained on future checkpoints, even for methods that are well-calibrated on the full
dataset. We provide evidence for the hypothesis that persistent calibration is challenging because there is a vast space of possible confidence functions that are well-calibrated on a given checkpoint, out of which only some rely on meta-knowledge features that would generalize to other checkpoints. Towards improving contrast-set calibration, we show that multi-checkpoint training helps, suggesting an avenue for identifying confidence features that remain robust across changing knowledge.
Read the full abstract ▾
All authors Victor Wang, Thomas Hofweber, Mohit Bansal, Elias Stengel-Eskin
Computation and Language
Recovering Off-Policy Supervision for Speculative Decoding
Jungseob Lee, Chanjun Park, Sugyeong Eo et al. (+1 more)
Block drafters for speculative decoding are commonly trained on corpora written by external models, where a single off-policy
token invalidates supervision for all subsequent slots in a block. Existing approaches discard these divergent slots, resulting in severe supervision loss. To resolve this problem while preserving the training corpus, we propose a rollout-based training framework that recovers full supervision through two complementary components. The first component, Anchor-Label Relabelling (ALR), replaces corpus labels with distributions from greedy target rollouts, restoring valid supervision across all predicted slots. The second component, In-Rollout Anchors (IRA), places draft blocks directly inside these rollouts to expose the drafter to target-generated context, reusing precomputed rollout features at no additional target cost. Across fixed
vision-
language and text corpora, our framework increases greedy accepted length by up to 36.5% over DFlash and consistently outperforms erasing baselines. Notably, a single epoch of our method surpasses the best erase schedules. After three epochs, it matches the acceptance length of training on target-regenerated responses. These results show that our framework provides an effective and compute-efficient approach for training speculative drafters on fixed corpora without modifying the original text. Code is available at https://github.com/js-lee-AI/ALR-IRA.
Read the full abstract ▾
All authors Jungseob Lee, Chanjun Park, Sugyeong Eo, Hyeonseok Moon
Human-Computer Interaction · Cross-listed from eess.AS
A barrier or a booster? Familiarity effects on Mandarin emotion prosody recognition using AI-powered voice cloning
Feng Xu, Gaoyuan Zhang, Shanshan Xue et al. (+4 more)
Emotion prosody perception requires simultaneous processing of acoustic cues and speaker identity. While listeners effortlessly decode natural
speech, AI synthetic voices introduce cognitive complexities due to subtle acoustic atypicalities. It remains unclear how these synthetic features interact with a listener's prior social knowledge and
memory of a familiar speaker. This study investigated how
speech sources (human vs. AI) and speaker familiarity affect
emotion recognition accuracy and cognitive load. A within-subject task with Mandarin-speaking adults evaluated behavioral (accuracy, reaction time) and physiological data (heart rate variability). Results showed that human voices yielded significantly higher accuracy and faster processing times than AI voices, while HRV did not significantly differentiate between conditions. These findings show that decoding synthetic
speech is gated by top-down social cognition, highlighting limitations in current AI synthesis technologies.
Read the full abstract ▾
All authors Feng Xu, Gaoyuan Zhang, Shanshan Xue, Yixiang Chen, Hanrui Zhou, Xurong Xie, Hui Chen
Computation and Language
Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
Ilgee Hong, Changlong Yu, Zhenghao Xu et al. (+5 more)
We study training LLM judges from natural
language feedback, especially for subjective tasks where the verdict depends strongly on which evaluation criteria the judge invokes and how it weighs them. The dominant approach, outcome-supervised RL (e.g., GRPO), credits every
token in the rollout with a single scalar determined only by the accuracy of the final verdict, providing no separate credit at the criterion-choice
tokens and ignoring the rich
language feedback (e.g., preference rationales) that naturally accompanies preference labels. Self-
Distillation (SD) is one natural way to use this
language feedback: the same model, conditioned on this feedback, acts as a teacher providing dense, position-level supervision. However, not all positions carry equally useful signal. Using the per-position
entropy shift between teacher and student, we identify two regimes: context sharpening, where the teacher concentrates probability on a particular feedback-aligned criterion expression, and context spreading, where the teacher distributes probability across multiple feedback-aligned alternatives. We interpret these patterns as follows: sharpening encourages memorization of a particular criterion expression, whereas spreading promotes semantic understanding by preserving these alternatives. Motivated by this asymmetry, we introduce position masking based on the
entropy shift that retains the lower tail of the
entropy-shift distribution. Experiments show that masking higher-
entropy-shift positions improves out-of-distribution generalization over naive SD. The resulting self-distilled judges outperform judges trained with outcome-supervised RL by 2-9 percentage points on the evaluated subjective subcategories, while remaining competitive on objective ones.
Read the full abstract ▾
All authors Ilgee Hong, Changlong Yu, Zhenghao Xu, Xin Liu, Yuwei Zhang, Qin Lu, Bing Yin, Tuo Zhao
Machine Learning
LEARN-TS: LLM-Enhanced Alignment and Reconstruction with Normality Guidance for Multivariate Time-Series Anomaly Detection
Jahyeob Koo, Kio Yun, Byoungmo Koo et al. (+1 more)
Reconstruction errors in multivariate time-series anomaly detection may not reliably distinguish abnormal behavior from benign deviations.
Language-derived semantics offer complementary context, but existing
multimodal approaches may rely on time-associated paired textual information that is difficult to obtain consistently and is not provided by standard multivariate time-series anomaly detection
benchmarks. This setting poses two challenges: (1) conditioning masked reconstruction on window-specific semantics without exposing exact numerical targets or anomaly-specific cues, and (2) using a window-independent concept of normality as a complementary semantic reference rather than an independent anomaly detector. We propose LLM-Enhanced
Alignment and Reconstruction with Normality Guidance for Time Series (LEARN-TS), which uses a frozen
language model to construct two role-separated semantic representations without requiring temporally paired external text. Window-specific observation semantics encode temporal and cross-variable context without exact numerical values to guide channel-shared patch-masked reconstruction. A fixed,
dataset-agnostic normality
prompt provides a window-independent semantic reference for aligning normal representations and estimating normality discrepancy. At inference, masking each temporal patch once yields timestamp-level reconstruction evidence, conditionally modulated by discrepancy from a separate unmasked view. Across four
benchmarks, LEARN-TS achieves the highest mean performance in 13 of 16
dataset-metric comparisons. Controlled ablations examine observation conditioning, joint normality
alignment and scoring, and reference content, showing
dataset-dependent ranking benefits and modest average gains from semantic over random references.
Read the full abstract ▾
All authors Jahyeob Koo, Kio Yun, Byoungmo Koo, Jun-Geol Baek