Computation and Language · Cross-listed from cs.LG
Geometry-Guided Layerwise FFN Width Allocation in Transformers
Timur Mudarisov, Mikhail Burtsev and Radu State
Feed-forward networks (FFNs) account for a large fraction of
Transformer parameters, yet their hidden width is usually constant across depth. We ask whether this capacity can instead be allocated from a forward-pass measurement of layer behavior. We view each FFN as transporting a cloud of
token representations and quantify the induced
geometric change using correspondence-preserving shift, Gromov-Wasserstein distortion, and degree-one persistent homology under raw and scale-normalized metrics. A layerwise approximation surrogate yields an exact fixed-budget optimizer. Across seven pretrained
language models, raw Euclidean work largely tracks residual-norm growth, whereas normalized work is predominantly front-loaded. Gromov-Wasserstein work is more consistently associated with perturbation-based layer sensitivity than the finite-sample topological estimate. In paired 128M and 256M training runs, several normalized-work schedules reduce mean validation loss relative to both uniform width and a hand-designed cosine taper. With the amplified paired differences at 440M, the best geometry-based allocations improve over uniform substantially larger than the cosine taper, while the anti-topological raw control is worse than uniform.
Read the full abstract ▾
Computation and Language
TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention
Avni Mittal, Avinash Anand, Ashutosh Kumar et al. (+7 more)
Can a strictly local, iterated, weight-shared computation primitive support
language modelling, and which of those three properties actually drives the model's behaviour? We define TextNCA, a 1D causal windowed-
attention realisation of the Neural Cellular Automaton primitive, and study a hierarchical variant that cascades three stages with windows $w \in \{8, 32, 128\}$ and $T_s$ shared-weight iterations per stage, all on WikiText-103 at roughly 30M parameters and 60k training steps. The model does not match a parameter-matched
Transformer at this scale (Hier-TextNCA $60.3$ vs.\
Transformer-6L $52.8$ and
Transformer-12L $44.7$ PPL), so we treat it as an analytical probe rather than a proposed alternative. The behaviour we observe is largely explained by the staged narrow-to-wide schedule: a non-iterating sliding-window
Transformer that reuses the same schedule comes within $+4.1$ PPL of the iterated model, while reversing, flattening, or breaking the monotonic ordering of the schedule costs between $+16.7$ and $+70.8$ PPL. Iteration adds a smaller bounded benefit on top of the schedule, with a clear optimum at $T_s{=}4$ and a U-shaped degradation beyond it. The GRU gate and learned per-step
embeddings are required for that benefit to appear, and training with random $T_s$ yields an inference-time iteration-count knob at the cost of substantially higher absolute PPL. We position the work as a controlled reading of which parts of NCA-style computation carry the weight in
language modelling.
Read the full abstract ▾
All authors Avni Mittal, Avinash Anand, Ashutosh Kumar, Dikshant Kukreja, Kritarth Prasad, Sushane Dulloo, Erik Cambria, Timothy Liu, Zhengkui Wang, Rajiv Ratn Shah
Computation and Language
CompanionBench: A Theory-Anchored, Real-World-Grounded Benchmark for AI Emotional Companionship
Yao Liu, Guangjia Chai, Yuming Huang et al. (+3 more)
LLM companions are deployed at scale in personally consequential settings, yet poorly evaluated. Existing
benchmarks use hand-authored scenarios and prompted simulators, aggregate empathy into one score, and overlook judge biases such as same-family favoritism and scale drift. We introduce CompanionBench, an interactive bilingual
benchmark. To our knowledge, it is the first companion
benchmark to ground both its scenarios and a trained user simulator in de-identified real-world data. A hidden disclosure gate branches each persona's trajectory on the
agent's own behavior, controlling the interaction state space without scripting dialogue. We operationalize ten capabilities derived from 25 theories across psychology and counseling, four of them not graded explicitly by prior work: holding ambiguity, selfobject responsiveness, positive resonance and calibrated challenge.
Agents are assessed on two complementary axes: a subjective ten-capability rubric and a deterministic measure of whether deeper disclosure was earned. A cross-family panel dilutes same-family favoritism; an Item Response Theory model separates
agent quality from judge severity. Theory fixes what to measure and how personas are structured; real data supply events, history, and profiles – coverage from theory, authenticity from data. Rankings are reproducible in both
languages (rho = 0.996 ZH / 0.953 EN). Evaluating 28
agents reveals capability-level differences obscured by aggregate scores.
Emotion regulation and calibrated challenge remain common weaknesses; holding ambiguity discriminates most. Role-play
agents rank near the bottom: immersion does not imply relational competence. Across
agents, the dominant failure mode is substituting surface warmth for substantive relational support. We will release 500 Chinese-English parallel pairs and the evaluation code.
Read the full abstract ▾
All authors Yao Liu, Guangjia Chai, Yuming Huang, Jihao Huang, Lei Wang, Junchen Wan
Computation and Language · Cross-listed from cs.CV
ET-Prune: Evidence-Aware Dynamic Budgeting for Visual Token Pruning in Text-Rich MLLMs
Zizhong Ding, Junxian Li, Kai Liu et al. (+4 more)
Visual
token pruning reduces the inference cost of
multimodal large
language models, but a fixed
token ratio is poorly matched to text-rich inputs. In OCR-centric tasks, decisive evidence can be a small number, label, or field whose relevance is specified by the question; indiscriminate
pruning can erase that evidence while retaining visually salient but irrelevant regions. We present ET-Prune, a training-free framework that casts
pruning as evidence allocation. It derives question-conditioned evidence from a decoder-side partial query-key block, safeguards text-like spatial regions, and converts evidence uncertainty and density into a sample-specific
token floor. Three progressive middle-layer events then move the sequence toward this budget, retaining more
tokens for diffuse or text-dense evidence and
pruning concentrated evidence more aggressively. At the observed point estimates from one deterministic pass per configuration, ET-Prune leads or ties among pruned methods in all six backbone-
benchmark comparisons at roughly half
tokens. On OCRBench-v2, it leads the strongest pruned baselines by 1.80 and 0.68 percentage points on Qwen3-VL-8B and InternVL3.5-8B, respectively, while retaining about half of the visual
tokens; on MMBench v1.1, it reaches 0.8467 circular exact-matching accuracy versus 0.8437 for Vanilla at 54.45% average visual-
token retention. These results show a favorable observed quality-cost trade-off for evidence-aware dynamic budgeting in text-rich
multimodal inference.
Read the full abstract ▾
All authors Zizhong Ding, Junxian Li, Kai Liu, Shaoqiu Zhang, Xiao Xiao, Linghe Kong, Yulun Zhang
Computation and Language · Cross-listed from cs.SE
TELLER: Non-intrusive Cross-Layer Root-Cause Analysis for LLM Inference
Ruilin Xu, Junyi Li, Pengfei Chen et al. (+1 more)
Large
language model (LLM) inference has evolved from an offline workload into a continuously operated software service, yet root-cause analysis remains difficult because a single request spans the inference engine, Python/C++ backend, host CUDA APIs, GPU kernels, and distributed communication. Existing profilers expose raw timelines, while log-based diagnosis often misses cross-layer execution semantics and request-level structure. We present TELLER, a non-intrusive Trace- and Log-aware LLM inference Root-cause analysis framework. TELLER first collects NVTX/CUPTI traces and service logs without modifying model binaries, then reconstructs per-request call-chain trees and aligns log lines with the corresponding execution steps. We introduce a dependency-aware causal-context slice that preserves parent-child structure, temporal order, and communication relations, and a Trace Pair Encoding (TPE) tokenizer that compresses such slices into compact structural
token sequences with parent, depth, and duration attributes. On top of these representations, TELLER combines numeric candidate localization with a
multimodal root-cause model that jointly predicts abnormal steps, localizes suspicious operators, and generates natural-
language explanations. Experiments on multi-node GPU inference workloads show a clear compression-accuracy trade-off: a moderate TPE vocabulary reduces per-step trace length by more than 80% while achieving the best overall performance on both horizontal (cross-node communication) and vertical (within-node execution stack) views, whereas more aggressive compression substantially degrades diagnosis quality. Further analyses under low-fault priors, strengthened baselines, modality ablations, explanation-quality checks, and tracing overhead show that TELLER provides a practical triage and evidence-localization substrate for LLM inference RCA.
Read the full abstract ▾
All authors Ruilin Xu, Junyi Li, Pengfei Chen, Zongxuan Xie
Computation and Language
Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation
Chishui Chen, Yaoyou Fan, Te Sun et al. (+11 more)
On-policy
distillation (OPD) provides teacher supervision on states visited by the student, reducing the distribution gap between training and inference. However, in multi-turn agentic tasks, student deviations may accumulate over time, gradually moving the trajectory away from states where teacher guidance remains effective. Our quantitative analysis further shows that high-disagreement states offer promising opportunities for teacher guidance, but determining whether such guidance is beneficial requires examining its effect on subsequent student trajectories. We propose FutureBridge-OPD (FTB), which executes a short teacher bridge at a high disagreement state and uses the resulting student continuation to assess whether the bridge increases the density of positive
distillation signals relative to the teacher. On ALFWorld, WebShop, and ScienceWorld, under the main Qwen3-32B teacher to Qwen3-1.7B student setting, FTB outperforms vanilla OPD and TCOD by an average of 16.6 and 7.6 points, respectively, and remains effective across student scales and teacher settings. Our code is publicly available at https://github.com/ChenChiShui/FutureBridge-OPD.
Read the full abstract ▾
All authors Chishui Chen, Yaoyou Fan, Te Sun, Yi Yang, Chenghao Sun, Delin Mao, Hongbo Qiao, Zuowei Zhang, Junxi Wang, Chenxing Sun, Yangen Hu, Lu Pan, Xuyang Liu, Linfeng Zhang
Computation and Language · Cross-listed from cs.CV
CultureVidBench: Benchmarking Cultural Understanding in Text-to-Video Generation
Xianjing Han, Yuhan Su, Yang Deng et al. (+3 more)
Text-to-
video (T2V) generation models have advanced rapidly, yet their ability to represent diverse cultural contexts remains underexplored. Existing
benchmarks mainly focus on perceptual quality, physical plausibility, and text-
video alignment, but do not directly assess whether generated
videos capture culturally specific objects, actions, rituals, visible text, or audio cues. We introduce CultureVidBench, a comprehensive
benchmark for evaluating cultural understanding in T2V generation. CultureVidBench contains 1,000 curated
prompts covering 12 countries, 6 continents, 8 cultural regions, and 14 cultural aspects organized into three categories: material culture, social practice & performance, and ritual & ceremony. Designed specifically for
video generation, CultureVidBench emphasizes dynamic and
multimodal cultural representation, including social interactions, ritual procedure, and culturally appropriate visible text and audio. We evaluate seven representative T2V models through human user studies and MLLM-based automatic assessment across cultural faithfulness,
multimodal cultural rendering, semantic adherence, and perceptual quality. Results show that although current models achieve strong semantic adherence and visual quality, they often fail to faithfully capture fine-grained cultural details, particularly for underrepresented regions, rituals, and
multimodal cultural cues.
Read the full abstract ▾
All authors Xianjing Han, Yuhan Su, Yang Deng, Dong Ma, Wee Peng Tay, Bin Zhu
Computation and Language
Automatic Annotation of Ancient Greek Vowel Length
Albin Thörn Cleland and Eric Cullhed
Prior work in Ancient Greek NLP relies on corpora that do not disambiguate the phonemic vowel length of alpha, iota, and ypsilon, together known as the dichrona. Depending on lexeme, morphology, sandhi, syntax, and conventions of period, genre, and verse form, each of these letters can represent either a long or a short vowel. Deciding and marking the correct length is known as "macronizing", a long-tail problem given the sheer mass of word forms and the context dependency of individual instances. No macronized corpus of Ancient Greek is publicly available at scale, so a stand-alone macronizer is needed. While previous work has shown how to build a static, corpus-bespoke vowel-length dictionary, the present paper constructs the first general-purpose macronizer for arbitrary Ancient Greek input. Given input carrying lemma, part-of-
speech, and morphological annotation in the standard CoNLL-U format, a set of recursive modules lets less common word forms inherit markup from more common forms of the same lexical word. The macronizer's chief application is generating training data for machine learning: we show that a small character-level
transformer trained on the macronizer's own output learns to generalize past the cases the rule-based system leaves unmarked, matching or exceeding its accuracy on a gold-standard, manually annotated
benchmark of verse and prose. We also show that macronization can improve downstream prosodical NLP tasks like verse scansion.
Read the full abstract ▾
Computation and Language
TRAM: Enhancing Multimodal Reasoning with Trajectory-Derived Auxiliary Memory
Kang Liu, Zijing Wang, Yongkang Liu et al. (+5 more)
Multimodal Large
Reasoning Models (MLRMs) have achieved strong performance on tasks requiring visual understanding and multi-step inference. However, as
reasoning trajectories grow, models may become less effective at using information established earlier in the context, increasing the risk of
reasoning errors. Existing approaches primarily address this problem by sustaining visual grounding throughout
reasoning. However,
reasoning also transforms visual observations into task-specific relations, constraints, and intermediate conclusions whose influence may weaken over long trajectories. Our attribution analysis suggests that correctness is not consistently separated by image attribution alone, but is more closely associated with whether trajectories retain and integrate such
reasoning-derived information across stages. Motivated by this, we introduce TRAM (TRajectory-derived Auxiliary
Memory), a training-free method that augments standard decoding with an auxiliary
memory pathway derived from the model's own
reasoning trajectory. TRAM consolidates completed
reasoning into a compact latent
memory, updates it online through fast and slow recurrent streams, and feeds it back into selected decoder layers through a lightweight residual pathway. Experiments across four MLRM variants on eight
benchmarks show that TRAM improves performance over vanilla decoding on mathematical, scientific, and general visual
reasoning tasks without additional training.
Read the full abstract ▾
All authors Kang Liu, Zijing Wang, Yongkang Liu, Mengjie Zhao, Xiaocui Yang, Shi Feng, Yifei Zhang, Daling Wang
Computation and Language · Cross-listed from cs.LG
HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses
Luan Zhang, Ruochen Zhou, Dandan Song et al. (+9 more)
Harness design plays a critical role in
agent performance by shaping how large
language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness
evolution, which iteratively improves the harness from
agent–environment interactions. However, existing methods often overfit to the
evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness
evolution framework built around constrained
evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints on
evolution, restricting modifications to task-agnostic harness changes that generalize beyond the
evolution tasks. It then augments trajectory-derived evidence with proactive first-person feedback from the
agent about harness usage, yielding richer signals for
evolution. Finally, it decouples the optimization of different harness components before consolidating them into a unified harness, reducing cross-component interference while preserving component synergy. On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54% to 66% in only 5
evolution iterations, outperforming AHE in both effectiveness and
evolution efficiency. In addition, the evolved harness transfers effectively to held-out tasks and other models, demonstrating substantially stronger generalization than prior automatic harness
evolution methods.
Read the full abstract ▾
All authors Luan Zhang, Ruochen Zhou, Dandan Song, Zhengyu Chen, Yuhang Tian, Jun Yang, Huipeng Ma, Chenhao Li, Guangyuan Feng, Xudong Li, Yizhou Jin, Yan Xu
Computation and Language · Cross-listed from cs.AI
Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents
Qi Liu, Jiaxin Mao, Fengbin Zhu et al. (+1 more)
Deep search
agents answer difficult information-seeking questions by iteratively issuing search queries to gather supporting evidence, but it remains unclear whether and how greater search effort leads to better answers. We study these questions through a trajectory-level diagnosis of long-horizon search
agents. Using human-annotated document-level relevance judgments, we evaluate the evidence retrieved at each search step and separate two stages of
agent behavior: what evidence an
agent retrieves and how effectively it uses that evidence. This distinction further allows us to decompose failures into
retrieval gaps, where the necessary evidence is never found, and utilization gaps, where relevant evidence is retrieved but not used correctly. With the
retrieval model and evaluation harness held fixed, we compare six
agents on BrowseComp-Plus and further validate our findings on BrowseComp with an open-web search API. Across settings, we find that search effort and answer quality are only weakly aligned. Answer accuracy is better correlated with the quality of retrieved evidence, especially cumulative
retrieval recall, than with the number of searches or the amount of context consumed. Useful evidence often appears early in the trajectory, yet
agents tend to continue searching, producing a long tail of low-yield
retrieval steps. At the query level, exploratory reformulations remain useful, but the best-performing
agents issue far fewer redundant queries. Overall, by systematically characterizing the search behavior and failure modes of long-horizon search
agents, this work points to practical directions for building better deep research systems, including stronger query formulation, more effective evidence selection and context management, and stopping criteria based on whether sufficient supporting evidence has been retrieved.
Read the full abstract ▾
All authors Qi Liu, Jiaxin Mao, Fengbin Zhu, Tat-Seng Chua
Computation and Language · Cross-listed from cs.CV
SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models
Jing Wu, Jianhua Wu, Jiayi Guan et al. (+5 more)
Vision-
Language Models (VLMs) perform well on commonsense
reasoning tasks but struggle with visual spatial
reasoning. Most existing solutions introduce extra 3D prior inputs or external spatial encoders, which increase complexity and degrade the underlying VLMs' general-purpose capabilities after spatial fine-tuning. To this end, we propose a parameter-efficient \textit{Spatio-
vision Language Models (SpatioLM)}, that enhances spatial intelligence without extra 3D prior inputs or third-party spatial encoders. Concretely, we design a plug-and-play and non-invasive spatio-
vision module that elicits the spatial knowledge inherent in VLMs. Furthermore, we innovatively leverage pseudo depth and camera information as supervision to guide the model in learning physically coherent representations. Extensive experiments show that SpatioLM achieves significant improvements in diverse tasks, including spatial perception and understanding while effectively limiting the degradation of general capabilities. Notably, the model achieves an impressive score of 71.6 on the VSI-Bench (the first model to surpass 70). In addition, it attains competitive performance when transferred to
embodied manipulation tasks. Code is available at \href{https://github.com/xiaomi-research/spatio-lm}{\faGithub~spatio-lm}.
Read the full abstract ▾
All authors Jing Wu, Jianhua Wu, Jiayi Guan, Jiahong Chen, Jinghui Lu, Hangjun Ye, Bingzhao Gao, Long Chen
Computation and Language · Cross-listed from cs.CY
No One Wins in Nuclear War: A Social Simulation of Military Decision-making
Glenn Matlin, Isaac Song, Anthony Wen-Ming Zang et al. (+1 more)
WOPR is a social-simulation environment for studying how organizations make high-stakes decisions, built on a deterministic, replay-validated rules engine and using wargames as the vehicle. We instantiate it first with the published card game
Nuclear War, traced against its published rules. We start with military decision-making because of its safety implications and because it needs further study, but the design is not specific to it: the decision-point contract that exposes the engine to
agents is reusable across verifiable rule systems. Existing social-simulation work emphasizes persona fidelity and synthetic opinion, but lacks a verifiable rules engine with replay-checkable mechanics and private-channel negotiation. WOPR supplies that engine, and its contract makes every strategic choice an explicit
agent decision. The method is agnostic to social-simulation frameworks; we adopt Concordia as the default harness for driving the game. On the same engine, WOPR layers a four-rung press ladder from silence to private single-recipient channels with structured commitments, and instantiates each faction as a collective command-and-control system rather than a single
agent. We make all code, example configurations, and replay data publicly available at https://github.com/eilab-gt/wopr.
Read the full abstract ▾
All authors Glenn Matlin, Isaac Song, Anthony Wen-Ming Zang, Mark Riedl
Computation and Language
CRISP: Critical Step Perception for Training Efficient Deep Search Agents
Haosi Mo, Zihao Yan, Ruiqing Zhang et al. (+4 more)
Large
language models (LLMs) are increasingly extended into deep search
agents that solve complex questions through multi-step interaction with external search and browsing tools. However, existing
agents often incur substantial computational and interaction costs, generating lengthy trajectories that contain redundant queries, inefficient exploration, and irrelevant observations. Existing efficiency-oriented methods usually encourage
agents to use tools less frequently, but treating all tool interactions uniformly may also suppress steps that gather necessary evidence. In this paper, we propose CRISP, a framework for training efficient deep search
agents through critical step perception. Unlike prior efficiency methods that uniformly penalize tool use, CRISP distinguishes interactions that gather necessary evidence from redundant ones and shapes the training reward to preserve the former while
pruning the latter, improving efficiency without sacrificing the evidence needed for correct answers. Specifically, CRISP first constructs critical-step labels with Backward Evidence Induction: starting from the final answer, a strong model traverses a completed search trajectory backward and judges whether each tool-interaction step provides or preserves evidence for the final answer. We then distill these step-wise judgments into a smaller critical-step recognizer, enabling full-trajectory analysis in a single pass. During policy optimization, an efficiency-aware reward is applied only to successful rollouts. Experiments on BrowseComp and HLE-Verified show that CRISP maintains competitive final-answer accuracy while reducing average interaction turns by 15.1% and 33.2%, respectively, demonstrating substantial improvements in interaction efficiency.
Read the full abstract ▾
All authors Haosi Mo, Zihao Yan, Ruiqing Zhang, Zhongli Li, Hexuan Deng, Xuebo Liu, Min Zhang
Computation and Language
Analyzing Speech Condition Effects in Dysarthric ASR: A Layer-wise Probing Study
Darwin Jelestin Muthu, Navya Gupta, Wei Lin Tay et al. (+3 more)
Automatic
speech recognition (ASR) performance degrades sharply on dysarthric
speech, yet how disordered articulation reshapes a model's internal representations is underexplored. We conduct a layer-wise probing analysis of a
transformer ASR encoder on Mandarin dysarthric
speech under three transcript-matched conditions: original dysarthric
speech, speaker conditioned zero-shot TTS resynthesis, and unconditioned TTS. Probing reveals a task- and condition-dependent representation hierarchy: phoneme boundary information remains weak across all layers for dysarthric
speech; phoneme identity is recoverable in deep layers for synthetic
speech, but remains poor for dysarthric
speech; and recognition difficulty is concentrated in the deepest layers. Furthermore, lexical tone is a persistent error source across all conditions. Guided by these insights, layer-selective LoRA shows that mid-layer adaptation (layer 7 or layers 5-8) recovers near-full encoder performance on dysarthric
speech within 6.67% and 2.89% relative margins while training only 0.16% and 0.65% of adapter parameters. Conversely, upper-layer adaptation benefits synthetic
speech more than dysarthric
speech. These findings link representation analysis to parameter-efficient fine-tuning and motivate layer-aware adaptation for low-resource Mandarin dysarthric ASR.
Read the full abstract ▾
All authors Darwin Jelestin Muthu, Navya Gupta, Wei Lin Tay, Zhengchen Zhang, Daniel Wang Zhengkui, Rong Tong
Computation and Language
Divergent large language model predictions from convergent representations in ambiguous word pairs
K. Jack Scott, Narun Pat and Veronica Liesaputra
In this work we investigate how decoder-only
transformers resolve lexical ambiguity through layer-by-layer analysis of three models spanning three parameter sizes (GPT-2-Small-117M, Llama-3.2-3B, Qwen2.5-32B). For both homonyms and polysemes, we find that representations become maximally distinct in middle layers, then partially reconverge in late layers, while the KL divergence between their next-
token predictions reaches its maximum in the final layers. The activation patching experiment provides causal evidence that late-layer representational differences directly determine outputs despite apparent increased similarity in
embedding space. Our single-layer ablation experiment indicates that models achieve equivalent disambiguation despite qualitatively different layer-wise vulnerabilities. These findings offer a mechanism for recent observations where models' internal
embedding similarities show low correlation with their behavioural outputs despite strong performance. The semantic distinctions therefore remain present but become increasingly invisible to similarity measures over the
embeddings, with implications for
embedding-based methods such as semantic search,
retrieval, and clustering that rely on late-layer cosine similarity.
Read the full abstract ▾
Computation and Language
RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluation
Divyansh Singh, Reza Davari and Afra Mashhadi
Rubric-based LLM-as-judge pipelines often assume that evaluation criteria provide independent signals. In practice, however, criteria can be behaviorally coupled: improving one criterion may systematically change scores on another, distorting aggregate scores used in model-release or product-update decisions. We introduce RADAR, a lightweight preflight diagnostic framework for estimating such coupling before large-scale evaluation. Given a rubric, RADAR generates targeted synthetic probes, scores each probe on all criteria, and produces a directional coupling
matrix that shows which criteria co-score and how. We validate RADAR on three industry-relevant evaluation settings: NVIDIA HelpSteer2, SumPubMed, and the Yale-Salesforce SummEval
benchmark. Using only a small number of probes per criterion, RADAR recovers human inter-criterion correlation structure (Pearson r > 0.84) and provides practitioners with concrete audit signals about redundancy, hierarchy, and aggregation sensitivity before committing to large-scale judging.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Illuminating Visual Identity in Universal Multimodal Embeddings
Jiawei Cao, Junyi Feng, Jiashen Hua et al. (+5 more)
Universal
Multimodal Embeddings (UMEs) aim to unify various modalities and tasks into a shared representation space. In recent years, this field has witnessed substantial progress driven by the development of
Multimodal Large
Language Models (MLLMs). However, a crucial capability, visual identity discrimination, remains underexplored in existing UME methods, despite its critical role in a wide range of tasks, including instance
retrieval, re-identification, and identity preservation in AI-generated content. To bridge this gap, we propose a unified formulation for visual identity discrimination~(VisID) and introduce $\textbf{MVEB}$ ($\textbf{M}$ultimodal $\textbf{V}$isual Identity $\textbf{E}$mbedding $\textbf{B}$enchmark), a large-scale
benchmark curated from both real-world and synthetic
datasets to support evaluation and training. Furthermore, we present a simple yet effective learning framework that jointly optimizes general
multimodal and visual identity representations through a carefully designed identity-aware sampling mechanism. Extensive experiments demonstrate that our approach successfully endows UMEs with strong identity discrimination capability and maintains competitive general
multimodal performance. We believe this work not only illuminates a critical yet neglected capability, but also takes a step toward more holistic universal
multimodal embeddings. Code and data are available at \href{https://chrisclear3.github.io/MVEB}{MVEB}.
Read the full abstract ▾
All authors Jiawei Cao, Junyi Feng, Jiashen Hua, Ziheng Huang, Bing Deng, Kaijie Wu, Chaochen Gu, Jieping Ye
Computation and Language · Cross-listed from cs.AI
Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction
Priyashree Roy, Sujitha Martin, Mohammad Rostami et al. (+6 more)
Intelligent document processing (IDP) with
vision-
language models (VLMs) hinges on confidence scores trustworthy enough to route extractions between automation and human review. Existing document
benchmarks are dominated by clean, high-quality samples, leaving low accuracy regions too sparse for calibration assessment. We introduce ConfBench, the first calibration-specific
benchmark for key information extraction (KIE), built by applying 20 controlled degradation pipelines to a diverse document set, yielding 1,346 variants and 70K+ entity-level evaluations spanning the full accuracy spectrum. We evaluate four proprietary and three open-weight VLMs under verbalized and log-probability confidence estimation methods across three input modalities, and find: (i) OCR+Image modality results in more accurate confidence estimates; (ii) model capability is the dominant factor: within the Claude family confidence quality scales monotonically with capability, while across families parameter count is a poor predictor; (iii) calibration quality varies widely across models, from near-perfect to severely overconfident, and per-model post-hoc correction rescales these absolute confidence values for threshold-based routing without altering ranking-based operational metrics; and (iv) log-probability with first-
token aggregation consistently outperforms mean-
token and margin aggregations. We also introduce ECARB, a review-budget metric translating discriminative gains into operational savings. We release ConfBench publicly to enable systematic study of confidence
estimators and calibration methods for trustworthy IDP application deployment.
Read the full abstract ▾
All authors Priyashree Roy, Sujitha Martin, Mohammad Rostami, Spencer Romo, Renhao Xue, Bob Strahan, Diego A. Socolinsky, Boyi Xie, Md Mofijul Islam
Computation and Language · Cross-listed from cs.AI
REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models
Xiang Xia, Cheng Yan, Yiming Zhang et al. (+3 more)
Mixture-of-experts (MoE) models increase parameter capacity by activating only a small subset of experts for each
token. This conditional-computation paradigm has enabled autoregressive
language models to scale model capacity without a proportional increase in per-
token computation. In
diffusion language models (DLMs), however, each denoising forward jointly revisits all
token positions despite their sharply different refinement demands, while the default fixed
token-choice routing assigns them a uniform expert budget, creating a mismatch between expert computation and refinement demand. We argue that MoE inference in DLMs should therefore be viewed as refinement-aware compute allocation across heterogeneous
token refinement states. We propose REFLEX (REfinement-aware FLEXible expert allocation), a training-free method that keeps the default router unchanged while reorganizing expert computation around the evolving refinement process. Specifically, REFLEX introduces a coarse-to-fine hierarchy for expert-budget allocation that aligns computation with block-relative refinement roles while using the Frontier-Progress Score to resolve active-block priorities. Across multiple widely used
benchmarks on two representative MoE-based DLMs, LLaDA-MoE and LLaDA2.0-mini, REFLEX reduces allocated expert computation by 15% on average while preserving or even improving generation quality on most
benchmarks relative to default routing. Compared with autoregressive-style variable-expert routing methods, REFLEX also yields a more consistent quality–computation trade-off, further supporting the importance of allocating expert computation according to the heterogeneous refinement demands exposed within each denoising forward.
Read the full abstract ▾
All authors Xiang Xia, Cheng Yan, Yiming Zhang, Jiazheng Liu, Hongyu Zhang, Wuyang Zhang
Computation and Language
Constructing Parallel Multidimensional Chromatic Lexicons for Corpus-Assisted Analysis of Russian and English Texts
Larisa Nikitina
This article addresses the relative scarcity of research tools for the corpus-assisted linguistic analysis of colour terms in literary texts. It describes the development of two multidimensional chromatic lexicons: one for Russian (224 entries) and one for English (141 entries). Lexicon construction involved sourcing colour vocabulary from specialised resources and research literature, comparing the two
language inventories, manually checking translated candidates, and addressing
language-specific morphological features. In addition to identifying colour terms and visual descriptors, the lexicons classify entries according to hue, saturation, and temperature. To demonstrate their practical application, a pilot study was conducted on purposively sampled corpora of poetry by Andrei Bely (20,373
tokens) and Emily Dickinson (28,479
tokens). All retrieved matches were checked in context and classified as Confirmed_chromatic, Ambiguous_visual, or Excluded. The analysis was implemented in two main stages: a strict analysis including confirmed chromatic lexis only, followed by a sensitivity analysis incorporating both confirmed and ambiguous chromatic lexis to determine whether coding decisions about borderline cases affected the main findings. The quantitative results indicated marked differences in the use of colour terms, visual descriptors, hue, saturation, and temperature. Specifically, the analysis revealed that confirmed chromatic terms occurred 3.4 times more frequently in the sampled Bely corpus than in the Dickinson corpus. These findings demonstrate the analytical value of a multidimensional approach, with the main contribution of this study being a transparent and reusable procedure for constructing and applying
multilingual chromatic lexicons.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning
Li Wang, Xiaodong Lu, Xiaohan Wang et al. (+4 more)
Reinforcement learning (RL) has become a central paradigm for large
language model (LLM) post-training, but optimization toward new objectives can degrade capabilities already present in the base model. KL regularization is widely used to mitigate such forgetting by constraining policy drift toward a reference model. However, standard full-policy KL regularization constrains the entire response distribution and may unnecessarily restrict exploration and target-task learning. This raises a natural question: can a more precise constraint preserve existing capabilities while minimizing interference with learning new tasks? To this end, we propose \underline{Co}rrectness-Conditioned \underline{KL} Regularization (CoKL), a conditional regularization framework that narrows the preservation constraint from the full output distribution to correctness-conditioned response distributions. We instantiate CoKL with forward KL divergence and derive a practical finite-group training objective for RL-based LLM post-training. At the population level, CoKL decouples the total probability assigned to correct responses from their correctness-conditioned distribution, thereby regularizing the relative probability allocation among reference-supported correct responses without directly anchoring incorrect outputs or total correctness mass. We further show that full-policy forward and reverse KL regularization induce a strict optimal correctness gap when the reference policy is imperfect, whereas CoKL avoids this limitation. Experiments in controlled multi-solution environments and continual post-training settings across multiple model scales demonstrate that CoKL achieves a more favorable balance between target-task improvement and prior-capability retention than existing regularization methods. Our code is available at https://github.com/Lumina04/CoKL.
Read the full abstract ▾
All authors Li Wang, Xiaodong Lu, Xiaohan Wang, Jiajun Chai, Wei Lin, Tianhao Peng, Guojun Yin
Computation and Language · Cross-listed from cs.AI
MemSIF: From Structured Interactions to Dual-Track Fact Memory for LLM Agents
YuFei Luo, Xiucheng Xu and Zhen Yang
Long-term
memory is critical for LLM
agents operating over long-horizon interactions. However, several persistent limitations of existing
memory systems can be traced to two recurring misalignment patterns in long-term interaction settings: Temporal-Structural Misalignment (TSM) and Delayed Utility Manifestation (DUM). TSM arises when temporal proximity does not reliably align with topical or event-level relatedness, whereas DUM arises when write-time salience does not reliably predict future query utility. To mitigate these misalignment patterns, we propose MemSIF (
Memory with Structured Interactions and Facts), a structured interaction-to-fact
memory framework. Structured Interaction
Memory organizes raw interactions into Topical Segments that preserve local topical coherence and Event Trajectories that maintain cross-time event continuity. Dual-Track Fact
Memory uses two complementary tracks: CoreFact
memory consolidates stable, schema-guided information at write time, whereas ActiveFact
memory forms facts on demand and promotes those supported by multiple historical sources and recurring query demand for reuse. Experiments on LoCoMo and LongMemEval-S across five backbone LLMs show that MemSIF achieves the highest Total ACC in all settings, outperforming the strongest baseline by 2.29%-8.79% on LoCoMo and 2.87%-6.15% on LongMemEval-S. These results support the effectiveness of combining Structured Interaction
Memory with Dual-Track Fact
Memory to mitigate TSM and DUM. Code is available at https://github.com/luoyufeihaha/MemSIF.
Read the full abstract ▾
Computation and Language
TIDES: A Longitudinal Bilingual Dataset for Modeling Multi-Party Social Dynamics
Heechan Lee, Jeonggyu Kang, Junho Myung et al. (+3 more)
Group conversations are fundamental to human collaboration, yet standard large
language models (LLMs) still struggle with the complexities of multi-party interaction. This challenge persists in part because existing group conversation
datasets are often limited to short-term lab settings with contrived tasks, failing to capture the long-term social dynamics of real-world teams. To bridge this gap, we introduce TIDES, a high-resolution longitudinal
dataset tracking 12 university project teams over a full semester. Comprising 75,971 utterances in both English and Korean from in-person meetings, TIDES provides a naturalistic record of teams working on self-managed projects. Our socio-structural annotations-covering interaction types, emergent roles, and development stages-allow for modeling of team
evolution over months. Experiments show that fine-tuning on TIDES improves next-speaker prediction by 13.8 percentage points over a bigram baseline (64.53%) and yields performance comparable to strong proprietary zero-shot models. The model also comes within 2.1 percentage points of the published state of the art on the AMI Meeting Corpus while using approximately 42% less training data. However, human evaluations suggest that better next-speaker prediction does not necessarily yield more natural or coherent utterances, as fine-tuned models were generally less preferred than vanilla models. This potential mismatch motivates further study of how structural modeling can support natural multi-party generation.
Read the full abstract ▾
All authors Heechan Lee, Jeonggyu Kang, Junho Myung, Jaywoong Jeong, Juho Kim, Joseph Seering
Computation and Language
PGMem: Tightly Coupled Persona-Memory Graph for Lifelong Personalized Agents
Wonjun Choi, Yerim Kim, Yukyung Lee et al. (+1 more)
Long-term personalized dialogue
agents must track user preferences as their personas evolve. Existing
memory systems organize past events well, but store personas as flat profiles detached from the events that justify them. This loose coupling leads to the
memory-persona validity gap and the persona-aware
retrieval gap. We propose PGMem, a heterogeneous persona-
memory graph that connects event and persona nodes through typed provenance and evidence edges, keeping each persona signal traceable to the events that support or revise it. At
retrieval time, PGMem expands from query-relevant seeds and ranks signals by evidential validity. Across three
benchmarks with small
language model backbones, PGMem consistently outperforms summary-based, persona-aware, graph-structured, and agentic
memory baselines, and improves performance as the context grows. The source code of PGMem is available at https://github.com/wonjunchoi23/pgmem/
Read the full abstract ▾
All authors Wonjun Choi, Yerim Kim, Yukyung Lee, Susik Yoon
Computation and Language · Cross-listed from cs.IR
Floor, Ceiling, and the Fusion Gap: How Much of Crowd Reading Attention Can Machines Predict?
Kazuki Nakayashiki and Keisuke Watanabe
A
benchmark score means nothing without knowing what a trivial method achieves and what the best possible method could achieve. We construct both bounds for a task with a rare kind of ground truth: predicting which sentences a crowd of readers – highlighting for their own purposes, unpaid, uninstructed, and blind to each other – marked in 120 web documents. The floor is naive truncation (lead); the ceiling is a split-half oracle: half the crowd predicting the other half. The gap between them is +0.2028 AP [+0.1698, +0.2342, domain-clustered], and three findings structure it. First, the gap is semantic: position and length features recover 5% of it. Second, frontier
language models reach 35-53% of it zero-shot – far above classical baselines, far below the crowd; a state-of-the-art
prompt compressor (LLMLingua-2) lands below the floor, indistinguishable from random selection. Third, an unweighted cross-vendor fusion of five frontier rankings plus a position prior reaches 60%, beating the best single model by +0.0159 [+0.0044, +0.0269; Holm p=0.019] – a gain that survives ablation of its best member, split-half arm selection,
prompt paraphrase, and label, gate, and seed perturbations, and was CONFIRMED by a pre-registered replication on 217 independent documents (+0.0179, Holm p=0.042). Finally, the bracket compresses: distilling the fusion into one open-weight 8B student that reads the whole document retains 90% of the fusion's edge and reaches statistical parity with the strongest single frontier model (+0.0070 [-0.0068, +0.0200]), where a local-context student retains only 63% – the crowd's signal lives in document-level structure, and the cheapest known improvement is to ask several different models and average.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Progressive Agent Skill Generation via Reinforcement Learning
Junhao Shen, Chengzhi Piao, Zhanqiu Zhang et al. (+2 more)
Recent large
language model
agents often use external skills as modular procedural units that condition inference and improve complex task solving. Thus, automatically generating high-quality skills from documents or experience has become an important problem. Existing skill generation methods largely rely on heuristics or pipeline-style consolidation, which must be specially designed for different evidence sources. In contrast, learning-based approaches offer a more unified way to model skill generation across heterogeneous sources. However, learning-based skill generation remains challenging because skills lack a natural supervision signal based on relevance or correctness; their value can largely be determined only by whether they improve the behavior of the
agent on downstream tasks. To address this challenge, we propose Skill-$\alpha$, a reinforcement learning method that learns a unified policy for progressive skill generation. Specifically, we construct each skill by repeatedly applying the learned policy to successive source evidence and introduce a novel rollback reward that evaluates each edit by comparing downstream execution under the original and edited skills on an anchored query. Extensive experiments show that Skill-$\alpha$ generates more effective skills than methods based on heuristics or pipelines in both document-to-skill and experience-to-skill settings. Under the main GPT-4o worker, Skill-$\alpha$ improves average downstream success rates over the strongest skill-generation baseline by 3.1 points on CL-Bench and 6.7 points on tau2-bench. Further ablations and analysis validate the importance of rollback reward and progressive generation.
Read the full abstract ▾
All authors Junhao Shen, Chengzhi Piao, Zhanqiu Zhang, Yiwen Guo, Hong Cheng
Computation and Language
Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation
Xingyu Ren, Youran Sun, Chugang Yi et al. (+1 more)
Sparse
attention is widely deployed in long-context serving stacks, yet no framework audits how discarding blocks changes the influence of specific content on model output. We first establish that the phenomenon is real and causal: Block Sparse Flash
Attention (BSFA) route replay across four architectures changes output decisions in 13 of 16
cells, with zero identity-replay label flips. We then introduce a dense-calibrated counterfactual audit using matched probe cards—Gold (carrying the correct answer label), Poison (carrying a target wrong label), and Benign (filler only)—under six-layout position
symmetry, isolating the sparsification-specific effect. Two patterns compete. Signal concentration: the selector preserves Gold and Poison blocks far above filler-matched Benign blocks (G$\approx$P$\gg$B across all model–task pairs). Integration loss: discarding blocks severs cross-block
attention—confirmed by an ablation where isolating the probe block collapses its influence from 4.48 logits to zero. Compression ratio governs the balance: a full sweep from mild ($c=0.25$) to aggressive ($c=0.75$) compression across four model–task pairs reveals that three of four
cells move toward stronger sparse amplification at higher compression, with two exhibiting sign reversals. Three independent arms—BSFA route replay, controlled block-top-$k$, and KV-cache eviction—converge: sparsification changes content influence in ways aggregate accuracy cannot detect. We provide an open measurement framework deployable on any model exposing block identities.
Read the full abstract ▾
All authors Xingyu Ren, Youran Sun, Chugang Yi, Haizhao Yang
Computation and Language
Learning What to Remember: Test-Time Training via Context Distillation
Zixuan Wang, Xingyu Dang, Rui-Jie Zhu et al. (+4 more)
Effective long-context modeling is not merely about retaining more of the past, but about preserving the information that may prove relevant later. Test-time training (TTT) is an appealing approach that performs online parameter updates for long-context modeling, yet existing TTT methods only optimize either reconstruction or online adaptation objectives without considering the future utility of retained information. In this work, we propose Test-Time Context
Distillation (TTCD), a TTT framework that introduces a self-supervised objective for allocating limited
memory capacity for future use. Specifically, TTCD uses a long-window teacher to supervise the fast weights of a short-window student, where the hidden-state discrepancy between them offers a dense, self-supervised signal guiding the model to memorize the contextual information crucial for future
token predictions. We focus on an in-place variant: In-Place TTCD (IP-TTCD), which uses the existing MLP parameters as the fast weights. Experiments on long-context
language modeling tasks show IP-TTCD consistently outperforms DeltaNet, Gated DeltaNet, sliding-window
attention, and TTT when pre-trained from scratch. Furthermore, IP-TTCD allows pre-trained
transformer models to adapt their parameters during inference through continual pre-training, gaining long-context capabilities with only a lightweight architectural augmentation. Our results position TTCD as a step toward architectural continual learning.
Read the full abstract ▾
All authors Zixuan Wang, Xingyu Dang, Rui-Jie Zhu, Zixin Wen, Hengyu Fu, Wenhao Chai, Jason D. Lee
Computation and Language
Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation
Fengxian Ji, Yuke Li, Jingpu Yang et al. (+8 more)
However, whether these judges truly evaluate the scientific substance of ideas or are influenced by superficial stylistic presentation remains an open question. To address this question, we propose SciStyleBench, a unified three-component
benchmark for diagnosing and mitigating stylistic bias in LLM-based idea evaluation: (i) First, SciStyleStage, a three-stage evaluation environment that applies controlled stylistic perturbations to fixed scientific content across three settings no context, fixed-domain context, and open-domain
retrieval context, covering 600 scientific ideas and 15 style variants, with 9,000 evaluation instances per setting; (ii) Second, SciStyleMetrics, a set of quantitative measures, including Style Bias Index (SBI), Substance Recognition Rate (SRR), and
Adversarial Win Rate (AWR), to characterize how stylistic variation affects scoring stability, substance discrimination, and ranking robustness; (iii) Third, SciStyleExtractor, a plug-and-play evaluation module that separates presentation style from scientific content by predicting style type and deviation before style-conditioned evaluation, enabling us to assess whether style awareness reduces stylistic bias. Experiments on SciStyleBench show that direct LLM judges remain sensitive to writing style and struggle to distinguish scientific substance. In contrast, SciStyleExtractor reduces SBI from 0.566 to 0.501 while increasing SRR and AWR from 0.504 and 0.554 to 0.759 and 0.899, respectively. These results suggest that robust idea evaluation requires invariance to stylistic variation without sacrificing sensitivity to scientific substance. Overall, SciStyleBench provides a systematic framework for identifying, quantifying, and mitigating stylistic bias in scientific idea evaluation.
Read the full abstract ▾
All authors Fengxian Ji, Yuke Li, Jingpu Yang, Juanfan Wu, Fan Zhang, Zhexuan Cui, Yu Xie, Min Peng, Qianqian Xie, Xiuying Chen, Zhuohan Xie